This is my second one, written mostly for me but it may be interesting to you. I write these because I have found having time to reflect during an annual review to be a great tool to clear the cluttered mind and track progress towards my goal. It is like a writing meditation session of sorts. Completing the review is often a positive experience that leaves me feeling more grateful. This year was quite unique in many ways. Let’s dive in.

What I learned

• Power of building the right data platform and culture: Travel industry was hit the hardest from COVID-19. Having a modern data platform and advanced self-serve BI practice gave Snaptravel strategic agility. This allowed the company to navigate the hard times with confidence and ease unlike others in the space. Data best practices and democratisation is a game-changer in today’s era. Our CEO published a post of some tips which below the surface require correct and contextual data to get it right.
• Data team structure and future of modern data stack: At SnapTravel, we were concerned with the correct composition, placement, and workflow of the team so that it can be efficiently used as an asset to the company. The structure of a company’s data team is an unsolved problem for many today. There were 3 different variations of how to best configure our data team members. I think it is hard to get it right because the data stack itself is fairly nascent as compared to software stack; it is a lot more complex (a16z blog does a good attempt to capture it in a diagram) and demands a variety of skills. There were a lot of great resources published online about it this year of the various approaches people have used to solve the said problem. [1], [2], [3]. Different stages of data maturity demands different team configuration.
• Data engineering consulting: Tried my hand at consulting/freelance projects on the side. I was surprised that it took quite a while to get my first project; I think it was because freelance/consulting budgets were the first ones to be on the covid budget cuts. Once I got started, it was a sudden flood of projects and opportunities. Personal network and Upwork was where I found the first two projects. I loved the feeling of being the expert in the room who was responsible to solve a specific problem. I did not like that I didn’t have as much influence or holistic understanding of the project as I was used to.
• Drive: I am much more externally driven to do things than internally driven. That was disadvantageous when a lot of the external things got cancelled due to Covid-19. At the start of the year, I was training to compete internationally for a Dragon Boating Competition. Instead by the end of the year, I became a de-trained athlete. Something to work for future-me is to stay motivated and inspired within and not lose sight of things when the shiny carrot is gone.
• Professional Network: In the past two years, I have been actively working on surrounding myself with the best and brightest of data engineering and analytics in Toronto. It has paid huge dividends: sharing of code libraries and best practices, ideas of what to build next, hiring, speaking opportunities and recruiting. Covid had surprisingly removed barriers and made it easier to do this than before. Any and everyone was a video call away.
• Airflow Summit 2020: I attended the first-ever Airflow Conference as a speaker. It was a lot of great talks and all for free. I loved it. Seeing the way large companies like Twitter, Pinterest and Paypal etc use Airflow was inspiring and a great learning opportunity. It was interesting to see that many data engineering folks using airflow were working on solving similar problems; observability, metadata and lineage, scalable ML pipelines and testing.

Big Life Events

Covid-19

The pandemic was the highlight of the year. Going fully remote had made me rethink my workflows, routines etc. Working in a fast pace startup where things are chaotic as it is, became harder when we suddenly went remote. For example, it was difficult to communicate with the newer members of the team as there wasn’t enough documentation and tutorials for tools and workflows for it to be completely self-serve. Gradually, I got a hang of the new way and loved it; all it required was a change of perspective on communication and a new routine. I have more time at my disposal which allows me to invest in myself outside work. I have been wanting to build a great mouseless development and internet browsing environment. I finally got a chance to do it and iteratively improve it. So for every loss, I found a win. Covid threw off my plans of going to France for an international dragon boat regatta for example but made me pursue digital minimalism. I spent more time with nature to keep my mental faculties healthy, I learnt that walks with solitude are great for removing mental blocks while solving a problem.

Moving on from SnapTravel

After 4 years at Snaptravel, it is time to move on to new adventures. It was an incredible journey to go from a 6 person startup to a multi-million dollar travel company in a few years time.  I am proud to have contributed to a great team and built a modern data stack from scratch involving Stitch, Airflow, Snowflake, DBT, Looker. I am moving to McKinsey QuantumBlack NYC for my next gig. After trying consulting on the side, I learnt that I liked it enough to try it full time.

What went well

• Public Speaking: I did 4 talks this year which was a big improvement over 1 last year. I continue to believe it is important to share knowledge and experience in the topics that I am passionate about. Will do more of this in the coming years.
• Networking: Continued to build a network in the data space. DBT Slack was phenomenal help in this
• Journaling: Started journaling inspired by Minimalist Journaling amidst covid anxieties. This had a great domino effect on me. Some benefits were more awareness, more gratitude, habit building became easier and improved goal tracking.
• Personal Website: Launched a personal website (nehiljain.com) with the proper domain name this time. I had been lingering on it for quite some time.
• Got engaged 😇
• Adventures: Due to covid I ended up spending a lot of time outside in nature alone or with someone close to me. I went for some great camping experiences and hikes all year round.

What I want to improve on

• Streaming Data Pipelines: At SnapTravel, I didn’t get a chance to play much with stream processing. A goal for 2021 is to work on projects where I can deploy some Kafka, Flink, or Spark stream processing to production.
• Data Insights Delivery: I want to get better at driving the last mile of data projects in organisations. That is the decision making after we have unearthed some data-driven insight. Conveying data stories and insights to relevant decision-makers is an important skill I want to become better at.
• Ukulele: I picked up the Ukulele during the holiday break. I would love to get good at finger-style playing this year
• Writing: I am really bad at writing but it has so many benefits to me. I want to continue making progress in the quality and quantity of writing.
• Digital minimalism: After reading Cal Newport’s book, I fell in love with the concept of digital minimalism and living digital life with purpose. I feel I have only begun the journey. By next year I will be writing a follow-up post on how I live an intentional digital life.

On to a great 2021!! 🎉 🚀 📈