Why Your Data Engineers Have the Worst Seat at the Table (And How to Change That)

See why thousands are shifting their perception of data engineers in the evolving data landscape.

In your enterprise if there is a common belief that data engineers are responsible for extracting and munging data to make it analysis-ready then I challenge you to think again.

10 years ago, the only way to process complex data sets was by coding handcrafted map-reduce/spark jobs. Until 2-3 years ago, custom scripts were the most common way to extract data from REST APIs and internal databases. Naturally, it felt on data engineers' lap to bring the data into the platform they would use to transform it into clean datasets. This need for specialized software engineers who work with data were called Data Engineers (DE); their role being focused on ETLing data. ETL: Extracting Transform and Load. These data engineers had the "worst seat at the table". They would do all the complex work to only give visibility to insight-generating data scientists and analysts.

You don't believe me when I say this is all changing? Here are some examples disrupting the traditional data engineer role:

• Data integration services bring data to your data platform with a few clicks

• DBT enables data analysts to own the core datasets

• Cloud Data Platforms (Databricks, Snowflake) has automated database administration and infrastructure management

The modern data ecosystem is rapidly removing the inefficiencies in the "Data ➡️ Biz Impact" Workflow.

This heavily impacts the role of our data engineers and what high leverage activities look like for them. This is where you should really lean into this article..

EL tools (Fivetran, Airbyte) + T tools (DBT) + Metadata tools (Atlan) make it easier for more and more people to participate in unlocking business value from data. I have consulted with fortune 500 enterprises in my role with Mckinsey and have seen data engineering as an expensive and slow dept to work with time and time again. If you leverage data engineers to build and stitch the right tooling together, you unlock their time for solving hard problems. This in turn makes your analytics org more efficient and impact generating.

My recommendation on how to get there.👇

Push your data engineers from plumbing data to building data tooling and platform

Make your data engineers responsible for data flowing across the organization and tooling required to supercharge your insight generation workflow.

If you do this right, data engineers will make it easier for teams to rapidly build and iterate on the gold mine of data assets and insights. This will completely shift the trajectory of your business. With reliable and easily discoverable data the enterprise will have a cultural shift where most teams will rely on data-driven decisions.

Today, a data analyst can build the entire analytics pipeline without the need for a data engineer.

Traditional DE is going to be obsolete soon

I encourage you to update your mental model regarding the role of DEs and not make the mistake that will handicap your entire data org.

As software automates the boring and repeatable parts of data engineering, DEs who write ETL's are on their way out. In the coming future, DEs will manage data infrastructure, aid in building complex performant transformations and tooling to efficiently build data analytics and ml products - this is your critical path to building high functioning data team.

If this resonates with you, follow me at @nehiljain to get more.