Data Engineering

Hey, hope you’ve had a great week! First, a couple of things from me.

The early bird pricing for my June implementing data contracts workshop in Belgium ends Monday! Would love to see you there :)

Yesterday I talked to software engineers about data quality.

It went well!

Specifically, it was about the responsibilities they have, as data producers, when they migrate a data contract to a new version.

Responding to yesterdays post on LinkedIn, Anna Bergevin wrote:

I agree and interface makes things less tightly coupled. But in order to change this we’d need a big cultural change at the company to seeing publishing data for secondary use cases as a core function of the job. Part of why data teams go in and get the data vs the other team publishing an API is because it’s the data team’s whole job to make data available and takes minimal effort from the software/system team. If the team is expected to make and maintain an API (including change management) that is a LOT of extra work. […]

I came across this on LinkedIn (as a screenshot from Bluesky) this morning.

Good software teams change their database schemas all the time, to add features, improve performance, fix bugs, and so on.

Your customers assume the data application they are paying for is always correct and always up to date.

Many data engineering teams spend a lot of their time struggling to deal with upstream data. That includes:

A new data team, perhaps being created in a business that is evolving from a startup to a scaleup, often focus on accessibility.

When you remove data engineering as a bottleneck:

• You have more data available
• You have more data being used
• You have more value being realised through data

It also benefits those generating and consuming data, who can now:

Writing code. Modelling data. Building transformations.

That’s the easy part.

The hard part is using all of that to deliver meaningful value to your organisation.

The goal of every data architecture has always been to provide accessible data to the organisation.