Daily data contracts tips

Many of us engineers enjoy building things.

That’s why we’re good at what we do!

There will be times when you look at the data platform you’ve got and think how great it would be to rip it out and start again.

An effective data platform provides the right tooling and abstractions that allow its users to get things done without the need for constant support from your team.

Data contracts underpin data products.

With data contracts we are explicitly saying data should be treated as a product by those teams who produce the data. That data is then provided through a stable interface.

We spend a lot of our time fixing the symptoms of poor quality data. We:

Passive data publishing is the norm in most organisations.

That’s where we’re using patterns like ELT or CDC to extract copies of the upstream database and replicate them in a data warehouse/lake. The data producer isn’t doing anything to facilitate this - they are passive.

As mentioned yesterday, data contracts are best stored with the code that generates the data.

Because data contracts are stored with the code that creates the data, and maintained by those who own the data, they become the source of truth for everything about that data.

Data contracts should be stored as close as possible to the place the data is generated.

Data contracts are great for declaratively describing the data and the infrastructure you need to make that data available to others and manage it in line with your governance requirements.

The more autonomy your data platform promotes, the more our users can get done without involving us directly.