Daily data contracts tips

A few months back Whatnot published a great post on how they are using data contracts.

I talk a lot about data quality, and how it can be improved.

And that’s because generally, it’s garbage!

Staging layers, medallion architectures, data testing, assigning data stewards, gatekeeping application changes until reviewed by a data team.

In a response to my LinkedIn post on how every data transform is technical debt, Tim Hiebenthal commented:

I totally agree with your statements, but I have doubts about the feasibility of implementing it.

Data quality can only be improved at the source.

If the source of the data isn’t capturing the data at the required accuracy, there’s nothing you can do later to increase the accuracy.

In response to my post on Medium on how every data transform is technical debt, Peter Flock commented:

Data contracts can be applied in various places, but they’re most useful at the boundaries of ownership.

It’s easy to see data contracts as something to enforce on your data producers.

But if that’s how you’re selling it to them, they won’t be keen on buying!

Data contracts set the expectations for the data.

These include:

• How to access the data
• How the data will be structured
• How often the data will be published
• Who to contact about the data (the owner)
• What will happen when the data needs to evolve

Without expectations, users make assumptions that are more optimistic than reality.

I had a great panel discussion with Amy Raygada and facilitated by Jean-Georges Perrin for Data Mesh Radio.