In a response to my LinkedIn post on how every data transform is technical debt, Tim Hiebenthal commented:
I totally agree with your statements, but I have doubts about the feasibility of implementing it.
Data contracts can be applied in various places, but they’re most useful at the boundaries of ownership.
It’s easy to see data contracts as something to enforce on your data producers.
But if that’s how you’re selling it to them, they won’t be keen on buying!
Data contracts set the expectations for the data.
These include:
Without expectations, users make assumptions that are more optimistic than reality.
I had a great panel discussion with Amy Raygada and facilitated by Jean-Georges Perrin for Data Mesh Radio.
As I wrote yesterday, many data professionals don’t trust the data they are building on. And many users of data and data applications don’t trust the data they’re being provided.
Earlier this week I was on a panel organised by Data Council and Soda titled “Data Contracts: The Next Frontier”. The recording is now available on YouTube.
Mostly when I talk about data contracts I’m talking about applying them to data generated within our organisations. That’s usually the most valuable data we have. It’s also the data we have most control over its generation.
APIs and data contracts have a lot in common, and APIs were part of the inspiration behind data contracts when I was coming up with the idea a few years back. The both provide the interface (see my post from a couple of weeks ago on the importance of interfaces), they both set expectations for the user (the structure, semantics, SLOs, and so on), and they both allow for integrations with other services, tools, etc.
In yesterday’s note I wrote about the problem with defaults. One response to my personal data example could be “why don’t we just infer it?”.
Want great, practical advice on implementing data mesh, data products and data contracts?
In my weekly newsletter I share with you an original post and links to what's new and cool in the world of data mesh, data products, and data contracts.
I also include a little pun, because why not? 😅
(Don’t worry—I hate spam, too, and I’ll NEVER share your email address with anyone!)