Limit what you have in your data contract
Hello 👋
In this week’s newsletter I write about limiting what you have in your data contract and only including fields that drive action, typically through automation.
There’s also links to articles on how a single source of truth initiative should focus on governance, inverting data architectures by federating data, and shifting quality left within a data warehouse.
In this week’s newsletter I write about limiting what you have in your data contract. There’s also links to articles on how a SSOT depends on governance, inverting data architectures, and shifting quality left.
Limit what you have in your data contract
Data contracts are a simple idea. They’re just a human- and machine-readable document that describes the data. It captures and manages that metadata, i.e. the data about the data.
You can put anything you like in that data contract! But, what should you have in it?
At a minimum the data contract must have a name, an owner, a version, and a schema. But it could also have Service Level Objectives (SLOs), data classifications, backup config - anything that gives context to your platform so it can act on the data.
That’s a key point. Avoid adding fields to your data contract that do not drive some action or automation. These fields just sit there as documentation and will go stale, while also causing more work and cognitive load for the data owners for no added benefit.
For example, asking data owners to categorise their data by whether it contains personal data, or contains public/secret/confidential information, might seem like a good idea, but if it doesn’t drive anything you’re just causing work for no reason.
If those categorisations are used to automate data retention policies, so the data owner no longer needs to apply retention rules manually on their data, they will see the benefit to themselves in completing this categorisation and keeping it up to date.
A field that drives automation stays accurate because there’s an incentive to keep it correct.
Interesting links
Why Most Single Source of Truth Initiatives Fail (And What Successful Teams Do Differently) by Deepika Saini (Housing/Proptiger/Makaan)
Nice article with the focus of governance, rather than building, if you’re trying to provide that “single source of truth”.
The Data Platform team isn’t deciding business logic. They’re implementing approved business logic.
To Every Agent Its Own Database by Joe Reis
Interesting approach and prototype. I do agree we should challenge some of the architectural assumptions we have had.
Broken Windows of Data. How Booking.com scales a shared DWH across multiple teams without turning governance into a delivery bottleneck by Tiago Ferreira
This is specifically about shifting code/data model quality left and enforcing standards and governance earlier in the development lifecycle and is a good read.
Being punny 😅
Police have just arrested the tongue-twister world champion. If found guilty, they’ll be given a very tough sentence.
Upcoming events
- Data Community Conference, September, Online
Thanks! If you’d like to support my work…
Thanks for reading this weeks newsletter — always appreciated!
If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.
Enjoy your weekend.
Andrew