The Contract-driven Data Platform
Hey 👋
This week I introduce the contract-driven data platform, and link to an in-depth post I wrote for Modern Data 101.
There’s also links to articles on the AI wave, the relevance of business models to data, and a self-improving Text-to-SQL engine.
If you have some time off over Easter I hope you have a great time 🐰🥚
The Contract-driven Data Platform
Traditional data platforms are a mix of disparate datasets with different attributes, different capabilities, and different levels of governance, each created and managed through complex custom workflows.
For data consumers, this results in data that is difficult to use. Every time someone wants to consume a dataset, they need to gather all the context required to do so, from scratch, often by tracking down and asking the right individual in the organisation. This increases the cost of using data, reducing its applicability.
On the other side, data producers find publishing data is expensive and time-consuming. They need to learn how to make this data available and create all the workflows to do so. They then, as data owners, need to learn how to manage this data in accordance with a dozen internal data standards and take responsibility for doing so.
All of this disincentivises the sharing of potentially valuable data.
What if there is a different way to build a data platform?
One that is made up of consistent, interoperable, governed data products that can be easily and confidently consumed to create valuable applications, either directly or through the use of AI.
There is, and it’s what I call the contract-driven data platform.
A contract-driven data platform moves the focus of data engineering and platform teams away from building point solutions to building a platform of generic capabilities that allow them, and anyone else, to easily and cheaply build interoperable data products.
It is a step-change in how we build data platforms that reduces the cognitive load for both data producers and data consumers, enabling the greater availability and applicability of data across the organisation.
I describe this platform more, including showing how to implement it, in my guest article for Modern Data 101.
Interesting links
The Wave by Timo Dechau
Interesting post on the impact of AI on data tooling.
Relevance of Business Models for Data by Dylan Anderson
A data strategy is a data-focused articulation of the business strategy
💯
Unified Context-Intent Embeddings for Scalable Text-to-SQL by Keqiang Li and Bin Yang (Pinterest)
Great detailed post on creating a self-improving Text-to-SQL engine.
Being punny 😅
What’s leather and sounds like a sneeze? A shoe!
Thanks! If you’d like to support my work…
Thanks for reading this weeks newsletter — always appreciated!
If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.
🆕 I’ll be running my in-person workshop, Implementing a Data Mesh with Data Contracts, in June in Belgium. It will likely be only in-person workshop this year. Do join us!
Enjoy your weekend.
Andrew