Data contracts vs APIs
Hey, welcome to July! Happy Independence Day for those in the US 🇺🇸
In today’s post I discuss how data contracts relate to APIs. There are also links to posts on choosing where to spend your teams effort, orchestrating dynamic data flows, and the sate of data mesh in practice.
Enjoy!
Data contracts vs APIs
APIs and data contracts have a lot in common, and APIs were part of the inspiration behind data contracts right from the start.
They both provide an interface that allows the provider to make changes to their applications without impacting downstream consumers, allowing consumers to build on them with confidence.
They both set clear expectations for the consumer, for example the structure, semantics, and SLOs.
The main difference is the type of interface provided.
APIs typically provide an endpoint you can call over HTTP, and when you call that endpoint it either triggers an action or returns a small amount of data.
While data contracts provide an interface for datasets, typically tables in a data warehouse but could also be a streams of data through Kafka, etc.
Both APIs and datasets need to be managed and governed. For example, both need access management, both may have fair use policies, and so on.
But datasets typically need more governance than an API endpoint, including data retention, data quality checks, backups, and more. So, we include more information in the data contract to allow for the automation of this governance.
That’s why I see data contracts at the intersection of interfaces, data governance, and data platform engineering.
I’ll be talking more about data contracts and how they relate to APIs with Jack Vanlightly and Shruthi Panicker from Confluent in this webinar on July 16th. There will also be a demo showing you exactly how to implement these interfaces.
Interesting links
Choosing where to spend my team’s effort by Frederick Vanbrabant
Good advice on working out what projects to work on to have the most impact.
Orchestrating Dynamic Data Flows, AgileData Engineering Pattern #1 by Shane Gibson
A nice pattern describing how to create reliable config-driven data flow orchestration.
The State of Data Mesh in Practice by Peter Baumann
Nice summary of where data mesh is today and how to apply it (or parts of it) in your organisation.
Being punny 😅
If I had a DeLorean, I’d probably only drive it from time to time.
(Yesterday was 40 years since Back to the Future was released. Classic film!)
Thanks! If you’d like to support my work…
Thanks for reading this weeks newsletter — always appreciated!
If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.
Enjoy your weekend.
Andrew