Data Platform

Your contracts-backed data platform should give your users a production ready way to generate, manage and consume data, and make that data available through a reliable interface.

APIs and data contracts have a lot in common, and APIs were part of the inspiration behind data contracts when I was coming up with the idea a few years back. The both provide the interface (see my post from a couple of weeks ago on the importance of interfaces), they both set expectations for the user (the structure, semantics, SLOs, and so on), and they both allow for integrations with other services, tools, etc.

In yesterday’s note I wrote about the problem with defaults. One response to my personal data example could be “why don’t we just infer it?”.

Let’s say 20 years ago you ran your code in an environment you configured simply as python. The obvious default would have been Python 2. But today the obvious default is Python 3. If you deployed that same code with the same configuration, what Python environment would you expect? What would you expect that to be in 20 years time?

How do you like your data?

Do you want it to be agile? So it can change at any moment, depending on the needs or wants of those producing the data? If a team decides it wants to model an object differently, with different IDs, they can do so. They are moving fast and breaking things.

It’s that time of year again where teams everywhere are considering a code freeze for the holiday season. Should we have one? How long for? What will we do while our code is frozen? Will we have lots to merge in January, and how will we manage that?

I wrote yesterday about how by promoting autonomy we can empower the users of our data platform to take on more ownership and responsibility. But if you took that message to the extreme, you wouldn’t build a data platform at all! You’d just leave them to it, to make all the decisions on how they should generate, process and consume data.

You might have noticed how I keep using the word autonomy in these posts, for example a couple of days ago when writing about a contract-based data platform and yesterday when linking to lessons from BlaBlaCar. It might actually be my favourite word! And certainly guides how I think about the products we provide through our data platform.

This blog post from BlaBlaCar shares some great lessons - not just for implementing data mesh, but for any platform team looking to build products for internal stakeholders.

Data contracts are powerful enough you can build a data platform based on them - one that is more flexible and more effective than those we built before and promotes a more decentralised approach to data.