Data Platform

David Jayatillake published an interesting post recently titled “We don’t need data contracts”. The title was enough to ensure it caught plenty of attention… but the subtitle more accurately describes what David is arguing for: “We need data to be part of product engineering”.

If you want your users to use the tools you are building or onboarding, it’s important to implement them exactly where they expect them to be.

The definition and evolution of the data contract should be a collaboration between the data producers and the data consumers.

An article that has stayed with me since it was published back in 2015 is Dan McKinley’s Choose Boring Technology.

Robert Sahlin wrote a nice post on the data platform at MatHem, a Swedish retailer.

Over on Medium I’ve published a post about a contract-based data platform, which is an updated version of my post to this newsletter back in December.

A few months back Whatnot published a great post on how they are using data contracts.

Data contracts set the expectations for the data.

These include:

• How to access the data
• How the data will be structured
• How often the data will be published
• Who to contact about the data (the owner)
• What will happen when the data needs to evolve

Without expectations, users make assumptions that are more optimistic than reality.

I mentioned yesterday that one way to reduce the amount of data transformations (and the costs of them) is to challenge the assumption we need to bring all data centrally before it can be useful.

Most of the problems we’re solving in our organisations are not unique to us:

• We need a way to store data
• We need a way to discover data
• We need a way to transform data
• We need a way to build and present dashboards
• We need a way to train and deploy machine learning models

And so on.