Data Platform

Data is published by many different systems, and as the number of systems increase it’s likely your organisations data will become less standardised over time, with different identifiers, different semantics, and so on.

It’s important to celebrate small wins.

We recently decommissioned a small utility service called subcontractor, which managed some of the resources we create from our data contracts.

Data contracts are typically used to apply change management to data in tables, in a data warehouse.

Implementing data contracts at the platform layer gives it the capabilities needed to enable domain ownership.

Software engineers often have many capabilities at the platform layer that helps them deploy changes with confidence, including:

Are the consumers of your data platform active?

Can they pick what they need from the solutions your providing?

Think of the processes you have in place. Which ones are reliant on a single person?

“You broke our data, so your PRs now need our signoff.”

This is a common reaction from data teams who are feeling the impact of upstream data changes causing breakages in their pipelines.

All abstractions leak some details of the complexities it is trying to hide.

And that’s ok!

Your data platform shouldn’t presuppose what can be built upon them.

It should provide the capabilities that allow your users to innovate.