Data Platform

Software engineers often have many capabilities at the platform layer that helps them deploy changes with confidence, including:

Are the consumers of your data platform active?

Can they pick what they need from the solutions your providing?

Think of the processes you have in place. Which ones are reliant on a single person?

“You broke our data, so your PRs now need our signoff.”

This is a common reaction from data teams who are feeling the impact of upstream data changes causing breakages in their pipelines.

All abstractions leak some details of the complexities it is trying to hide.

And that’s ok!

Your data platform shouldn’t presuppose what can be built upon them.

It should provide the capabilities that allow your users to innovate.

Many of us engineers enjoy building things.

That’s why we’re good at what we do!

There will be times when you look at the data platform you’ve got and think how great it would be to rip it out and start again.

An effective data platform provides the right tooling and abstractions that allow its users to get things done without the need for constant support from your team.

Data contracts are great for declaratively describing the data and the infrastructure you need to make that data available to others and manage it in line with your governance requirements.