We're no longer building a data platform

A few years ago I, as tech lead of a data platform team, proposed that we shouldn’t be building a data platform.

Instead, we should be providing tooling and infrastructure.

I did this because I wanted us to move from a team that manages and transforms other peoples data - data we had little context on - and instead provide tooling so they could easily manage the data themselves.

It was deliberately an attempt to shift this responsibility left, to those who owned the data and had the most context. It was also an attempt to remove a handover of data from one team to another, and bring those using data and those generating data a bit closer together. (I wrote about the cost of handovers a few days back.)

It would require us, as a team, to build quality tooling and infrastructure in a way that could be delivered to teams so they have the autonomy to manage their data, by self-serving what they needed, with appropriate guardrails so they do so correctly and in line with our policies. But I had confidence we could do so. We had a very good team of well-rounded software engineers who happened to be working with data, but who could easily be delivering infrastructure products.

Now, some of you may argue that what we’re providing is still a platform, depending on how you define it. And you’d be right! But that wasn’t really important. What was important is that by changing what we called it we could start thinking about it differently. We had a name for the old, a name for the new, and a vision on how it would be different. We even changed the team name to match!

And it proved to be a success! The moment I knew for sure was a bit later on from the change, when a member of the team looked at something we managed and said something like “this doesn’t align with our vision for how this capability should be provided”, and proposed how we could work on it so it did.

It’s also still how the team are building their “platform” today.

All this was around the same time I was starting to come up with data contracts, and I realised a data contract was the perfect vehicle to deploy this infrastructure and tooling. More on that tomorrow!