You broke our data, so your PRs now need our signoff
“You broke our data, so your PRs now need our signoff.”
This is a common reaction from data teams who are feeling the impact of upstream data changes causing breakages in their pipelines.
And it’s understandable. They’re feeling the heat, and it’s much cheaper to prevent issues than it is to fix them.
Of course there are many downsides: it doesn’t scale, it slows engineering down, and you’re causing friction between your teams and theirs which over time could break that relationship.
But the biggest problem is that you’ve misunderstood the root cause of these data issues.
It’s not that producers are intentionally to do the wrong thing - it’s the result of the systems you have and then culture you’re working in.
So, to fix it you need to:
- Improve the systems. For example, ensuring data is provided through an interface
- Improve the culture. For example, ensuring the data producers see the production of data as part of what they do
We do both with data contracts.