Change management can't be left to humans
A question I received on a LinkedIn post about change management with data contracts asked (lightly edited for clarity):
@Andrew Jones could you please walk us through it? Upstream data owner intend to make a change to the data schema.
There are 100 data contracts created and managed by multiple data contract owners.
How these 100 data contracts would drive change management if there is no Data Governance in place that could discover such a change?
In this question there is an assumption that change management must be done by humans, as part of data governance, and those humans need to discover and manage that change.
But you can’t rely on humans to manage changes to 100 data contracts. It’s simply:
- Too error prone
- Too expensive
- Too slow
Many breaking changes will get missed, and so you’ll still have many preventable data incidents caused by the lack of good change management.
With data contracts you’re not just relying on humans to perform change management - you’re using the platform too.
So, once you deploy a data contract into production, you have checks in place in the platform to prevent a breaking change to the schema. Typically that will be a CI check in a Git repo that makes it impossible for a breaking change to be made to that version of the data contract.
Of course, data will evolve and changes will need to be made. Non-breaking changes are fine (e.g. adding a new field), so they can be made without change management (low friction). But breaking changes must be deployed as a new data contract version (high friction).
Now, the human process for making a breaking change to a data contract needs to be defined, and that could still be part of data governance.
But as I wrote yesterday, data governance is best implemented as tools over rules.