Stop writing data governance policies

Hello 👋 I’m writing to you from Data Mesh Live at Antwerp, where I’ve given a full day workshop and a well received talk on implementing a self-service data platform. Besides that I have lots to digest from some excellent talks and made many new great connections!

This week’s newsletter comes from that talk and is about scaling data governance by moving away from documents and towards automation.

There’s also links to articles on data product managers, context graph architecture, and autonomous web scraping.

Enjoy!

Stop writing data governance policies (Automate them instead)

The way data governance is implemented in many organisations is there is a data governance authority (maybe a team, maybe an individual such as a Chief Data Officer (CDO)) who publish data standards and policies that owners of data must follow.

You could argue this works ok when a central data team owns much of the data. They can take the time to understand these policies and the reasons behind them and invest in tooling to help them manage their data.

However, in federated architectures such as data mesh, where data ownership is moved out of the data team to people across the organisation, this approach fails quickly.

These data owners don’t have the time nor the wish to read all these documents, to become experts in data management, to invest in their own tooling. It’s too much!

On the other side, the data governance authority has less visibility on the implementation of their policies, and little confidence they are being followed consistently across the organisation, even if people say they are.

To scale data governance we need to move away from written standards and policies that require humans to implement, and instead define these in a way they can be codified, and then automate the policies through the data platform.

The automation gets the direction from the policies, and then uses data contracts to find the data across the organisation and gain the context it needs to correctly apply the data governance policies to that data.

For example, you may have a policy that customer data is deleted/anonymised after 2 years of inactivity. That is something that can be easily codified in your platform tooling.

Then the data contract provides the context required to apply that policy to the data, such as the categorisation of the data and its anonymisation strategy, as shown below using ODCS.

version: 1.0.0
name: Customer
schema:
    properties:
      - name: name
        logicalType: string
        description: The name of the customer.
        classification: restricted
        customProperties:
          - property: anonymizationStrategy
            value: hex
      - name: email
        logicalType: string
        description: The email address of the customer.
        classification: restricted
        customProperties:
          - property: anonymizationStrategy
            value: email

With this platform approach to data governance, the data owner can focus their time and energy on publishing data products that enable others to use their data, and the data governance authority can be confident their standards and policies are being correctly applied across the organisation.

Interesting links

There are not enough Data Product Managers - And that’s not the problem by Anna Bergevin, Gaëlle Seret and Juha Korpela

A great post on the emerging role of Data Product Managers and what exactly they should be doing.

Context Graphs Are a Convergence — And Convergence Needs Architecture by Kurt Cagle (on LinkedIn)

Interesting article on the architecture required for context graphs.

Autonomous web scraping with Claude Code by Max Halford

Nice, practical example of having an LLM automatically update a web scraping script as web pages change over time.

Being punny 😅

Too many authors to quote? No problem et al.

Upcoming events

• Data Community Conference, September, London

Thanks! If you’d like to support my work…

Thanks for reading this weeks newsletter — always appreciated!

If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.

Enjoy your weekend.

Andrew