The challenges of implementing a data mesh
Hey! Today’s newsletter is the start of a 5-part series I will be writing on implementing a data mesh, which I’m writing in the run up to my Implementing Data Mesh with Data Contracts workshop and my talk at the Data Mesh Live conference.
This post is on the challenges of implementing a data mesh. I’ll then follow up with a post on implementing each of the four principles of data mesh.
Hope you find this series interesting! Feel free to hit reply if you have any comments or feedback :)
There’s also links to an in-depth article on data contracts, another good article on data platforms, and an introduction to the columnar storage format. Enjoy!
The challenges of implementing a data mesh
It’s only 6 years since the concept of data mesh was published by Zhamak Dehghani on Martin Fowler’s site, and it’s hard to understate the influence it has had on the data industry.
It provided a dramatically different view of how data platforms and the organisation around them could be implemented to solve many of the problems the industry has been struggling with for years.
Data mesh is a large concept that promises a lot — perhaps too much — and that makes it attractive as a target but difficult to achieve.
It’s a bit like that how to draw an owl meme. There’s likely a large gap between the expertise and tools you have and implementing a full data mesh.
Even the most well-cited success stories, such as at Netflix, are arguably not particularly aligned with the data mesh as defined by Dehghani.
It’s for this reason you might hear people saying data mesh is “dead”. Or as Gartner put it, data mesh will be “obsolete before the plateau of productivity”.
- Domain-oriented ownership: It would unquestionably be better if the data was owned by the people who know it best, and those who have the ability to change the data (for better or worse).
- Data as a product: Many fields are looking to adopt a product mindset to tools and services that target internal users, and are seeing success doing so. Why would data be different?
- Self-serve data infrastructure: Again this is a concept we already see elsewhere, particularly in platform engineering and team topologies, and it would surely be better for data platforms to provide the functionality needed for anyone to build reliable, quality datasets.
- Federated computational governance: Data governance that depends on people following processes, accepting opaque roles, has long been a challenge. Let’s automate that away!
Each of those principles are worth considering on their own, to see if they solve the problems your organisation is having.
And each of those on their own are not too difficult to implement. You don’t need a multi-year programme to do this.
That’s why, in the run up to my Implementing Data Mesh with Data Contracts workshop and my talk at the Data Mesh Live conference I’ll be writing about how to implement those ideas, quickly and cheaply, in your organisation.
The term data mesh may or may not survive in the long term.
But the underlying ideas are sound and could solve some of the problems you’re currently having.
Interesting links
Preventing Issues with Data Contracts & Testing by Dylan Anderson
A great overview of data contracts in the context of data quality, observability, and testing. Also highlights the shift to the left to ensure quality data at the source.
Building data platforms that actually help people by Robert Sahlin
Treat your data platform as a product and focus on the user experience.
Back To The Basics: What Is Columnar Storage by Ben Rogojan (AKA the Seattle Data Guy)
Nice introduction to columnar storage any why it’s better suited for analytic workloads.
Being punny 😅
One of the shops in my town has moved. Ironically it’s the stationary shop.
Upcoming workshops
- Implementing a Data Mesh with Data Contracts - Antwerp, Belgium - June 5
- Alongside the inaugural Data Mesh Live conference, where I’ll also be speaking.
- Sign up here
Thanks! If you’d like to support my work…
Thanks for reading this weeks newsletter — always appreciated!
If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.
Enjoy your weekend.
Andrew