I’m no longer publishing daily data contract tips, but I am still writing! Check out my new weekly newsletter.
I had a really great chat last week with Juan and Tim on their Catalog & Cocktails podcast. They were both so interested in the topic and asked really great questions, and I think we did their “no BS” tagline justice!
I liked this post by Robert Sahlin and Johan Gunnarsson about how they used BigQueryML to create alerts for cost anomalies.
I get asked a lot if there is, or should be, a standard for a data contract. My answer has always been that a successful standard could be valuable and potentially unlock a lot of innovation.
I wrote yesterday about how by promoting autonomy we can empower the users of our data platform to take on more ownership and responsibility. But if you took that message to the extreme, you wouldn’t build a data platform at all! You’d just leave them to it, to make all the decisions on how they should generate, process and consume data.
You might have noticed how I keep using the word autonomy in these posts, for example a couple of days ago when writing about a contract-based data platform and yesterday when linking to lessons from BlaBlaCar. It might actually be my favourite word! And certainly guides how I think about the products we provide through our data platform.
This blog post from BlaBlaCar shares some great lessons - not just for implementing data mesh, but for any platform team looking to build products for internal stakeholders.
Data contracts are powerful enough you can build a data platform based on them - one that is more flexible and more effective than those we built before and promotes a more decentralised approach to data.
A few years ago I, as tech lead of a data platform team, proposed that we shouldn’t be building a data platform.
It’s a reality that when working in a team that supports or enables others you’ll have a steady stream of people asking you questions. Sometimes this stream can turn into a flood, and you spend more of your time answering questions than you do delivering projects.
No matter what we do, when working at sufficient scale and/or sufficient speed it’s inevitable that things will go wrong. This is well accepted in software, and the same for data too.