The impact of AI-assisted code on your data platform

Hey, hope you had a good week!

In this week’s newsletter I write about the impact of AI-assisted code on your data platform, and how it moves the bottleneck.

There’s also links to articles on how most teams are not building governance, going from data engineering to knowledge engineering, and the case for agent contracts.

And finally, a colourful pun.

Enjoy!

The impact of AI-assisted code on your data platform

AI-assisted coding is increasing the velocity of engineers, which is moving the bottleneck of data engineering teams from writing code, to deploying and running code.

This increases the impact the data platform has on your organisation, for better and worse.

For example, if your engineers are constrained by a lack of suitable testing environments, this constraint will be amplified as data engineers increase their coding velocity.

So, as a data platform team you will want to invest in testing environments that run locally on their machine or ephemeral environments in the cloud.

Another example is the increasing amount of changes deployed to production between pipeline runs, which make it harder to diagnose problems when something goes wrong.

As a data platform team, you therefore should invest in improving observability and tracing, and assisting data engineering teams as they split their larger pipelines into smaller units of work.

There are many more examples, and you will likely already know where the bottlenecks are in your data platform. Reducing those bottlenecks will be key in ensuring data engineers and your organisation benefit from AI-assisted coding.

Interesting links

Most teams don’t build data governance. They declare it. by John Wernfeldt

The boring work is the real work

💯

From Data Engineering to Knowledge Engineering in the blink of an eye by Veronika Heimsbakk

I’ve been enjoying this practical series on connecting data to an ontology. This is the first post in the series.

The Desperate Need For An “Agent Contract” by Michael Segner and Barr Moses

Every handoff between systems needs an interface.

Being punny 😅

A ship carrying red paint collided with a ship carrying purple paint. Both crews thought to be marooned.

Thanks! If you’d like to support my work…

Thanks for reading this weeks newsletter — always appreciated!

If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.

🆕 I’ll be running my in-person workshop, Implementing a Data Mesh with Data Contracts, in June in Belgium. It will likely be only in-person workshop this year. Do join us!

Enjoy your weekend.

Andrew