The data reliability question you're avoiding

Hello again 👋

This week I write about the data reliability question you’re avoiding, and whether you are making the right trade-offs.

There’s also links to articles on relating rigour, how your pipeline might succeed but your data did not, and how decisions are really made.

Finally, last call for the early bird pricing for my in-person Implementing a Data Mesh with Data Contracts workshop, as that ends on 31st March.

The data reliability question you’re avoiding

Data engineers typically favour moving quickly over reliability.

That’s not how most of us would say it, but it’s true!

How often do you put a quick fix into your ETL that stays forever?
How often do you invest in tooling to help you build more reliable pipelines?
How often do you ship a pipeline without proper monitoring or alerting, hoping it fails loudly enough that someone notices?

If you’re deliberately making that trade-off, then that’s fine! Ensure your users have the same expectations and continue delivering this way.

If, however, your users want reliable data they can build on with confidence to deliver a data application that drives key business processes, or powers a (possibly ML-based) product feature that generates revenue…

Then either you’re setting the wrong expectations.

Or making the wrong trade-off.

Interesting links

Relocating Rigor by Chad Fowler

This is a great explanation on what the real impact of AI-assisted coding is. It simply moves where the rigour is applied, as other innovations did before it.

Your Pipeline Succeeded. Your Data Didn’t. by Robert Sahlin

While a data pipeline may succeed in execution, that doesn’t mean all the data has been ingested. Nice write up of building anomaly detection to catch this.

The Business Case Isn’t the Decision by James Miller

How decisions really get made.

Being punny 😅

Did you know vending machines kill more humans than sharks? Maybe it’s because sharks rarely use vending machines.

Thanks! If you’d like to support my work…

Thanks for reading this weeks newsletter — always appreciated!

If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.

🆕 I’ll be running my in-person workshop, Implementing a Data Mesh with Data Contracts, in June in Belgium. It will likely be only in-person workshop this year. Do join us!

Enjoy your weekend.

Andrew