The internal cost of data quality issues

Happy Halloween, if that’s your thing!

Today I write about the internal cost of data quality issues - the costs to your team.

Also links to articles on why you’ll never have a FAANG data infrastructure, handling multiple data platforms, and the dbt blues.

The internal cost of data quality issues

Data teams often spend a lot of time articulating the cost of poor data quality to the business, but what about the internal cost of data quality on your team and your individual team members?

How frustrating do they find it to be constantly looking for the data they need, finding that one person in the organisation who understands the data and can explain what it means to you?

How do they feel when the log on most mornings and find the pipelines have failed again, through no fault of their own, but it’s they who have to tell everyone and they who get the blame?

How apprehensive are they about making changes to their ETL, knowing it’s already a complex mess of workarounds and COALESCE statements attempting to work around poor quality data, and another workaround is likely to cause unknown problems later?

Is this data hell?

It is of course important to articulate the cost of poor data quality in business terms, highlighting the impact on processes and outcomes that are being caused by these data issues.

But don’t forget the internal impact it is having on your team.

Interesting links

Why You’ll Never Have a FAANG Data Infrastructure and That’s the Point by Travis Thompson

Don’t copy the tech stack from a FAANG or any other company. Understand the problems they were trying to solve and how they approached it and apply that approach to your similar problems.

Also good points on the benefits of platforms.

Handling of Multiple Data Platforms by Peter Baumann

A good look at how to handle multiple data platforms, which is a common situation in many organisations.

dbt blues by Timo Dechau

dbt is great. It made orchestrating data transforms easy (maybe too easy, which is why teams end up with hundreds or thousands of them). But it was never going to be as valuable as their valuation suggested.

Timo also starts thinking about what would be that valuable - whats’s the next problem to solve?

Being punny 😅

Does anyone want to buy my broken barometer? No pressure.

Thanks! If you’d like to support my work…

Thanks for reading this weeks newsletter — always appreciated!

If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.

🆕 Also check out my self-paced course on Implementing Data Contracts.

Enjoy your weekend.

Andrew