A business case for data quality

Hey, hope you’ve had a great week.

Today I write about creating a business case for improving data quality.

There’s also links to articles on ODPS hitting 1.0, a modern data platform with a focus on data products, and the data platform at Decathlon.

A business case for data quality

Many data engineering teams spend a lot of their time struggling to deal with upstream data. That includes:

Implementing pipelines to “clean” data
Responding to unexpected schema changes
Trying to find the owner of the data to give them the business context they need

This is high effort, low value work.

Worse, it affects your ability to deliver higher value work.

If you want this to change you need to:

Make this effort (and cost) visible
Propose how to reduce that cost

You can do this by collecting data on how often these things affect your work, what are the most common root causes, and how much time it costs your team.

Once you know the time, you can convert that to dollars (e.g. hours * avg hourly wage of your team).

And once you have dollars you can make a business case to propose investments which can be made that will reduce those costs.

That business case could be to get the upstream team to do a bit of work (such as adopting data contracts) to save a greater amount of work for your team (such as responding to unexpected schema changes).

A business case that improves data quality.

Interesting links

Announcing ODPS v1.0.0: Building the Language of Data Products by Jean-Georges Perrin

The Open Data Product Standard from Bitol hits version 1.0.

How Stifel built a modern data platform using AWS Glue and an event-driven domain architecture by Amit Maindola and Srinivas Kandi, Hossein Johari, Ahmad Rawashdeh, Lei Meng

Nice writeup of building an event-driven data platform on AWS, with a focus on data products.

Polars at Decathlon: Ready to Play? by Arnaud Vennin

Interesting read on the data platform at Decathlon.

Being punny 😅

Tried kangaroo flavoured beer last night. You can really taste the hops.

Thanks for reading, Andrew