Data Quality

dbt Lab’s 2024 State of Analytics Engineering found 57% of data practitioners highlight data quality as one of their chief obstacles in preparing data for analysis.

For a data consumer to know if the data is applicable to their use case, they need to know what to expect from the data.

I wrote yesterday how data quality shouldn’t be the focus.

But even with that, at some point you’ll be looking at a dataset and thinking “is this data good enough for what I want to use it for?”.

Data quality shouldn’t be the focus.

It’s too subjective, and on its own doesn’t really mean much at all.

What can you do with data you don’t trust?

Let’s take an example.

I’ve been looking at some data about our cloud costs, and that data is suggesting a clear action I can take to reduce our costs. Which sounds great!

We spend a lot of our time fixing the symptoms of poor quality data. We:

The costs of producing data increases as it passes through all the different pipelines and teams needed to refine the data into something useable.

Why would you create and maintain good quality data.

If you don’t know how it is being used.

Nash Squared’s Digital Leadership Report from last year recorded that 64% of organisations they interviewed think that big data and analytics are the way to deliver competitive advantage, yet only 1 in 5 are using it to deliver increased revenue.

On Thursday I’ll be presenting at the AIDA User Groups Pi Day event.

I’ll be talking about data quality, and why prevention is better than the cure.