Data Quality

While yesterday I wrote about prioritising data quality projects, I think it’s important to note that while for many of us the quality does need to be improved, we don’t need to be aiming for perfect.

It’s likely going to be difficult to get a project prioritised if the goal is defined only as “improving data quality”. So what? What’s the business value that will provide or enable? And does that align with the organisations wider goals?

If you’re building key product features on top of your data, shouldn’t it be as dependable as your software-backed features?

No matter what we do, when working at sufficient scale and/or sufficient speed it’s inevitable that things will go wrong. This is well accepted in software, and the same for data too. 

Most of the time we’re reacting to data quality issues.

Maybe someone has made a change to their database schema, and since we’re pulling that into our data warehouse directly from their database that breaks everything we’ve built. Or maybe the business logic has changed upstream, and we had our own version of that logic built on the data warehouse that has fallen out of sync.

If you’re working in one of the data teams then it’s useful to consider how important data is to your organisation. Does it (or something driven by it, such as ML/AI) appear in your company strategy? How easy is it to get investment for people and tooling? Where in the org chart does your data team sit?