↓Skip to main content

Your data quality problem is an incentives problem

·3 mins
Data quality

Hey everyone! In today’s newsletter I write about how data quality is really about incentives. There’s links to posts on shifting left and right, and to an interesting interview with Snowflake CEO Sridhar Ramaswamy by Ben Thompson. Finally there’s a particularly silly pun…

Enjoy!

Your data quality problem is an incentives problem

The root cause of your data quality problem is not a lack of tooling, poor data governance, or software engineers “not caring.”

The problem is caused by dependencies and incentives.

As a data team, you will always have dependencies on other teams for the data they have, which you need to deliver for your users.

Those teams have their own goals, and they are incentivised to deliver on those goals. That’s where they will focus and prioritise their investments.

Improving the quality of their data to help you is not one of their goals, and so, no matter how well-meaning they might be, they cannot prioritise it.

The only way to improve the quality of the data you depend on is to incentivise upstream teams to do so by explicitly making it one of their goals.

Exactly how you do this will depend on your organisation, but it will be the same as whatever your organisation uses to highlight dependencies and set team goals. For example, quarterly planning, KPIs, OKRs, and so on.

You bring to that process the value you are providing to the organisation through the use of data, the problems caused by data quality, and make improving the quality of the data an explicit dependency that needs to be addressed.

If the value is strong enough, there’s no reason why the dependency will not be highlighted and the upstream team will set a goal to provide data to you at the right quality.

Their incentives will then align with yours, and data quality will be prioritised.

Inspired by John Cutler’s post on dependencies and prioritisation on LinkedIn.

The Convergence - Shifting Left AND Right by Joe Reis

Data Engineering is changing, and it’s being driven by the business need to deploy data into more valuable business applications.

An Interview with Snowflake CEO Sridhar Ramaswamy About Data and AI by Ben Thompson Great interview, touching on business models, open data formats, AI, and more.

See also Ben’s interview last year with Databricks CEO Ali Ghodsi (behind paywall).

Being punny 😅

Why is it spelt camouflage and not

Upcoming workshops

Thanks! If you’d like to support my work…

Thanks for reading this weeks newsletter — always appreciated!

If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.

Enjoy your weekend.

Andrew

↑