Data teams in dysfunctional environments

Hey everyone, hope you had a good week! Today’s post is on changing your environment.

Btw, I’ve spent a bit of time refreshing the website for my data contracts book. What do you think?

Data teams in dysfunctional environments

Many data leaders are struggling with similar problems, including:

• Data that is so poor it’s difficult and expensive to deliver anything.
• Their team spending all their time fire-fighting as issues are introduced upstream.
• A constantly growing backlog of things to do.
• Always having to rejustify their value.

You could argue they are working in a dysfunctional environment that is not set up for their team to succeed.

But despite this, few attempt to change their environment. They assume that is just what they have to live with.

Even when new approaches come along that prescribe how they could change their environment, they resist.

“That won’t work for us”, they say, without explaining why.

Which reminds me of this quote from Marshall Goldsmith:

After living with their dysfunctional behavior for so many years, people become invested in defending their dysfunctions rather than changing them.

I think that’s true. Data leaders end up defending the status quo. It’s almost part of their identity, to be complaining about the way things are, and celebrating every small win they have as one they achieved despite the odds being stacked against them.

It’s not easy to change the environment you’re operating in.

But it is possible.

For example, if the data you are receiving is so poor it takes so long to deliver anything, expose that problem to the rest of the organisation. Articulate how much it’s costing. Explain the opportunity costs from projects you’ll never get to.

Then propose solutions. Create explicit project dependencies on your data producers, explaining you can only deliver when you have quality data to build on. Educate them on what data quality means, and why it is important. Integrate data contracts to monitor the quality and support the data producers in providing this data.

Whatever is your biggest environmental problem is, think about what would have to change to solve that problem. Then create a plan for making this change.

There’s enough proof that things can change in organisations.

There’s no reason why you can’t change yours.

Interesting links

Why database APIs promote bad API design principles by Daniel Kocot

I wrote last week about data contracts and interfaces, and why we shouldn’t be building directly on top of databases via CDC. This article also looks at why we shouldn’t be building directly on top of databases via database APIs.

Data Product vs. Data Contract: What’s the Difference? by Jean-Georges Perrin

A data contract is the agreement, a data product is the deliverable.

Being punny 😅

I had to weigh an elephant recently. Actually not that different to weighing a human, just on a larger scale.

Upcoming talks & workshops

• Data Quality: Prevention is Better Than the Cure - Virtual Meetup - March 12 4pm GMT / 12:00 ET / 09:00 PT
  • I’ll be speaking about data quality and data contracts at the online Data Vault User Group meetup.
  • This will probably be the last time I’ll give this particular talk, which has been well received over the last 18 months or so.
  • Sign up for free here
• Implementing a Data Mesh with Data Contracts - Antwerp, Belgium - June 5
  • Alongside the inaugural Data Mesh Live conference, where I’ll also be speaking.
  • Early Bird pricing available until April 30
  • ​Sign up here​

Thanks! If you’d like to support my work…

Thanks for reading this weeks newsletter — always appreciated!

If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.

Enjoy your weekend.

Andrew