Arriving at a solution

This is part 4 in my Data Reliability series. You can read the rest of the series on my website.

Following the root cause analysis described yesterday we now know what are the fundamental causes of our data reliability problem.

It’s now time to work on the solution.

Again, this is not something we want to do in isolation.

Build on those relationships we’ve forged during the incidents and the root cause analysis and bring your data producers into this conversation. They now have a much greater appreciation of the impact data reliability has on the organisation, and that gives them more incentives to help solve this problem.

For our example we’ll continue to look at upstream breaking schema changes, as they are a common problem in many organisations.

If we look at our fishbone diagram again we can see the major categories contributing to this problem.

It can be tempting to start with tools and systems and think about solutions there, because that feels more in our control than the other categories. However, the tools and systems are there to support people and processes, not the other way round.

The good news is we’ve already started solving made of the causes in the people category from the incidents and root cause analysis we’ve been working on. We’ve increased awareness of the downstream process that depend on data, we’ve explained the value, and we’ve started to foster some collaboration.

There’s always more to do, and you need to be consistent with this over the long term, but we’ve made a start.

We have a category named change management, and that’s looking promising as we search for solutions. If we could apply better change management on our data, we’d prevent many of this incidents from occurring.

This is going to require some more collaboration. It’s going to require the data producers to do more work.

But we can make a business case for that because we have data that shows how expensive these incidents are. By spending a little more upfront we’re saving a lot more later.

We also know that this is essential if we are going to meet our business goal of using data to drive revenue.

So that’s the next step - making that business case, and getting buy-in and agreement from data producers and other stakeholders.

Once we’ve done that, we can think about the tools and systems to implement change management. We’ll explore that tomorrow.