Data Engineering

Recently I’m finding it useful to split some of the data-driven initiatives I’m working on into two parts: measure for reporting, and measure for action.

I talk a lot about data quality, and how it can be improved.

And that’s because generally, it’s garbage!

In response to my post on Medium on how every data transform is technical debt, Peter Flock commented:

Since Hadoop came along in 2006 and significantly reduced the cost of storing “big data” we’ve often been focused on how much data we can bring in centrally, with the assumption that we’ll use it to create value later.

I mentioned yesterday that one way to reduce the amount of data transformations (and the costs of them) is to challenge the assumption we need to bring all data centrally before it can be useful.

I enjoyed this post by Paul McMahon on how all code is technical debt. Paul argues that the more code an application has, the slower the development, due to the assumptions that exist in that code and the features they support. As such, he views all code as technical debt.

As I wrote yesterday, many data professionals don’t trust the data they are building on. And many users of data and data applications don’t trust the data they’re being provided.

At most of my recent talks I’ve asked the audience - who are made up of data professionals - a simple question: Do you trust your data?

Most of the problems we’re solving in our organisations are not unique to us:

• We need a way to store data
• We need a way to discover data
• We need a way to transform data
• We need a way to build and present dashboards
• We need a way to train and deploy machine learning models

And so on.

3 common data assumptions I believe are worth challenging:

1. No one else cares about data quality
2. You have to bring all data centrally before you can work on it
3. Your problems are unique to you, and therefore you need a unique solution