Data Engineering

Following on from yesterdays post about measuring for reporting and measuring for action, one data-driven initiative I’m working on is our FinOps program.

Recently I’m finding it useful to split some of the data-driven initiatives I’m working on into two parts: measure for reporting, and measure for action.

I talk a lot about data quality, and how it can be improved.

In response to my post on Medium on how every data transform is technical debt, Peter Flock commented:

Since Hadoop came along in 2006 and significantly reduced the cost of storing “big data” we’ve often been focused on how much data we can bring in centrally, with the assumption that we’ll use it to create value later.

I mentioned yesterday that one way to reduce the amount of data transformations (and the costs of them) is to challenge the assumption we need to bring all data centrally before it can be useful.

I enjoyed this post by Paul McMahon on how all code is technical debt.

As I wrote yesterday, many data professionals don’t trust the data they are building on.

At most of my recent talks I’ve asked the audience - who are made up of data professionals - a simple question: Do you trust your data?

Most of the problems we’re solving in our organisations are not unique to us: