Data Engineering

In response to my post on Medium on how every data transform is technical debt, Peter Flock commented:

Since Hadoop came along in 2006 and significantly reduced the cost of storing “big data” we’ve often been focused on how much data we can bring in centrally, with the assumption that we’ll use it to create value later.

I mentioned yesterday that one way to reduce the amount of data transformations (and the costs of them) is to challenge the assumption we need to bring all data centrally before it can be useful.

I enjoyed this post by Paul McMahon on how all code is technical debt.

As I wrote yesterday, many data professionals don’t trust the data they are building on.

At most of my recent talks I’ve asked the audience - who are made up of data professionals - a simple question: Do you trust your data?

Most of the problems we’re solving in our organisations are not unique to us:

3 common data assumptions I believe are worth challenging:

How do you like your data?

It’s that time of year again where teams everywhere are considering a code freeze for the holiday season.