Data Engineering

Many of us engineers enjoy building things.

That’s why we’re good at what we do!

Data engineers typically favour moving quickly over reliability.

That’s not how most of us would say it, but it’s true!

If you’re creating data, you’re modelling data.

You’re making a decision on how that data is presented to your users.

Computation is the most expensive part of your data stack.

So, if you’re thinking about becoming more cost effective, you need to reduce the amount of computation you need to do in order to produce value for the business.

I talk a lot about data quality and how it could be improved, because I strongly believe that with a little bit more discipline we can do a lot better.

An article that has stayed with me since it was published back in 2015 is Dan McKinley’s Choose Boring Technology.

You can’t do self-serve just by providing datasets that you understand, but your users don’t.

What do you want from your data?

Do you want it to be fast changing?

I enjoyed this post from Nicole Radziwill, PhD on LinkedIn:

How fragile are your pipelines? Start with this simple metric: COUNT THE JOINS. Every time you have to join, you’re making multiple assumptions about the underlying raw data, the biggest one being: you’re assuming it’s not going to change.

Following on from yesterdays post about measuring for reporting and measuring for action, one data-driven initiative I’m working on is our FinOps program.