Data Engineering

How do you like your data?

Do you want it to be agile? So it can change at any moment, depending on the needs or wants of those producing the data? If a team decides it wants to model an object differently, with different IDs, they can do so. They are moving fast and breaking things.

It’s that time of year again where teams everywhere are considering a code freeze for the holiday season. Should we have one? How long for? What will we do while our code is frozen? Will we have lots to merge in January, and how will we manage that?

Interfaces are powerful. That’s why we see them everywhere in software engineering. In fact, I’d say they are essential when you want to depend on something provided by someone else.

It’s a reality that when working in a team that supports or enables others you’ll have a steady stream of people asking you questions. Sometimes this stream can turn into a flood, and you spend more of your time answering questions than you do delivering projects.

Reposting last weeks post on the IKEA effect to LinkedIn kicked off an interesting discussion on the complexities of build vs buy.

Every handoff has a cost.

Yesterday I wrote about handoffs in the context of data. That was partly as a nice introduction to today’s note which I think is even more interesting - handoffs between teams.

Every handoff has a cost.

As data engineers we see this most often in the cost of handing off data. We move it from one system to another, paying the cost in the compute needed to do that, paying the cost in the duplication of storage, paying the cost in building and maintaining the pipelines doing it.

Most of the time we’re reacting to data quality issues.

Maybe someone has made a change to their database schema, and since we’re pulling that into our data warehouse directly from their database that breaks everything we’ve built. Or maybe the business logic has changed upstream, and we had our own version of that logic built on the data warehouse that has fallen out of sync.

The IKEA effect is where one places greater value on something they built, or partially built themselves. It’s named after the Swedish firm who famously provide their furniture as flat-pack, requiring assembly by the customer.

I’m so proud to announce that my book on data contracts is out now! Available wherever you buy your books, including Amazon, Packt, ebooks.com, Kobo and through the O’Reilly learning platform.