Daily data contracts tips

Reposting last weeks post on the IKEA effect to LinkedIn kicked off an interesting discussion on the complexities of build vs buy.

Every handoff has a cost.

Yesterday I wrote about handoffs in the context of data. That was partly as a nice introduction to today’s note which I think is even more interesting - handoffs between teams.

Every handoff has a cost.

As data engineers we see this most often in the cost of handing off data. We move it from one system to another, paying the cost in the compute needed to do that, paying the cost in the duplication of storage, paying the cost in building and maintaining the pipelines doing it.

Yali Sassoon wrote a great article about What is, and what isn’t, a data contract. It’s quite timely as data contracts is becoming one of those terms that can mean whatever you want it to mean, like data governance or (possibly) data mesh.

Most of the time we’re reacting to data quality issues.

Maybe someone has made a change to their database schema, and since we’re pulling that into our data warehouse directly from their database that breaks everything we’ve built. Or maybe the business logic has changed upstream, and we had our own version of that logic built on the data warehouse that has fallen out of sync.

If you’re working in one of the data teams then it’s useful to consider how important data is to your organisation. Does it (or something driven by it, such as ML/AI) appear in your company strategy? How easy is it to get investment for people and tooling? Where in the org chart does your data team sit?

The IKEA effect is where one places greater value on something they built, or partially built themselves. It’s named after the Swedish firm who famously provide their furniture as flat-pack, requiring assembly by the customer.

I’ve written before how much I love writing.

Often, I’m writing for myself. By writing things down I feel forced to think critically about the subject. What do I know? What don’t I know? Am I able to explain it clearly? It’s a forcing function to help clarify my ideas.