Data Engineering

The whole point of enabling self-serve is autonomy.

It’s allowing others to do things you would usually have to do yourself, without you.

When it comes to defining SLOs, I’m aiming for good enough.

Good enough to meet the requirements of the consumers.

If you’re looking at implementing data contracts or moving towards a data mesh you will hear a lot about shifting left.

Data engineering is complex, and things will go wrong. Following up on incidents with some form of analysis helps:

In electrical systems, a circuit breaker is a device designed to automatically stop the flow of electricity in a circuit as a safety measure to prevent damage from faults like short circuits or overloads. When a fault is detected, the circuit breaker “trips,” breaking the circuit and stopping the flow of electricity to prevent further damage or fire.

Following on from yesterdays mail, when I talked about the language of data engineering, it’s worth asking why you’re asked to accept “unrefined”, poor quality data.

The language we use is important. It helps define culture, it shapes behaviours, and it defines our identity.

Data is created by various systems and applications owned by various teams. Without attention that will naturally lead to divergent data as they each decide how best to create and model the data they own.

An ounce of prevention is worth a pound of cure.

Or put another way, the earlier you manage something, the cheaper it is.

I wrote a few weeks ago about the difference between active vs passive data publishing, and how the active publishing of data leads to better outcomes for your organisation.