Software Engineering

Yesterday I talked to software engineers about data quality.

It went well!

Specifically, it was about the responsibilities they have, as data producers, when they migrate a data contract to a new version.

An ounce of prevention is worth a pound of cure.

Or put another way, the earlier you manage something, the cheaper it is.

Interfaces are powerful. That’s why we see them everywhere in software engineering. In fact, I’d say they are essential when you want to depend on something provided by someone else.

The dead letter queue (DLQ) pattern is commonly used to increase the resiliency of services that process data, allowing them to recover from incidents that prevent the processing or writing of that data without losing that data.

We recently had an incident with our data pipeline, resulting in data being lost on route to our data platform. Of course, you never want an incident, but failures are a fact of life. What’s important is how you prepare for them and respond to them, and in that sense this was a great incident.

Postmortems are a well established process followed in the aftermath of an incident. Often the most visible output from the process is a structured document, with some or all of the following components: