The circuit breaker pattern

In electrical systems, a circuit breaker is a device designed to automatically stop the flow of electricity in a circuit as a safety measure to prevent damage from faults like short circuits or overloads. When a fault is detected, the circuit breaker “trips,” breaking the circuit and stopping the flow of electricity to prevent further damage or fire.

The concept was first adapted into software engineering by Michael Nygard in his book Release It!, and is used to detect and handle failures gracefully, preventing system-wide failures.

In data engineering we use the circuit breaker pattern to stop a data pipeline when the data does not meet our quality or integrity expectations. That prevents further damage that could lead to a worse outcome, such as the poor data spreading widely through other pipelines and services or being used to make automated decisions that could affect our customers.

We can use this pattern whenever we make a change to data, such as between our ETL steps or when we publish the data ready for use by downstream consumers. If the checks pass we’ll publish the data. If it fails we’ll break that circuit and instead send an alert. This is shown in the following diagram.

You want to use this pattern only when a crucial issue has been detected. For example, you could add a check ensuring no null values are present, but in practice your data might be fine for your consumers if there are a few missing values and that wouldn’t be worth halting your pipeline and raising a high priority incident. Instead you might add a threshold to that check, allowing for a certain number or percent of nulls that you might still need to investigate, but which do not need to break the circuit.

So, it’s a good pattern to keep it mind, but when you do use it ensure the thresholds are set appropriately depending on the severity of the potential issue.