The dead letter queue pattern

The dead letter queue (DLQ) pattern is commonly used to increase the resiliency of services that process data, allowing them to recover from incidents that prevent the processing or writing of that data without losing that data.

There are several reasons why an event might end up in a dead letter queue:

1. The structure of the data might be invalid or unexpected. For example, an expected field is not present, a data type has changed, and so on
2. The value of a particular field might be unexpected. For example, a timestamp that is usually in the past is in the future, or a string expected to be an email address does not match that format
3. We may not be able to write to the data store. For example, it may be unavailable, or our credentials might be invalid

The dead letter queue acts as a holding area for these events. It moves them out of the processing pipeline, allowing the service to carry on processing valid data if it can.

The dead letter queue also helps with debugging and troubleshooting. The service owner can inspect these events to investigate and identify the problem.

Once the issue is resolved, they can reprocess or resubmit these events through the system, ensuring no data is lost.

While most commonly used when working with event streaming platforms, dead letter queues are a useful pattern whenever creating resilient data-driven services and can be implemented using a variety of data stores.

The following diagram shows the dead letter queue pattern:

In summary, the dead letter queue pattern enhances the resiliency of data-driven services. It provides a safety net for handling failed events, ensuring event integrity, and facilitating efficient debugging and error resolution processes.