The listen to yourself pattern

Hey friends 👋

Today we have the final part of my mini-series on data publishing patterns, covering the listen to yourself pattern.

There’s also links to skipping the DWH, caching, and TimescaleDB.

The listen to yourself pattern for publishing events

As we’ve been discussing, with data contracts we want to move to a model where data and events are published to consumers, rather than replicated from databases. That requires adopting new patterns and providing the tooling that enables them.

We’ve looked at two so far, one where we simply publish events, which is easy and quick, but will lead to data inconsistencies. We also looked at the outbox pattern TODO, which is more complex and can impact database performance, but improves data consistency.

Today I’ll introduce the listen to yourself pattern, which like the outbox pattern aims to solve the dual-writes problem in a different way and with different trade-offs.

First a refresh on the dual-writes problem. This is where you have a system that needs to update two different places, in our case the database and the event broker. Logically, that should be one operation, but in this case it will be two operations, as shown below. If one fails, such as the write to the event broker, then data in those two systems will be inconsistent.

The listen to yourself pattern solves this by only writing the event to one place – the event broker.

The publishing service then consumes that event from the broker to update its local state. So, it’s “listening” to its own event, hence the name of the pattern.

This means your local database will match the data warehouse, solving the data inconsistencies problem caused by dual-writing.

Performance should be good, as you’re only writing to the message broker rather than complex data processing and database interactions.

However, the major trade-off we are making is our database now becomes eventually consistent, i.e. there’s a window where the system state hasn’t been updated yet. If the user of this service is expecting that state change to be reflected immediately, such as through a web UI or a subsequent API call, they may not receive the response they expect.

This is why the listen to yourself pattern is not as widespread as the outbox pattern, as many systems have not been designed to work with an eventually consistent local datastore, and refactoring them so they can is difficult.

However, it is a nice pattern to consider when you have the right opportunity.

Interesting links

Five Signs You Can Skip the DWH in 2025 by Aleksei Aleinikov

Do you really need a data warehouse? Good questions to ask yourself.

Caching by Ben Dicken at Planetscale

Comprehensive article on caching, with nice animations to help explain key concepts.

How TimescaleDB helped us scale analytics and reporting by Robert Cepa at Cloudflare

Good write up, including how they evaluated different solutions and tried to avoid adding complexity to their architecture.

Being punny 😅

I can cut a piece of wood in half just by looking at it. It’s true, I saw it with my own eyes.

Thanks! If you’d like to support my work…

Thanks for reading this weeks newsletter — always appreciated!

If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.

Enjoy your weekend.

Andrew