Schema-on-read vs schema on write: It's about the costs

Hello 👋

In this week’s newsletter I look at the costs of schema-on-read vs schema on write.

There’s also links to articles on an AI-ready warehouse, modelling and semantic layers, and where data goes.

Also, I’ll be speaking at the Compass AI & Tech Summit on the 1-2 October in Budapest. Let me know if you’ll be there too! And if you want to join, use the code AndrewSentMe for 20% off :)

Schema-on-read vs schema on write: It’s about the costs

The schema-on-read vs schema-on-write debate is often presented as an architecture choice.

But really it’s not two architectures. It’s just about where the schema is applied. With schema-on-read it’s later, at the point of consumption, while with schema-on-write it’s earlier, at the point of creation.

The result of that decision is the cost of transforming the data into a usable structure.

Anyone who has worked on a pipeline that applied schema-on-read knows the high costs applied to them as a result of that decision. You have to infer the structure as best you can, react quickly to incidents when the schema changes and breaks your ingestion, and handle all schema evolution through a large amount of IFNULLs, COALESCEs, and so on, adding a load of complexity and tech debt to your pipeline.

This cost is then multiplied by each consumer of the data.

The cost to the producer is near zero, but the overall cost to the business of having usable data is high.

Schema-on-write transfers a little bit of the cost to the producer. They now have to design a schema and maintain it as their code changes. When they make a change to the code, their CI checks will fail if the schema breaks, and they can then make an informed decision about whether to maintain the current schema, or invest in a schema migration to a new version.

These are known costs because they are the same as the costs of maintaining APIs, and therefore easy to estimate and factor into any implementation plan.

Meanwhile the costs to the consumer are much reduced, avoiding almost all the costs of incidents resulting from unexpected schema changes and the resulting complex pipeline logic.

This leaves a much more balanced cost between the two teams, and a lower cost to the overall business.

This isn’t an architecture decision, it’s simply a decision on where the costs are paid, and the total cost to the business.

Interesting links

BlaBlaCar’s AI-ready warehouse by Maxime Rosina (BlaBlaCar)

Interesting design choices with a focus on context.

The Semantic Layer and Data Modeling Dilemma in AI Analytics by Timothy Wong (on LinkedIn)

Model the core. Curate the context. Evaluate continuously.

Where Does Data Go? by Tanusri Mishra

This is an interesting and fun read. It’s not inaccurate, but I’d love to reduce the amount of steps.

Being punny 😅

They thought I’d get the name of the Black Eyed Peas frontman wrong. I said Will I Now?

Upcoming events

🆕 Compass AI & Tech Summit, 1-2 October, Budapest. Use the code AndrewSentMe for 20% off!

Thanks! If you’d like to support my work…

Thanks for reading this weeks newsletter — always appreciated!

If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.

Enjoy your weekend.

Andrew