The draw of open storage formats

Hey, hope you had a good week! I was at Big Data LDN for day 1 and it was great to meet some of you!

On to the newsletter and this week I write about the draw of Iceberg. There’s also links to articles on real-time personalisation, shifting from imperative to declarative, and dbt-contracts.

The draw of open storage formats

There’s been a lot of talk about Apache Iceberg over the last year or so, but beyond the tech what’s the draw of an open storage format?

One often quoted benefit is to avoid vendor lock-in. That is a benefit, though data warehouse migrations should be rare, so I wouldn’t over-index on that when making a decision to use Iceberg.

For me the biggest benefit Iceberg gives us is optionality.

It allows us to make use of different query engines without paying the cost of integration between those different systems.

For example, by storing your data using Iceberg in S3 you can use Snowflake to query the data, which gives you flexible compute, advanced data management features such as data masking, and access to their AI tools.

Or you could load the data into DuckDB, either for local analysis or to provide an isolated query engine for an AI agent, ensuring the queries it runs do not overload your production data warehouse or lead to high compute bills.

Or run Apache Spark against the same data for large scale data transformation. Or any number of compute engines that are optimised for specific use cases.

This flexibility is a real benefit for your data architecture, particularly as tools are changing quickly as we explore new ways to make data available to applications, humans, and agents.

And that’s why people are drawn to the Iceberg.

Interesting links

Real-time Personalisation on Autotrader by Richard Wilmer (Autotrader)

Interesting read on how they built it and their design choices.

Data Engineering’s Shift from Imperative to Declarative - and Why It Matters More in the Age of AI by Brad Coles

Data contracts is part of this shift.

dbt-contracts by Joeri Van Dingenen

Interesting project to create dbt models from ODCS data contracts.

Being punny 😅

I was sitting drinking tea in my slippers this morning, when I thought to myself… I really need to wash some mugs.

Upcoming events

NEXT WEEK! Compass AI & Tech Summit, 1-2 October, Budapest. Use the code AndrewSentMe for 30% off!

Thanks! If you’d like to support my work…

Thanks for reading this weeks newsletter — always appreciated!

If you’d like to support my work consider buying my book, Driving Data Quality with Data Contracts, or if you have it already please leave a review on Amazon.

Enjoy your weekend.

Andrew