Undisciplined use of Kafka leads to poor quality data

I was at Confluent’s Data In Motion event in London yesterday and this quote by Tim Berglund in they keynote caught my attention:

Undisciplined use of Kafka leads to poor quality data.

That’s 100% true - not just for Kafka but for any data movement/integration tool.

So, how do we use Kafka with discipline?

We:

1. Design schemas that meet our users requirements
2. Implement change management on those schemas
3. Publish the in the schema registry for serialisation and deserialisation

The producers of the data can then publish data that matches the schema, and the consumer can be confident that the schema won’t change without going through an agreed change management process.

These can all be implemented with data contracts.

Data contracts apply discipline to your data production.