Daily data contracts tips

The bystander effect is a social phenomenon where individuals are less likely to offer help to a victim when other people are present.

Data engineering is complex, and things will go wrong. Following up on incidents with some form of analysis helps:

Once you have a data contract defined you can represent it in as many different ways as you want.

People don’t like being told what to do.

But people do like being part of something bigger.

In electrical systems, a circuit breaker is a device designed to automatically stop the flow of electricity in a circuit as a safety measure to prevent damage from faults like short circuits or overloads. When a fault is detected, the circuit breaker “trips,” breaking the circuit and stopping the flow of electricity to prevent further damage or fire.

Following on from yesterdays mail, when I talked about the language of data engineering, it’s worth asking why you’re asked to accept “unrefined”, poor quality data.

The language we use is important. It helps define culture, it shapes behaviours, and it defines our identity.

Data contracts can be used to implement a data mesh.

Consider the four principles of data mesh:

Great data platforms reduce organisational complexity.

They:

• Establish standards, such as processes, interfaces and guidelines, by providing a golden path for teams across the organisation to follow
• Automate tasks, particularly around data governance, allowing users of the platform to focus on what they do best - delivering value
• Ease integration, allowing data to flow between different systems, applications and users seamlessly, preventing silos

Building a platform on data contracts enables each of these.

People new to data contracts often ask “how is that different to a schema?”.

And it’s a fair question, as schemas also describe data structures in a human and machine readable format.