System Design

When some people talk about a Data Lake (or Hadoop, or even just Big Data), they go on to say that we can store all our data, unstructured, forever, and be able to analyse it at any time (maybe even in real-time!

Thinking about how your master dataset is stored and managed is arguably the most important part of architecting a data-driven system. Your master dataset is your source of truth. An issue here, be it hardware failure, human error, or something else, could easily result in irreparable damage to some or all of your data.