Big Data

As I start to think about some of the upcoming projects we’ll be working on over the next year and how we might go about building them, I wanted to consider where lambda architecture fits in our toolbox for building data services.

When some people talk about a Data Lake (or Hadoop, or even just Big Data), they go on to say that we can store all our data, unstructured, forever, and be able to analyse it at any time (maybe even in real-time!

Thinking about how your master dataset is stored and managed is arguably the most important part of architecting a data-driven system. Your master dataset is your source of truth. An issue here, be it hardware failure, human error, or something else, could easily result in irreparable damage to some or all of your data.

Last week I attended Strata + Hadoop World 2014 in New York.