Notes from the final day of QCon London 2014.
Discussion on how as more is automated, all professions are at risk of being replaced. I think this has been clear since the Industrial Revolution.
Not everyone was keen on the keynote:
That was utter bollocks #qconlondon
Another good tweet:
Feel like this keynote is from 10 years ago. All developers are not autistic and it's a bad excuse for IBMs 20% women tech staff #qconlondon
Have to agree.
A good introduction to Hadoop and related technologies, including YARN. Sets the foundation for the later Hadoop focussed talks.
A futher introduction to Hadoop. Very similar to the previous talk, though made a couple of different points.
Hadn’t heard of Vert.x before, so found this talk really interesting. Has some good ideas and features, helping you create a microservice achitecture.
Will be keeping an eye on Vert.x and watching how it develops.
I’ve been reading the eary access edition of Nathan’s Big Data book and this talk helped reinforce some of the ideas behind the lambda architecture.
If you are dealing with large volumes of data you really need to be aware of the lambda architecture. I highly recommed getting the book.
A look at how Shutl uses Neo4J. Didn’t go into enough depth for me - was like they wansn’t sure how technical to make the talk. However, graph databases do look really useful for certain problems.
A great talk on how the Guardian use Elasticsearch to produce real-time analytics on what people were reading on their website.
As I mentioned after the “Revealing the Uncommonly Common with ElasticSearch” talk on day one, Elasticsearch looks like a great tool for quickly analysing strcutured or semi-structured data on the fly.
So, that’s it for QCon this year. One of the best conferences I have been too (and certainly the biggest). Looking back on it and trying to think about what I have learned, there were a number of obvious trends over the 3 days.
It’s clear any big application these days ought to be split into microservices. Netflix and Groupon were two of the companies who spoke about this here, while Vert.x gives you a microservice architecture for free.
It doesn’t look like there are any emerging standard tools/technologies to help with your microservice architecture. Netflix has developed their own tools, and open sourced them. Sounded like Groupon have done the same. It would be interesting to know of other companies building microservice architecture and what they are using.
2. The lambda architecture
Another trend was the discussion of the lambda architecture as a way to build your Big Data applications. Many of the tools to build it have already emerged and are becoming mature (Hadoop, Flume, Storm, Spark, etc), we just need to use them together in the right way.
Once again I recommend you check out the Big Data book by Nathan Marz and James Warren.
3. Open Source
All the Big Data technologies over the last 10 years are open source (with the exception of Splunk), which is great! No upfront costs; allowing you to start small, experiment and iterate, and no vendor lock in.