Many tech companies follow the “you build it, you run it” mantra - and for good reason. You want to empower your teams to build high quality services and solutions, and that means feeling the pain when things aren’t quite as good as they should be.
The revelation by British, Dutch, and American spies of the systematic espionage by Russia was astonishing in many ways. One point to note was how Western governments are using transparency as a tool to counter the actions of the Russian GRU. Through transparency, they aim to sway global public opinion against the Russian government.
When some people talk about a Data Lake (or Hadoop, or even just Big Data), they go on to say that we can store all our data, unstructured, forever, and be able to analyse it at any time (maybe even in real-time!).
I’ve been investigating using Apache Kafka with Apache Beam and set up a small prototype environment using Docker Compose. It’s published on GitHub.
For the last year I’ve been using Wercker to deploy this site each time I make a change (normally a new post) and push it to BitBucket.
A while ago I wrote about using Finagle for an API endpoint. I now have another need to write a simple API in Scala and this time I’m going to use Finch on top of Finagle.
I recently spent a couple of nights in Berlin. Here are some notes and photos from my trip.
The benefits of using Docker for your development environments are well known by now. You get consistency across machines, the ability to easily install databases locally, and everything is self-contained.
A partition is a group of your data. When loading from HDFS, Spark will create a partition for each one of your files. This is normally fine, but there are times when you want to repartition, and Spark has a few functions to do that. The most commonly used are repartition and coalesce.
