Container Camp London 2015

Some notes after attending this years Container Camp in London. This got a bit long, so first a tl;dr with the highlights:

• sysdig looks awesome for monitoring and troubleshooting servers and the services running on them, including inside containers
• IPFS builds on the Git and BitTorrent protocols (amongst others) to help you move data around and build distributed systems. They’ve implemented Starship to enable this for Docker images, but could be used for anything (don’t think Starship is released anywhere yet)
• What VMWare have done to run containers as a “special type of VM” looks very interesting

Towards a Containerized Future - Bryan Cantrill

Started off with a talk from Joyent on their history in containers and the key selling points of their offerings.

Apparently their OS, SmartOS, is based on OpenSolaris, but is still able to run Docker containers by looking enough like Linux.

Introducing the Private Container Service - Shannon Williams

Racher Labs talked about the different parts that make up a Container Service, something we’ve started building at work. They include:

• SDN (Sofware Defined Network)
• Service discovery
• Load balancer
• Health checks
• Storage management
• Configuration and secret management

Users should be able to install applications and services from a catalog. These applications might define what kind of host it requries (lots of memory, fast IO/SSDs, etc).

Organisations should be able to use different cloud providers alongside local datacenters, without the users knowing or caring.

Rancher Labs are building a implementation of a private container service with open source software.

Distributed systems with (almost) no consensus - Bryan Boreham

Bryan spoke about how Weave maintains state using eventual consistency rather than the Paxos or Raft algorithms, for supposably better performance and less network traffic.

Kubernetes: Changing the way that we think and talk about computing - Mandy Waite

Had already seen this talk at DevOps London. High level introduction to Kubernetes, what it’s trying to achieve and where it’s going.

An introduction to LXD, the container lighter-visor - Stéphane Graber

Stéphane from Ubuntu showed us LXD, which is a REST API for interacting with LXC. With LXD you can run any container (Docker, Rocket, etc) as well as LXC.

From the demo, it looked much like interacting with the Docker daemon, though probably nicer and working accross hosts. So I guess LXD can be seen as a replacement for the Docker daemon.

There is also an OpenStack plugin for LXD.

Lightning talks

Over lunch there were four lightning talks. The first was someone from VMWare showing how they have made Dokcer containers a “special type of VM”, which runs directly on the x86 layer alongside regular VMs.

Next was someone from ClusterHQ showing how Flocker allows you to move a container and take its local state with it, using the new volume drivers in Docker 1.8.

The thrird talk was from Cloud66 who are building a service which can either build a Docker image for you, from code, or take an existing image and run it on your cloud provider. They have around 900 customers, running on 4,000 servers.

Lastly was Force12, who are looking at building technology that monitors a demand metric (i.e. traffic on a web site) and automatically spin up containers in response to demand. Currently runs on ECS and Mesos/Marathon.

Docker network performance in the public cloud - Arjan Schaaf

Arjan spoke about benchmarking SDNs on the public cloud (AWS and Azure in this case), using standard network benchmarking tools.

Out of Weave, Project Calico, Flannel UDP and Flannel VXLAN, Flannel seemed to perform best in his case, and that is what they are using.

Managing Kubernetes and OpenShift with ManageIQ - Alissa Bonas

ManageIQ is an open source cloud management platform. It talks to your cluster managers (Kubernetes, OpenShift, etc) and models what is running on each cluster and where, helping you to manage and monitor all of your seperate clusters.

The modelling process is able to deduce relationships between your images, show utilisation of each node and the cluster as a whole, and provide a topology of your clusters.

In future, they will be adding dashboards, alerting, further analysis, etc.

Leveraging the DNS for fun and profit reactive infrastructure - Miek Gieben

Miek works on infrastructure and operations at improbable.io and dicussed some of the tools they use, including SkyDNS (service discovery and configuration via DNS, similar to Consul), Prometheus (monitoring), dinit (small init system for containers) and FlagZ (dynamic flags via etcd or Consul).

Lessons from running potentially malicious code inside containers - Ben Hall

This talk was from the founder of Scrapbook, which uses Docker to provide learning environments in the browser. Ben gave some examples of security holes in Docker that could allow malicious users or code to attack your services, such as:

• Exposing /etc/hosts in every container by default, giving you the names and IP of other containers on the same host (can be disabled with --icc=false)
• Running shutdown on an image with --net=host will shut down the host OS
• No CPU restrictions, so one bad image will bring the host to a halt
• Docker logs can keep growing, filling your disk
  • 1.8 brings log rotation
  • Move Docker logs to a seperate partition
• Bandwidth can’t be restricted

There are some tools to help with these issues:

• docker diff shows all changes made by the running image
• sysdig can also help inspect what happened (more on that later)
• They developed The Warden to implement other checks (not released yet)
  • As Docker matures, they expect The Warden to be retired

A Docker image walks into a Notary - Diogo Mónica

A talk from a member of the Docker security team about the improved security coming in Docker to prevent malicious code being downloaded and ran.

It’s based on TUF (The Update Framework) developed by Tor, which is currently the best standard for security distribution.

Dockers implementation of TUF is called Notary, which is implemented as a library. They are then using Notary to implement Docker Content Trust.

Interesting talk. Docker Content Trust is currently disabled by default, but will be enabled in future.

The Dark Art of Container Monitoring - Loris Degioanni

Loris showed off sysdig, which frankly looks awesome. It’s an open source monitoring tool which provides a wealth of data to help monitor and debug a service, and has built in support for containers. I need this!

Check out the demo video for an idea of what it can do.

Containers at Hyperspeed - Juan Batiz-Benet

The final talk of the conference started off by introducing IPFS, which aims to turn anything (including the Internet) into a fully distributed system. It builds on protocols such as Git and BitTorrent (but doesn’t use them directly) to reduce the amount of data that needs to be fetched, and if it is needed it can load from peers rather than centrally.

On top of IPFS they have built Starship, which provides a distibuted model for containers. When pulling a Docker image with Starship, it firstly makes sure we don’t have any of the data locally (similar to how Git does). It can then fetch new data from any of the Starship peers it knows about.

So we are no longer wholy reliant on the central Docker repository for all our data. If it goes down, we can continue working. We can also get the data from local servers, which would be faster and save bandwidth.

Starship doesn’t look like it’s released yet, but when it is it will be are open source under the MIT license (same as IPFS).

Other observations

• Almost everything in this space is written in Go
• Almost everything is Open Source
• There are so many competing tools and players. Presumably there will be some consolidation and some casualties in the next few years. Lucky the tooling is Open Source then…

Cover photo by Diogo Mónica.