[Previously published at the now defunct MetaBrite Dev Blog.]
I attended DockerCon 2016 in Seattle over the last two days and I learned a lot. It was a well-run conference with an enthusiastic audience.
I’m astounded at the growth of Docker. Three-and-a-quarter years ago, Docker was revealed to the public for the first time, in a five-minute lightning talk at PyCon 2013. In January 2016, Docker Hub had received 1.6 billion image pulls; by this month, that number had jumped to over 4 billion pulls! DockerCon had over 4,000 attendees and nearly 100 exhibitors, who clearly believe there’s a multi-billion dollar market for containers. DataDog concurs, in a report on Docker adoption.
The Sad State of Clustering
I’ve become pretty good at building Docker images, thanks to my work on FlyingCloud, a tool that MetaBrite open-sourced earlier this year. FlyingCloud helps you build Docker images using SaltStack and makes it easy to run tests inside the container.
I’m generally happy with running containers but I’ve had less luck with deploying them in clusters. In fact, I spent last week fighting with deployments of various kinds of clusters on AWS: ECS, Kubernetes, and Elastic Beanstalk. Everything seems very immature.
I like the theory of Kubernetes; in practice, it’s been very painful for us on AWS. We have a couple of apps that we’ve deployed as Kubernetes clusters, using a homegrown script that uses kubectl, kube-aws, and boto3 to spin up a new cluster, provision load balancers, and so on. Both apps required heroic effort to bootstrap into working Kubernetes clusters. Worse, our deployment script has grown increasingly temperamental, failing to detect if the newly deployed cluster is ready. I talked to someone from the Kubernetes team yesterday, who promised some relief in a few months when 1.4 ships. He mentioned Terraform in passing.
I tried setting up a new app on ECS, but eventually I gave up. The very first time you create an ECS cluster, it configures 15 obscure pieces of AWS infrastructure. If you create another cluster, you have to jump through a series of non-obvious hoops to correctly configure your EC2 instances. I have not discovered a reliable way to update the cluster with a new image—sometimes the new container runs, sometimes not—much less a zero-downtime update.
I fell back to using Elastic Beanstalk, which is working fine for a Single-Container Docker app. Multi-Container Docker utterly failed—trying to start up ECS.
Therefore, I was very happy at yesterday’s keynote to hear all the announcements about Orchestration, Services, and Docker Swarm for AWS: Docker 1.12: Now with Built-in Orchestration!, Docker for AWS and Azure Beta, Distributed Application Bundles, More Microservices Bliss with Docker 1.12 and Swarm only Running Services in Docker 1.12, and DockerCon 2016 - What is new in Docker 1.12. The demos were very slick and very compelling. Docker for AWS has just entered private beta. As a DockerCon attendee, I should receive an invitation soon. Although it has been out for some time, I have yet to try Docker Compose. It looks like it simplifies multi-container apps.
I was also happy to hear about the Docker for Mac and Windows Public Beta. I’ve been a satisfied user of Docker for Mac for a couple of months, and now it’s available to everyone. I saw several demos where people were Developing Inside Docker Containers with OS X. They were able to edit and debug in Visual Studio Code with the source code shared between a Mac host and a Docker container. I should rethink some of my development workflow and stop installing so much in my host operating system.
Tuesday’s Keynote was about Democratizing Docker for Enterprise. Docker are touting Incremental Revolution. Certainly, containers are something that even big stodgy IT shops can no longer ignore.
All the talks that I attended were good. They were all recorded and videos should be available later.
Microservices + Events + Docker = A Perfect Trio made an interesting case for Event-driven microservices. More at LearnMicroservices.io and A decade of DDD, CQRS and Event Sourcing.
The Dockerfile Explosion and the Need for Higher Level Tools was very close to my heart, as I had written such a tool, the afore-mentioned FlyingCloud. Dockerfiles are not good for building complex images, as you tend to end up with a mess of imperative Bash scripts. The speaker covered several tools that I hadn’t previously known about, such as Dockramp and Rocker. Two new tools have been announced in the last week, Chef’s Habitat and Ansible Container. Both do far more than FlyingCloud, as they not only build but deploy and manage containers.
Immutable Awesomeness was a reprise of an earlier presentation at DOES15 about reducing operational pain. See also: Docker and the Three Ways of DevOps: Systems Thinking, Amplify Feedback Loops, and Culture of Continuous Experimentation and Learning.
Efficient Parallel Testing with Docker discussed Codeship’s approach to testing using a suite of Docker containers in a series of parallel stages. The Jet CLI can run your CI/CD pipeline on your local machine.
I was only able to attend one talk on the second day, Sharding Containers: Make Go Apps Computer-Friendly Again. The speaker’s Tesson uses the principles of Sharding, local Load Balancing, and Hardware Locality to pin Docker containers to processor cores, reducing Go garbage collection and CPU cache thrashing. The sharding (partitioning) and cache friendliness reminded me of my own LKRhash, a highly scalable hashtable.
I missed Microservices Best Practices.
The best hacks talks were also good. "Entropy is a failure orchestration microservice for Docker platforms." Building serverless apps with Docker was an experiment in spinning up containers on-demand to handle incoming requests.
Most of the dozens of sponsors and exhibitors had booths in the Exhibition Hall. There were an awful lot of people selling monitoring, hosting, storage, networking, and security.