The Sad State of Clustering

I’ve become pretty good at building Docker images, thanks to my work on Fly­ing­Cloud, a tool that MetaBrite open-sourced earlier this year. Fly­ing­Cloud helps you build Docker images using SaltStack and makes it easy to run tests inside the container.

I’m generally happy with running containers but I’ve had less luck with deploying them in clusters. In fact, I spent last week fighting with de­ploy­ments of various kinds of clusters on AWS: ECS, Kubernetes, and Elastic Beanstalk. Everything seems very immature.

I like the theory of Ku­ber­netes; in practice, it’s been very painful for us on AWS. We have a couple of apps that we’ve deployed as Kubernetes clusters, using a homegrown script that uses kubectl, kube-aws, and boto3 to spin up a new cluster, provision load balancers, and so on. Both apps required heroic effort to bootstrap into working Kubernetes clusters. Worse, our deployment script has grown in­creas­ing­ly tem­pera­men­tal, failing to detect if the newly deployed cluster is ready. I talked to someone from the Kubernetes team yesterday, who promised some relief in a few months when 1.4 ships. He mentioned Terraform in passing.

I tried setting up a new app on ECS, but eventually I gave up. The very first time you create an ECS cluster, it configures 15 obscure pieces of AWS in­fra­struc­ture. If you create another cluster, you have to jump through a series of non-obvious hoops to correctly configure your EC2 instances. I have not discovered a reliable way to update the cluster with a new im­age—­some­times the new container runs, sometimes not—much less a zero-downtime update.

I fell back to using Elastic Beanstalk, which is working fine for a Single-Container Docker app. Multi-Container Docker utterly failed—try­ing to start up ECS.

Keynotes

Therefore, I was very happy at yes­ter­day’s keynote to hear all the an­nounce­ments about Or­ches­tra­tion, Services, and Docker Swarm for AWS: Docker 1.12: Now with Built-in Or­ches­tra­tion!, Docker for AWS and Azure Beta, Dis­trib­uted Ap­pli­ca­tion Bundles, More Mi­croser­vices Bliss with Docker 1.12 and Swarm only Running Services in Docker 1.12, and DockerCon 2016 - What is new in Docker 1.12. The demos were very slick and very compelling. Docker for AWS has just entered private beta. As a DockerCon attendee, I should receive an invitation soon. Although it has been out for some time, I have yet to try Docker Compose. It looks like it simplifies multi-container apps.

I was also happy to hear about the Docker for Mac and Windows Public Beta. I’ve been a satisfied user of Docker for Mac for a couple of months, and now it’s available to everyone. I saw several demos where people were Developing Inside Docker Containers with OS X. They were able to edit and debug in Visual Studio Code with the source code shared between a Mac host and a Docker container. I should rethink some of my de­vel­op­ment workflow and stop installing so much in my host operating system.

Tuesday’s Keynote was about De­moc­ra­tiz­ing Docker for Enterprise. Docker are touting In­cre­men­tal Revolution. Certainly, containers are something that even big stodgy IT shops can no longer ignore.

Talks

All the talks that I attended were good. They were all recorded and videos should be available later.

High Security Mi­croser­vices dove deep into hitherto obscure areas. More at Stop Buying Bad Security Pre­scrip­tions and Un­der­stand­ing and Hardening Linux Containers.

Mi­croser­vices + Events + Docker = A Perfect Trio made an in­ter­est­ing case for Event-driven mi­croser­vices. More at Learn­Mi­croser­vices.io and A decade of DDD, CQRS and Event Sourcing.

The Dockerfile Explosion and the Need for Higher Level Tools was very close to my heart, as I had written such a tool, the afore-mentioned Fly­ing­Cloud. Dock­er­files are not good for building complex images, as you tend to end up with a mess of imperative Bash scripts. The speaker covered several tools that I hadn’t previously known about, such as Dockramp and Rocker. Two new tools have been announced in the last week, Chef’s Habitat and Ansible Container. Both do far more than Fly­ing­Cloud, as they not only build but deploy and manage containers.

Immutable Awe­some­ness was a reprise of an earlier pre­sen­ta­tion at DOES15 about reducing op­er­a­tional pain. See also: Docker and the Three Ways of DevOps: Systems Thinking, Amplify Feedback Loops, and Culture of Continuous Ex­per­i­men­ta­tion and Learning.

Efficient Parallel Testing with Docker discussed Codeship’s approach to testing using a suite of Docker containers in a series of parallel stages. The Jet CLI can run your CI/CD pipeline on your local machine.

I was only able to attend one talk on the second day, Sharding Containers: Make Go Apps Computer-Friendly Again. The speaker’s Tesson uses the principles of Sharding, local Load Balancing, and Hardware Locality to pin Docker containers to processor cores, reducing Go garbage collection and CPU cache thrashing. The sharding (par­ti­tion­ing) and cache friend­li­ness reminded me of my own LKRhash, a highly scalable hashtable.

I missed Mi­croser­vices Best Practices.

The best hacks talks were also good. "Entropy is a failure or­ches­tra­tion mi­croser­vice for Docker platforms." Building serverless apps with Docker was an experiment in spinning up containers on-demand to handle incoming requests.

Exhibition Hall

Most of the dozens of sponsors and exhibitors had booths in the Exhibition Hall. There were an awful lot of people selling monitoring, hosting, storage, networking, and security.