George V. Reilly

Jenkins #2: EC2 Slaves

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

The “slave” ter­mi­nol­o­gy is un­for­tu­nate, but the utility of running a Jenkins build on a node that you’ve configured at Amazon’s EC2 is undeniable.

#2 in a series on Jenkins Pipelines

We needed to install system packages on our build nodes, such as Docker or Postgres. For obvious reasons, Cloud­Bees—our Jenkins hosting provider—­won’t let you do that on their systems. You must provide your own build nodes, where you are free to install whatever you like.

We already use Amazon Web Services, so we chose to configure our CloudBees account with EC2 slaves. We had a long and fruitless detour through On-Premise Executors, which I will not detail here.

Ultimately, it turns out to be straight­for­ward to create and manage EC2 slaves.

Create an AMI

First, build a custom Amazon Machine Image (AMI).

Here’s the script we use to provision Ubuntu 16.04. We SSH into the instance, then run this script.

# Provision an Ubuntu 16.04 AMI for MetaBrite CI and Jenkins

echo "Adding PPAs"
# jo: JSON output from shell:
sudo apt-add-repository ppa:duggan/jo --yes

echo "Updating package list"
sudo apt-get update -q

echo "Install Docker"
curl -sSL | sh                      ➊

echo "Installing Ubuntu packages"
sudo apt-get --yes install \
    build-essential default-jre \                           ➋
    git vim wget \
    paperkey gnupg \                                        ➌
    libffi-dev libpq-dev libxslt1-dev libyaml-dev \
    python libpython2.7-dev python-dev python-lxml  \
    postgresql python-psycopg2 \
    jo jq \
    unzip zip

echo "Installing Python packages"                           ➍
curl -sSL --retry 5 | sudo -H python2.7
sudo -H pip install --upgrade virtualenvwrapper setuptools  ➎

echo "Adding GitHub to $KNOWN_HOSTS"
mkdir -p ~/.ssh/                                            ➏
ssh-keyscan -H >> $KNOWN_HOSTS                   ➐
chmod 600 $KNOWN_HOSTS
  1. This installs the latest stable Docker package, which is more recent than the packages supplied in the Ubuntu LTS.
  2. The default-jre package is needed to run the Jenkins Slave JAR. We work a lot with Python 2.7; the packages that you need are probably different.
  3. We’ll have more to say about handling secrets at build time with Paperkey in a future post.
  4. We also want the latest Pip for managing Python packages, in preference to the older system python-pip package. The -H argument to sudo sets $HOME to the target user (root).
  5. We install up-to-date system-level virtualenv (via vir­tualen­vwrap­per) and setuptools. We have all that we need to install all other Python packages into virtual en­vi­ron­ments at build time.
  6. We do not install a private SSH key for GitHub. (We did at first, but there’s a better way to handle this.)
  7. We establish GitHub as a known host using ssh-keyscan. This is needed to prevent Git+SSH saying that the au­then­tic­i­ty of the host can’t be es­tab­lished and asking if we want to continue. This would be a major problem in a non-in­ter­ac­tive build. Note: GitHub’s SSH key fin­ger­prints are not being verified here.

After you’ve installed everything you need in this EC2 instance, you need to create a new AMI from the instance.

Configure the Amazon EC2 Plugin

Once your AMI is available at AWS, you can configure Jenkins. Go to Manage Jenkins > Configure System, then scroll down to Cloud > Amazon EC2. (You may need to install the Amazon EC2 plugin.)

You’ll need an AWS Access Key/Secret Key pair from IAM. You’ll also need an SSH keypair so that Jenkins can SSH into your EC2 instance; don’t lose this or you’ll never be able to SSH into your instance to debug it.

The Setting Up Jenkins EC2 Slaves article covers most of this. The other pieces that you need to know:

# based on

echo "Downloading boot script"                              ➊
sudo curl https://<JENKINS_MASTER>/plugin/ec2/AMI-Scripts/ -o /usr/bin/userdata
sudo chmod +x /usr/bin/userdata

echo "Adding boot script to run after boot is complete"     ➋
sudo sed -i '/^[^#]/ s/exit 0/python \/usr\/bin\/userdata\n&/' /etc/rc.local
  1. Adjust <JENK­IN­S_­MAS­TER>. You may need to change https to http.
  2. The Init Script is run once, installing as a boot script at /etc/rc.local.

Let’s examine the boot script. You don’t need to copy this, as it’s available from your Jenkins Master.

import os
import httplib
import string

# To install run:
# sudo wget http://$JENKINS_URL/plugin/ec2/AMI-Scripts/ -O /usr/bin/userdata
# sudo chmod +x /etc/init.d/userdata
# add the following line to /etc/rc.local "python /usr/bin/userdata"

# If java is installed it will be zero
# If java is not installed it will be non-zero
hasJava = os.system("java -version")

if hasJava != 0:
    os.system("sudo apt-get update")
    os.system("sudo apt-get install openjdk-7-jre -y")      

conn = httplib.HTTPConnection("")            
conn.request("GET", "/latest/user-data")
response = conn.getresponse()
userdata =

args = string.split(userdata, "&")
jenkinsUrl = ""
slaveName = ""

for arg in args:
    if arg.split("=")[0] == "JENKINS_URL":
        jenkinsUrl = arg.split("=")[1]
    if arg.split("=")[0] == "SLAVE_NAME":
        slaveName = arg.split("=")[1]

os.system("wget " + jenkinsUrl + "jnlpJars/slave.jar -O slave.jar")     
os.system("java -jar slave.jar -jnlpUrl " + jenkinsUrl + "computer/" + slaveName + "/slave-agent.jnlp")

  1. Note that installing openjdk-7-jre will not work on stock Ubuntu 16.04, as openjdk-8-jre is now current. This is why we pro­vi­sioned the AMI with default-jre.
  2. This script reads the instance metadata to discover its con­fig­u­ra­tion. The Jenkins Master supplied this when it started the instance.
  3. Download slave.jar from the Jenkins Master; run it pointing back to the Master node.
  4. This boot process never exits. The slave code will continue running until the EC2 instance is stopped.

In your Pipeline scripts, be sure to use the label you configured above in your node blocks:

node('ubuntu') {                        
    timestamps {
        ansiColor('xterm') {
            stage("Source Checkout") {
                checkout scm
                // …

Jenkins will au­to­mat­i­cal­ly start up an EC2 instance running your AMI, or use an existing one if it has enough capacity.

blog comments powered by Disqus
Jenkins #1: Migrating to Pipelines » « Trump: Media Suppresses Coverage of Terrorist Attacks