George V. Reilly

Jenkins #6: Miscellenea

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

A collection of mis­cel­la­neous tips on using Pipelines in Jenkins 2.0.

#6 in a series on Jenkins Pipelines

En­vi­ron­ment Variables

Use the withEnv step to set en­vi­ron­ment variables. Don't manipulate the env global variable.

The confusing example that you see in the documents, PATH+WHATEVER=/something, simply means to prepend /something to $PATH. The +WHATEVER has no other effect.


The withEnv step should not be used to introduce secrets into the build en­vi­ron­ment. Use the with­Cre­den­tials plugin instead.

    [$class: 'StringBinding', credentialsId: 'GPG_SECRET', variable: 'GPG_SECRET'],
    [$class: 'AmazonWebServicesCredentialsBinding',
     credentialsId: '0defaced-cafe-f00d-badd-0000000ff1ce',
     accessKeyVariable: 'AWS_ACCESS_KEY_ID',

Jenkins #5: Groovy

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

Jenkins Pipelines are written in a Groovy DSL. This is a good choice but there are surprises.

#5 in a series on Jenkins Pipelines

Groovy as a DSL

Groovy lends itself to writing DSLs (Domain-Specific Languages) with a minimum of syntactic overhead. You can frequently omit the paren­the­ses, commas, and semicolons that litter other languages.

Groovy has in­ter­po­lat­ed GStrings, lists, maps, functions, and closures.


Closures are anonymous functions where state can be captured at de­c­la­ra­tion time to be executed later. The blocks that follow many Pipeline steps (node, stage, etc) are closures.

Here's an example of a Closure called ac­cep­tance_in­te­gra­tion_tests, where the re­lease_lev­el parameter continue.

Jenkins #4: The sh Step

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

If there isn't a built-in Pipeline step to accomplish something, you'll almost certainly use the sh step.

#4 in a series on Jenkins Pipelines

The sh step runs the Bourne shell—/bin/sh, not Bash aka the Bourne-again shell—with the -x (xtrace) and -e (errexit) options.

The xtrace option means that every step in the sh block is echoed to the Jenkins log, after commands have been expanded by the shell. This is useful but you could echo the contents of passwords or secret keys in­ad­ver­tent­ly. Use set +x in your sh block to control this.

The errexit option means that the continue.

Jenkins #3: GitHub Integration

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

Much of our code is in one large GitHub repository, from which several different ap­pli­ca­tions are built. When changes are pushed to the master branch, we want only the ap­pli­ca­tions in affected di­rec­to­ries to be built. This was not easy to get right with “Pipeline script from SCM” builds.

#3 in a series on Jenkins Pipelines


Jenkins #2: EC2 Slaves

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

The “slave” ter­mi­nol­o­gy is un­for­tu­nate, but the utility of running a Jenkins build on a node that you've configured at Amazon's EC2 is undeniable.

#2 in a series on Jenkins Pipelines

We needed to install system packages on our build nodes, such as Docker or Postgres. For obvious reasons, Cloud­Bees—our Jenkins hosting provider—­won't let you do that on their systems. You must provide your own build nodes, where you are free to install whatever you like.

We already use Amazon Web Services, so we chose to configure our CloudBees account with EC2 slaves. We had a long and fruitless detour through On-Premise Executors, which I continue.

Jenkins #1: Migrating to Pipelines

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

The MetaBrite dev team migrated most of their builds from At­las­sian's Bamboo Cloud to Jenkins Pipelines in late 2016/early 2017. This is a series of blog posts about that experience.

Jenkins Pipeline Series

The series so far:


For three years, we used At­las­sian's hosted Bamboo Cloud service to build and deploy most of our code. In the summer of 2016, Atlassian announced that they were going to dis­con­tin­ue Bamboo Cloud on January 31st, 2017.

We looked around for a suitable re­place­ment. We did not find continue.

DockerCon 2016

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

I attended DockerCon 2016 in Seattle over the last two days and I learned a lot. It was a well-run conference with an en­thu­si­as­tic audience.

I'm astounded at the growth of Docker. Three-and-a-quarter years ago, Docker was revealed to the public for the first time, in a five-minute lightning talk at PyCon 2013. In January 2016, Docker Hub had received 1.6 billion image pulls; by this month, that number had jumped to over 4 billion pulls! DockerCon had over 4,000 attendees and nearly 100 exhibitors, who clearly believe there's a multi-billion dollar market for containers. DataDog concurs, in a report on Docker adoption.

The continue.

PuPPy Startup Row Pitch Night

Last night, Adam Porad and I were one of five teams pitching our startups at the PuPPy-organized PyCon Startup Row Pitch Night:

Techstars Seattle and PuPPy [Puget Sound Pro­gram­ming Python] presents PyCon Startup Row Pitch Night. The time has come again for you, the members of PuPPy, to select Seattle’s startup rep­re­sen­ta­tive to travel to PyCon in Portland to represent our Python community and startup scene at the annual conference produced by the Python Software Foundation.

We were pitching MetaBrite and our technology that captures receipts, yielding receipt in­for­ma­tion to users and onsumer insights. We use Python ex­ten­sive­ly—we've written 120,000 lines of Python code for web services, web apps, machine learning, image processing, and continue.

Sorting Python Dictionaries by Value

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

I needed to sort a Python dictionary by value today, rather than by key. I found it confusing, so I'll share what I learned.

Assume the following dictionary, where each value is a tuple of (ID, score). How do we sort by score; i.e., the second item in the value tuple? (For the purposes of this discussion, ignore the meaning of the dic­tio­nary's key.)

>>> some_dict = dict(a=(123, 0.7), b=(372, 0.2), e=(456, 0.85), d=(148, 0.23), c=(502, 0.1))
>>> some_dict
{'a': (123, 0.7), 'c': (502, 0.1), 'b': (372, 0.2), 'e': (456, 0.85), 'd': (148, 0.23)}

Python dic­tio­nar­ies are inherently unsorted, unless you continue.

URLs from Unicode Strings

[Pre­vi­ous­ly published at the now defunct MetaBrite Dev Blog.]

Some time ago, we made an ill-considered decision to use recipe names for image URLs, which simplified image management with our then-rudi­men­ta­ry tools. For example, the recipe named "Twisted Pasta With Browned Butter, Sage, and Walnuts" becomes a URL ending in "Twist­ed%20­Pas­ta%20With­%20Browned%20But­ter%2C%20Sage%2C%20and%20Wal­nuts.jpg".

Life becomes more in­ter­est­ing when you escape the confines of 7-bit ASCII and use Unicode. How should u"Sautéed crème fraîche Provençale" be handled? The only reasonable thing to do is to first convert the Unicode string to UTF-8 and then hex-encode those octets: "Saut%C3%A9ed%20cr%C3%A8me%20fra%C3%AEche%20Proven%C3%A7ale".

That seems reasonable, but it was giving us in­con­sis­tent results when the images were uploaded to an S3 bucket. When continue.

Previous »