Jenkins #1: Migrating to Pipelines
[Previously published at the now defunct MetaBrite Dev Blog.]
The MetaBrite dev team migrated most of their builds from Atlassian’s Bamboo Cloud to Jenkins Pipelines in late 2016/early 2017. This is a series of blog posts about that experience.
Jenkins Pipeline Series
The series so far:
Eviction
For three years, we used Atlassian’s hosted Bamboo Cloud service to build and deploy most of our code. In the summer of 2016, Atlassian announced that they were going to discontinue Bamboo Cloud on January 31st, 2017.
We looked around for a suitable replacement. We did not find anything would work well for us. We had requirements that were—surprisingly—hard to satisfy. Much of our code is in one large GitHub repository, from which we build several very different applications. Many of the CI systems that we rejected were designed to build only one project per repository. Another hard requirement was that a push to the master branch in this repo should not trigger a build of every application; only the applications in the affected directories should be built. We also didn’t want to host our own build system, preferring to pay for a service that considered hosting CI to be one of their core competencies. We liked some of the systems that we evaluated, but none of them satisfied our needs.
Failing to find a new system that we liked, we reluctantly experimented with hosting Bamboo Server ourselves, reasoning that it should be fairly easy to get our Bamboo Cloud builds working on Bamboo Server. It wasn’t. Getting Bamboo Server even half working was painful and unsatisfactory. Time for Plan C.
We took a closer look at Jenkins; more specifically, at CloudBees‘ hosted offering. Some of us had previously used Jenkins Free-Style Builds at other jobs in the past and were unenthusiastic about doing so again. In fact, we had already been customers of CloudBees for a couple of years, as most of our Android builds were built there.
We initially experimented with Free-Style builds and found them no more pleasing than we remembered. Hesitantly, we tried the Pipeline builds (formerly the Workflow Plugin) that formed the heart of last year’s Jenkins 2.0 release.
With Jenkins Pipelines, we finally found something that made us happy. But there was a long, steep learning curve and we haven’t reached the summit yet. This series of posts describes much of our hard-won knowledge. Had we found similar posts written by someone else before we started, we would have saved ourselves considerable time.
Pipelines
The Pipeline is the heart of Jenkins 2.0. The traditional Free-Style build has been supplanted by a more powerful mechanism that supports a rich Domain-Specific Language (DSL) and a variety of other features.
Having our build system be under source control was almost enough to sell us on Pipeline. After Bamboo, where a build had to be configured across multiple screens, tabs, and forms, having everything in a single Groovy file was appealing. (A few settings do still need to be configured in a Jenkins webform.) Being able to use an expressive language like Groovy as the basis of the DSL leads to compact, terse, maintainable, and reusable functionality.
This series of posts is not intended to be a comprehensive introduction to Pipelines. These resources may help:
- Pipeline Tutorial
- Wilson Mar
- SysAdvent
- DZone RefCard #218
- Top 10 Best Practices for Jenkins Pipeline Plugin
- Best Practices
- Best Practices for Scalable Pipeline Code
- Introducing CloudBees Jenkins Enterprise
The Pipeline documentation is, bluntly, not very good. It’s frequently confusing and often incomplete. You need to search and to be willing to paw through Java code in GitHub repositories.
Acknowledgements
The work on which this series is based was jointly done with Adam Porad. Several of CloudBees’ support engineers were very helpful, including Owen Wood, Allan Burdajewicz, Travis Sweet, and Adrien Lecharpentier.