Closures

Closures are anonymous functions where state can be captured at de­c­la­ra­tion time to be executed later. The blocks that follow many Pipeline steps (node, stage, etc) are closures.

Here’s an example of a Closure called ac­cep­tance_in­te­gra­tion_tests, where the re­lease_lev­el parameter is a String which must be either "dev" or "prod".

def acceptance_integration_tests = { String release_level ->
    assert release_level =~ /^(dev|prod)$/
    String arg = "--${release_level}"

    def branches = [
        "${release_level}_acceptance_tests": {
            run_tests("ci_acceptance_test", arg, '**/*nosetests.xml')
        },
        "${release_level}_integration_tests": {
            run_tests("ci_integration_test", arg, '**/*nosetests.xml')
        }
    ]
    parallel branches
}

We create a Map called branches with dy­nam­i­cal­ly named keys, such as "prod_in­te­gra­tion_tests", thanks to GString in­ter­po­la­tion. The values in the branches map are in turn also closures, where arg is bound to "--dev" or "--prod".

The branches map is passed to Pipeline’s parallel command, which causes the two run_tests closures to be executed on two different ex­ecu­tors—even­tu­al­ly.

stage("Deploy to Dev") { deploy "dev" }
stage("Dev Tests") { acceptance_integration_tests "dev" }

stage("Deploy to Prod") { deploy "prod" }
stage("Prod Tests") { acceptance_integration_tests "prod" }

The ac­cep­tance_in­te­gra­tion_tests closure is used in two different stages. Each stage is passed an anonymous closure, which invokes the parallel tests at a suitable time.

Not So Groovy

Fiction: Jenkins Pipeline scripts are written in Groovy.

Fact: Pipeline scripts are written in a sandboxed subset of Groovy and you need to be aware of the dif­fer­ences.

The Pipeline plugin presents the illusion that a Groovy program is executing on your agent node. It’s more com­pli­cat­ed than that. The Pipeline script compiles the script into a series of steps on the master node, where they are executed in a flyweight executor. Many steps actually result in the cor­re­spond­ing code being executed in an agent node. These steps execute in a restricted sandbox. Many steps are asyn­chro­nous and the complete state of the build is persisted each time such a step is executed. Jenkins may be restarted during a build and will resume executing. A lot of this magic is achieved through rewriting the Groovy program using a con­tin­u­a­tion-passing style trans­for­ma­tion.

The com­bi­na­tion of the sandbox and se­ri­al­iza­tion imposes re­stric­tions on the Groovy code that you can write. Many Java and Groovy classes and methods are not al­lowlist­ed and cannot be executed at all. Some may be used only in restricted cir­cum­stances. Es­sen­tial­ly, you’re limited to strings, numbers, and lists and maps built out of strings and numbers.

You can work around this to some extent by decorating a Groovy function with @NonCPS and operating on otherwise forbidden objects as local variables. You must, however, return se­ri­al­iz­able objects from such a function.

If you control your Jenkins master, you can allowlist JVM classes that you may need. CloudBees hosts our Jenkins builds, so this was not available to us.

I spent a ridiculous amount of time trying to get this apparently straight­for­ward code to work:

def label = compute_build_label(
    new Date(), env.GIT_LOCAL_BRANCH, env.BUILD_NUMBER, env.GIT_COMMIT)

def compute_build_label(
    Date build_date, String branch_name, Integer build_number, String git_sha)
{
    String date = build_date.format("yyyyMMdd't'HHmm'z'", TimeZone.getTimeZone('UTC'))
    String branch = branch_name.replaceAll("/", "-")
    String number = String.format("%05d", build_number)
    String sha = git_sha.substring(0, 7)
    return "${date}-${branch}-b${number}-${sha}"
}

Here are just some of the reasons why this didn’t work:

• org.jenkinsci.plugins.script­se­cu­ri­ty.sandbox.Re­jectedAc­ces­sEx­cep­tion: Scripts not permitted to use new java.util.Date: you can’t use new Date(). I ended up shelling out to the system date command.
• java.lang.No­Such­Meth­od­Er­ror: No such DSL method 'com­pute_build_la­bel' found among steps […]: This very unhelpful error means that the function signature didn’t match its use, as env.BUILD_NUM­BER is a String, not an Integer.
• org.jenkinsci.plugins.script­se­cu­ri­ty.sandbox.Re­jectedAc­ces­sEx­cep­tion: Scripts not permitted to use sta­t­icMethod java.lang.String format: I was attempting to use String.format("%05s", build_num­ber).
• I switched to "0000${build_num­ber}"[-5..-1] to take the last five digits and got org.jenkinsci.plugins.script­se­cu­ri­ty.sandbox.Re­jectedAc­ces­sEx­cep­tion: Scripts not permitted to use sta­t­icMethod org.codehaus.groovy.runtime.Script­Byte­codeAdapter cre­at­eRange.
• Despite my reading of GitSCM.java, env.GIT_LO­CAL_BRANCH and env.GIT_COMMIT did not contain useful values.

I ended up with this:

String label = compute_build_label(
    sh(script: 'date --utc +%Y%m%dt%H%Mz', returnStdout: true).trim(),
    GIT_BRANCH,
    env.BUILD_NUMBER,
    sh(script: 'git rev-parse HEAD', returnStdout: true).trim()
)

String compute_build_label(
    String date, String branch_name, String build_number, String git_sha)
{
    String branch = branch_name.replaceAll("/", "-")
    String number = "00000".substring(0, 5 - build_number.length()) + build_number
    String sha = git_sha.substring(0, 7)
    return "${date}-${branch}-b${number}-${sha}"
}

I would have been better off writing an external script.

Line Numbers

If you see a callstack like this:

…
at WorkflowScript.run(WorkflowScript:39)
at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:48)
…

Look in your Pipeline Script at the line number specified in the Work­flowScript entry (here, line 39).

Install Groovy

For debugging some Groovy syntax issues, it can be much faster to try them out in a local Groovy script.

I installed Groovy using Brew on my Mac. I found that I had to set the $JAVA_HOME and $GROOVY_HOME en­vi­ron­ment variables before it worked properly for me.

• export JAVA_HOME="$(/usr/libexec/java_home)"
• export GROOVY_HOME="$(brew --prefix)/opt/groovy/libexec"