George V. Reilly

Decrypting Blackbox secrets at build time with Paperkey

“Security is 1% technology plus 99% following the procedures correctly” — Tom Limoncelli

Having dealt with GPG last week at work, I remembered that I had intended to write a blog post about how we used GPG, Blackbox, and Paperkey to store secrets in Git at my previous job.

We used Blackbox to manage secrets that were needed during de­vel­op­ment, build, deployment, and runtime. These secrets included AWS cre­den­tials, Docker registry cre­den­tials, our private PyPI cre­den­tials, database cre­den­tials, and cer­tifi­cates. We wanted these secrets to be under version control, but also to be secure.

For example, we had a cre­den­ that exported en­vi­ron­ment variables, which was managed by Blackbox:

# Save current value of xtrace option from $-; disable echoing of executed commands
{ if echo $- | grep -q "x"; then XT="-x"; else XT="+x"; fi; set +x; } 2>/dev/null

# export environment variables, many containing secrets
export AWS_ACCESS_KEY_ID='...'

export PYPI_USER='build'
export PYPI_PASSWD='...'
export PIP_INDEX_URL="https://$PYPI_USER:$"

# Restore previous value of xtrace option
set $XT

The XT prologue ensures that even if this script is source’d with set -x (debug tracing) enabled, that executing this script will not leak secrets into build logs. The epilogue turns the xtrace option back on again if it was on at the start.

We used LastPass to manage personal cre­den­tials that were needed in a browser, but it wasn’t suitable for automated use in CI.

How Blackbox Works

Blackbox builds on top of GNU Privacy Guard (aka GnuPG aka GPG) to automate the secure management of a set of files containing secrets that are “encrypted at rest” and stored in a Version Control System (VCS), such as Git. These registered files are owned col­lec­tive­ly by a set of ad­min­is­tra­tors, each of whom has their own separate keypair (a public key and a private key) stored in their own keyrings. The ad­min­is­tra­tors’ public keys are also present in Blackbox’s keyring, which is stored in the VCS. Using Blackbox’s commands, any ad­min­is­tra­tor can decrypt a file containing secrets, update the secrets in the file, encrypt the updated secrets file, and commit that encrypted file into the VCS. Ad­min­is­tra­tors can be removed from a Blackbox in­stal­la­tion, after which they will not be able to decrypt the updated secrets files[1].

How does Blackbox encrypt a file so that any ad­min­is­tra­tor can decrypt it? It uses GPG to encrypt the file for multiple recipients, say, Alice, Bob, and Carol.

When GPG encrypts a file, it:

Only the recipients have the private keys (in theory, at least). Therefore, only a recipient can decrypt the encrypted file.

To decrypt the file for a recipient, GPG:

This is a hybrid scheme. Symmetric encryption is a lot faster and more compact than public key/private key asymmetric encryption, so it’s used to encrypt the actual data. Fur­ther­more, if the data were entirely encrypted with a recipient’s public key, then encrypting for N recipients would mean that the size of the result would be pro­por­tion­al to the (number of recipients) × (the length of the original data). With the hybrid scheme, the header grows a few hundred bytes for each recipient but the data is encrypted only once, with faster encryption.

Blackbox encrypts a registered file with all of the ad­min­is­tra­tors as the recipients, so any ad­min­is­tra­tor can decrypt the file.

Typical PGP Message

Typical PGP Message

(Figure from Network Security: Email Security, PKI, Tuomas Aura)

You can use gpg --list-packets to dump the contents of any GPG message. An Advanced Intro to GnuPG dives into the message format in more detail.

Going back to my original example, cre­den­ is a file registered in blackbox-files.txt. This file should never be committed to the VCS—add it to gitignore to prevent ac­ci­den­tal­ly committing it. Instead, cre­den­ is committed. Since the latter is a binary file, comparing two versions in cleartext is tricky.

[1]If they have a snapshot of the VCS before their access was revoked, they will still be able to decrypt the secrets as they were then. In principle, you should be changing passwords and cer­tifi­cates every time someone’s access is revoked.

Private Keys and Paperkey

Ad­min­is­tra­tors can encrypt and decrypt Blackbox’d files because they have their private key on a local keyring.

Getting a private key onto other hosts can be tricky. We developed this technique when we were using Atlassian’s hosted Bamboo CI service. We later used it with hosted Jenkins at Cloudbees. Because we were using a hosted Continuous In­te­gra­tion (CI) service, we had limited control over what we could install. If I remember correctly, Bamboo had support for secret en­vi­ron­ment variables, but did not provide a way to store a keyring file. There was also a limit on the length of the en­vi­ron­ment variables, I believe.

We were able to get past this by using Paperkey to (de)serialize the secret key. Paperkey can extract just the secret part of a secret key: ‘Due to metadata and redundancy, OpenPGP secret keys are sig­nif­i­cant­ly larger than just the "secret bits". In fact, the secret key contains a complete copy of the public key.’

We created a keypair for the CI on a secure host, serialized the secret with Paperkey, and pasted the secret into the CI’s UI to become an en­vi­ron­ment variable. At build time, we used Paperkey on the CI box to de­se­ri­al­ize the secret key from the en­vi­ron­ment variable, before decrypting the secrets needed with Blackbox.

To create the CI keypair, follow the portion of the Blackbox "role accounts" in­struc­tions that create a sub-key with no password for

Then, serialize the public key and the secret with Paperkey:

gpg --homedir . --export \
    | base64 > public_key.txt
gpg --homedir . --export-secret-keys \
    | paperkey --output-type=raw \
    | base64 > secret.txt

Copy and paste the contents of public_key.txt to the GPG_PUB­LIC_KEY en­vi­ron­ment variable in the CI. Similarly, copy secret.txt to GPG_SECRET.

Securely delete everything in /tmp/NEWMASTER.

We used a script like this on the CI to re­con­sti­tute the keypair and to decrypt the other secrets from Blackbox:

#!/usr/bin/env bash

# Run during a CI build to decrypt all Blackbox-encrypted files in this repo.
# Can also be used interactively.

set -ex

# Root of Git working tree
SERVICES_DIR="$(cd "$(dirname "$0")/.."; pwd)"

if [ "$CI_BUILD" = "true" ]; then
    GPG_HOMEDIR="$(mktemp -d -t gnupg.XXX)"

    # this variable is how you can customize how GPG is used in Blackbox
    GPG="gpg --homedir=$GPG_HOMEDIR"

    # Remove secrets from filesystem on exit.
    function clean_up {
        # TODO: use shred, if available
        rm -rf "$GPG_HOMEDIR"
    trap clean_up EXIT;

    echo "Unpacking keys; exiting debug mode to redact..."
    set +x

    if [ -z "$GPG_PUBLIC_KEY" -o -z "$GPG_SECRET" ]; then
        echo "Missing CI credential env vars for GPG key and secret"
        exit 1

    # unpack public key
    echo "$GPG_PUBLIC_KEY" | base64 --decode > "$PUBLIC_KEY_FILE"

    # unpack secret key
    echo "$GPG_SECRET" | base64 --decode > "$SECRET_KEY_FILE"

    echo "Secrets unpacked..."
    set -x

    # reconstitute and import full key into $GPG_HOMEDIR
    paperkey --pubring "$PUBLIC_KEY_FILE" --secrets "$SECRET_KEY_FILE" \
        | $GPG --import

    # TODO: vendor Blackbox
    BLACKBOX_DIR="$(mktemp -d -t blackbox.XXX)"

    # Shallow clone of Blackbox with most-recent commit only
    git clone --depth 1 $BLACKBOX_DIR
    # So that you only have to enter your password once when running interactively
    eval "$(gpg-agent --daemon)"

    # No custom GPG_HOMEDIR needed

    BLACKBOX_POSTDEPLOY="$(command -v blackbox_postdeploy)" || ret=$?
    if [ -n "$BLACKBOX_POSTDEPLOY" ]; then
        # Use the Blackbox that's on the path
        # Assume Blackbox is checked out in a sibling dir to $SERVICES_DIR
        BLACKBOX_BIN="$(cd "$SERVICES_DIR/.."; pwd)"/blackbox/bin
        if [ ! -f "$BLACKBOX_BIN/blackbox_postdeploy" ]; then
            echo "Can't find Blackbox binaries"
            exit 1

# decrypt secrets in $SERVICES_DIR using custom GPG_HOMEDIR
GPG="$GPG" $BLACKBOX_BIN/blackbox_postdeploy

# test that decryption worked
grep 'congrats!' test_secret.txt

At the end of the build, run black­box_shred_al­l_­files to destroy any decrypted files.

More Reading

blog comments powered by Disqus
Review: Fire and Blood » « A Use for Octal: Calculating Modulo 36 from Modulo 9