How Blackbox Works

Blackbox builds on top of GNU Privacy Guard (aka GnuPG aka GPG) to automate the secure management of a set of files containing secrets that are “encrypted at rest” and stored in a Version Control System (VCS), such as Git. These registered files are owned col­lec­tive­ly by a set of ad­min­is­tra­tors, each of whom has their own separate keypair (a public key and a private key) stored in their own keyrings. The ad­min­is­tra­tors’ public keys are also present in Blackbox’s keyring, which is stored in the VCS. Using Blackbox’s commands, any ad­min­is­tra­tor can decrypt a file containing secrets, update the secrets in the file, encrypt the updated secrets file, and commit that encrypted file into the VCS. Ad­min­is­tra­tors can be removed from a Blackbox in­stal­la­tion, after which they will not be able to decrypt the updated secrets files[1].

How does Blackbox encrypt a file so that any ad­min­is­tra­tor can decrypt it? It uses GPG to encrypt the file for multiple recipients, say, Alice, Bob, and Carol.

When GPG encrypts a file, it:

• creates a random session key for symmetric encryption
• writes a header for each recipient, containing:
  • the ID of the recipient’s public key
  • the result of asym­met­ri­cal­ly encrypting the session key with the recipient’s public key
• possibly signs the data
• compresses the (signed) data
• sym­met­ri­cal­ly encrypts the compressed data with the session key
• writes the encrypted, compressed data

Only the recipients have the private keys (in theory, at least). Therefore, only a recipient can decrypt the encrypted file.

To decrypt the file for a recipient, GPG:

• finds the encrypted session key packet whose keyID matches the recipient’s public key
• decrypts the session key using the recipient’s private key
• decrypts the encrypted, compressed data using the session key
• de­com­press­es the decrypted data
• verifies the signature, if present
• writes the cleartext

This is a hybrid scheme. Symmetric encryption is a lot faster and more compact than public key/private key asymmetric encryption, so it’s used to encrypt the actual data. Fur­ther­more, if the data were entirely encrypted with a recipient’s public key, then encrypting for N recipients would mean that the size of the result would be pro­por­tion­al to the (number of recipients) × (the length of the original data). With the hybrid scheme, the header grows a few hundred bytes for each recipient but the data is encrypted only once, with faster encryption.

Blackbox encrypts a registered file with all of the ad­min­is­tra­tors as the recipients, so any ad­min­is­tra­tor can decrypt the file.

Typical PGP Message

(Figure from Network Security: Email Security, PKI, Tuomas Aura)

You can use gpg --list-packets to dump the contents of any GPG message. An Advanced Intro to GnuPG dives into the message format in more detail.

Going back to my original example, cre­den­tials.sh is a file registered in blackbox-files.txt. This file should never be committed to the VCS—add it to gitignore to prevent ac­ci­den­tal­ly committing it. Instead, cre­den­tials.sh.gpg is committed. Since the latter is a binary file, comparing two versions in cleartext is tricky.

Private Keys and Paperkey

Ad­min­is­tra­tors can encrypt and decrypt Blackbox’d files because they have their private key on a local keyring.

Getting a private key onto other hosts can be tricky. We developed this technique when we were using Atlassian’s hosted Bamboo CI service. We later used it with hosted Jenkins at Cloudbees. Because we were using a hosted Continuous In­te­gra­tion (CI) service, we had limited control over what we could install. If I remember correctly, Bamboo had support for secret en­vi­ron­ment variables, but did not provide a way to store a keyring file. There was also a limit on the length of the en­vi­ron­ment variables, I believe.

We were able to get past this by using Paperkey to (de)serialize the secret key. Paperkey can extract just the secret part of a secret key: ‘Due to metadata and redundancy, OpenPGP secret keys are sig­nif­i­cant­ly larger than just the "secret bits". In fact, the secret key contains a complete copy of the public key.’

We created a keypair for the CI on a secure host, serialized the secret with Paperkey, and pasted the secret into the CI’s UI to become an en­vi­ron­ment variable. At build time, we used Paperkey on the CI box to de­se­ri­al­ize the secret key from the en­vi­ron­ment variable, before decrypting the secrets needed with Blackbox.

To create the CI keypair, follow the portion of the Blackbox "role accounts" in­struc­tions that create a sub-key with no password for ci@example.com.

Then, serialize the public key and the secret with Paperkey:

cd /tmp/NEWMASTER
gpg --homedir . --export ci@example.com \
    | base64 > public_key.txt
gpg --homedir . --export-secret-keys ci@example.com \
    | paperkey --output-type=raw \
    | base64 > secret.txt

Copy and paste the contents of public_key.txt to the GPG_PUB­LIC_KEY en­vi­ron­ment variable in the CI. Similarly, copy secret.txt to GPG_SECRET.

Securely delete everything in /tmp/NEWMASTER.

We used a script like this on the CI to re­con­sti­tute the keypair and to decrypt the other secrets from Blackbox:

#!/usr/bin/env bash

# Run during a CI build to decrypt all Blackbox-encrypted files in this repo.
# Can also be used interactively.

set -ex

# Root of Git working tree
SERVICES_DIR="$(cd "$(dirname "$0")/.."; pwd)"

if [ "$CI_BUILD" = "true" ]; then
    GPG_HOMEDIR="$(mktemp -d -t gnupg.XXX)"
    SECRET_KEY_FILE="$GPG_HOMEDIR/secret.key"
    PUBLIC_KEY_FILE="$GPG_HOMEDIR/public_key.gpg"

    # this variable is how you can customize how GPG is used in Blackbox
    GPG="gpg --homedir=$GPG_HOMEDIR"

    # Remove secrets from filesystem on exit.
    function clean_up {
        # TODO: use shred, if available
        rm -rf "$GPG_HOMEDIR"
    }
    trap clean_up EXIT;

    echo "Unpacking keys; exiting debug mode to redact..."
    set +x

    if [ -z "$GPG_PUBLIC_KEY" -o -z "$GPG_SECRET" ]; then
        echo "Missing CI credential env vars for GPG key and secret"
        exit 1
    fi

    # unpack public key
    echo "$GPG_PUBLIC_KEY" | base64 --decode > "$PUBLIC_KEY_FILE"

    # unpack secret key
    echo "$GPG_SECRET" | base64 --decode > "$SECRET_KEY_FILE"

    echo "Secrets unpacked..."
    set -x

    # reconstitute and import full key into $GPG_HOMEDIR
    paperkey --pubring "$PUBLIC_KEY_FILE" --secrets "$SECRET_KEY_FILE" \
        | $GPG --import

    # TODO: vendor Blackbox
    BLACKBOX_DIR="$(mktemp -d -t blackbox.XXX)"
    BLACKBOX_BIN=$BLACKBOX_DIR/bin

    # Shallow clone of Blackbox with most-recent commit only
    git clone --depth 1 https://github.com/StackExchange/blackbox.git $BLACKBOX_DIR
else
    # So that you only have to enter your password once when running interactively
    eval "$(gpg-agent --daemon)"

    # No custom GPG_HOMEDIR needed
    GPG="gpg"

    BLACKBOX_POSTDEPLOY="$(command -v blackbox_postdeploy)" || ret=$?
    if [ -n "$BLACKBOX_POSTDEPLOY" ]; then
        # Use the Blackbox that's on the path
        BLACKBOX_BIN="$(dirname $BLACKBOX_POSTDEPLOY)"
    else
        # Assume Blackbox is checked out in a sibling dir to $SERVICES_DIR
        BLACKBOX_BIN="$(cd "$SERVICES_DIR/.."; pwd)"/blackbox/bin
        if [ ! -f "$BLACKBOX_BIN/blackbox_postdeploy" ]; then
            echo "Can't find Blackbox binaries"
            exit 1
        fi
    fi
fi

# decrypt secrets in $SERVICES_DIR using custom GPG_HOMEDIR
GPG="$GPG" $BLACKBOX_BIN/blackbox_postdeploy

# test that decryption worked
grep 'congrats!' test_secret.txt

At the end of the build, run black­box_shred_al­l_­files to destroy any decrypted files.

More Reading

• Blackbox
• Paperkey
• Protect your documents with GPG
• Anatomy of a GPG Key
• Creating the perfect GPG keypair
• How GPG works: Encrypt
• GPG import and export
• An Advanced Intro to GnuPG
• Network Security: Email Security, PKI