The Secure Task Runner

Even if you’ve never heard of Jenkins, you might be benefiting from it already: many companies and open source software projects use Jenkins to automatically build, test, and distribute their software. This distinguished group includes Mozilla, the Apache Software Foundation, Creative Commons, the Eclipse Foundation, and open source projects such as AngularJS and coreboot.

The Conjur team uses Jenkins too, and we’ve developed some specific techniques we’d like to share with you. These will be useful when you’re starting out with Jenkins or re-architecting your server for greater security and developer happiness.

(If you’re using Jenkins already, don’t miss our previous post, “Untangling Jenkins: Lessons learned radically overhauling Jenkins for developer and operator happiness”. It’s got a great, approachable high-level overview of the processes that motivated the techniques described here.)

Typical security issues with Jenkins

Privileges: your typical Jenkins server is highly privileged. It has lots of secret keys and passwords which allow it to perform all its duties. This sets up hackers for a sweet score if they can get access to the Jenkins master. Those secrets are the keys to privileged accounts that can control your infrastructure and allow access to all your data.

Unsecured secrets: often, sensitive secrets such as SSH private keys and service API keys are stored in plaintext on the Jenkins master and executors, waiting for somebody to find them in a moment of weakness. Furthermore, this makes those secrets difficult to see, rotate, and audit.

Unauthenticated access: sometimes, Jenkins executors are authenticated only during provisioning, when their secrets are delivered and software is installed. Further use of those secrets is unsupervised and does not trigger re-authentication, which makes it impractical to control or audit access per-machine.

Unauthorized access: not every job needs to be authorized to access every secret. But when all your secrets are frozen into the executor at provisioning time, the concept of authorization mostly goes out the window.

Enter The Secure Task Runner

We have a practical approach to these issues that allows you to get the benefits of Jenkins task automation without compromising on security. It is described in total here: Securing Jenkins

Here are the steps in brief:

1. Use executor machines to run your tasks. Don’t store secrets in Global Credentials or run your tasks on the Jenkins master.
2. Store your secrets in a secure, highly available vault like Conjur.
3. Configure your Jenkins executors to fetch secrets on-demand as you need them for jobs and forget them as soon as the jobs are done. Treat ephemeral access to secrets as a precious resource, to be used sparingly and released as soon as possible, like CPU cores or memory.
4. Create and enforce policies about which executors can access which secrets for which tasks, and always fetch the smallest number of secrets at a time to complete the current task. (Conjur helps you do this automatically.)

Storing secrets for Jenkins tasks in Conjur

In order to store secrets in Conjur, you need to create a policy that defines variables for them to reside in. This is like defining the schema of a database.

Here’s an example policy using Conjur’s Machine Authorization Markup Language (MAML)

# one-secret.yml
- !policy
  id: example-policy
  body:
    - !variable example-api-key

To load the policy into Conjur and store a secret:

1. Do the Conjur Quick Start tutorial.
2. Start up a conjurinc/cli5 container in Docker
3. Copy (docker cp) the policy file (one-secret.yml above) into the container
4. Use the Conjur CLI to authenticate to your server (conjur init and conjur
authn login) and load the policy (conjur policy load)
5. Generate a hypothetical API key (perhaps using uuidgen)
6. Store it (conjur variable values add example-policy/example-api-key $myAPIkey)

Getting secrets back out when needed

When running a Jenkins task, split it into a pipeline composed of scripts. For example, suppose your task builds some software in a container, tests it, and pushes the container to DockerHub on a success. Your Jenkinsfile might look like this:

pipeline {
    agent { label 'executor' }

    stages {
        stage('Build container image') {
            steps {
                sh './build.sh'
            }
        }

        stage('Test container') {
            steps {
                sh './test.sh'
            }
        }

        stage('Push container to DockerHub') {
            steps {
                sh './push.sh'
            }
        }
    }
}

Now those scripts can fetch secrets as needed and hold on to them for just a short amount of time. We built Summon as a command line tool to make this convenient.

Summon lets you describe what secrets you need using a secrets manifest and fetch them on demand, placing them into environment variables or memory-mapped files while an inferior process runs.

Here’s an example secrets manifest:

# secrets.yml
API_KEY: !var example-policy/example-api-key
API_KEY_FILE: !var:file example-policy/example-api-key

Then we can run summon -f /path/to/secrets.yml bash to get a shell with access to the API key. In this case, it would be available in the environment variable called API_KEY and also in a temporary memory-mapped file, the name of which is stored in the API_KEY_FILE environment variable.

When your bash session exits (or whatever other inferior process you might launch using Summon, such as a Docker command, finishes) then those secrets will immediately be discarded. This happens without any orchestration by Jenkins, reducing the complexity of your infrastructure.

Conjur and Jenkins together

Jenkins lets you define tasks using code and automatically run them on demand. Conjur is a great fit because it lets you define security policy using code and automatically rotate and fetch secrets on demand. With these powers combined, you can build a task running platform that is secure by default following the principle of least privilege.

For more information and examples, read through the full Conjur + Jenkins tutorial on Conjur.org.