Managing Testing Secrets in Jenkins Pipelines

In the first and second articles of this series, we discussed managing the information needed to build and test applications. We described the requirements for a solution and discussed why CyberArk’s Conjur is an excellent option. This article demonstrates a Conjur-based solution by presenting a way to use secrets stored in Conjur in a Jenkins scripted pipeline and a Jenkins freestyle project.

Integration tests usually have a long life. It’s very common to use the test suite after development is complete as a diagnostic tool and for regression testing during maintenance. For these reasons, integration tests usually run using a framework that produces results that other tools archive and report.

This solution uses the Bash Automated Testing System because Bash test scripts can run almost any code that needs to be tested, regardless of the language used to develop it. Jenkins has a plugin that we’ll integrate, called TAP, that can read the BATS reports and include that information on the Dashboard. The result is that we’ll have a secure framework that produces results like this:

To follow along with this tutorial, you’ll need to have implemented the solution described in the second article of this series. Additionally, you’ll need to:

  • Install BATS
  • Install the TAP plugin in Jenkins

Integrating Conjur

The test script uses two secrets: the hostname of the Tomcat server hosting our gateway sample application, and the API key needed by the OpenWeatherMap service. Both are available in Conjur.

Jenkins makes secrets available to pipelines and projects using environment variables. Our scripts use bindings to tell Jenkins that these environment variables are proxies for data held in Conjur. In the second article in this series, we created a set of Jenkins credentials that tell Jenkins how to fetch a Conjur secret. All our scripts need to do is tell Jenkins which Conjur secret we want to make the association between a credential and an environment variable.

In our example pipeline scripts, we create bindings with the withCredentials construct:

withCredentials([
     conjurSecretCredential(credentialsId: 'AWS_WEBSERVER_HOSTNAME',
                               variable: 'AWS_WEBSERVER_HOSTNAME'),
     conjurSecretCredential(credentialsId: 'OPENWEATHER_API_KEY',
                               variable: 'OPENWEATHER_API_KEY')
                ])

We are purposefully matching the environment variable name to the Jenkins Credential ID so that the data source is clear. Freestyle projects provide a binding block for this information. Our example creates these bindings.

Although we only need two bindings, we can create any number of bindings. This way, every secret we need can be stored and easily referenced in Conjur.

Now that we have the bindings set up, we’re all set to start developing tests.

Integration Tests

This article demonstrates how to pass Conjur secrets to integration tests, so we’ve kept our example very simple. The OpenWeather Gateway application provides a single API that obtains the current weather data for a location. This means we only need a single test to verify that the gateway works. We can expand the pattern to support any number of tests in a suite of test scripts using all the secrets we’ve stored in Conjur.

BATS and Bash Scripts

A BATS test script is a collection of Bash functions where each function ends with a test statement. Our test script is below. We’ve created the url variable to highlight how we use the two environment variables.

#!/usr/bin/env bats

url="http://$hostname:8080/OpenWeatherGateway/Weather?city=Birdsboro,PA,USA&api=
$apikey"

@test "Get Local Weather" {
  if curl $url | grep -q 'Weather @ Birdsboro,PA,USA'; then
    result=0
  else
    result=1
  fi

  [ "$result" -eq 0 ]
}

We can add any number of tests to a BATS script. The TAP plugin records the results of each test, and the script passes if all tests pass.

In the scripted pipeline, we can execute the BATS script directly. But, in the freestyle project, we must run an intermediate script.

#!/bin/bash

exec &> ./batsScript.results

hostname=$1 apikey=$2 bats /var/lib/jenkins/openWeatherScripts.bats

The syntax of this script is essential. The variable assignments must be on the same line as the bats command to copy the Conjur secrets to the environment variables that the BATS test uses. We see this same line of script in the scripted pipeline.

Scripted Pipeline Test

Now that we have a test script, let’s look at putting this all together in a scripted pipeline. We created a new Jenkins pipeline, added a description, and then provided this pipeline script:

timestamps

{
      node ()

{
stage('Integration Testing')
{
                  withCredentials([
                        conjurSecretCredential(
                               credentialsId: 'AWS_WEBSERVER_HOSTNAME',
                               variable: 'AWS_WEBSERVER_HOSTNAME'),
                        conjurSecretCredential(
                               credentialsId: 'OPENWEATHER_API_KEY',
                               variable: 'OPENWEATHER_API_KEY')
                         ])
                  {
                        sh 'hostname=$AWS_WEBSERVER_HOSTNAME \
                              apikey=$OPENWEATHER_API_KEY \
                              bats /var/lib/jenkins/openWeatherScripts.bats'

                        step([$class: "TapPublisher", testResults: "**/*.results", outputTapToConsole: true, includeCommentDiagnostics: true ])
                  }
}
}
}

The withCrdentials block binds the Conjur secrets to pipeline environment variables. We used this construct instead of the Jenkins environment construct to tie these credentials to this stage and make each stage a “stand-alone” test block. The sh command copies the Conjur secrets to the BATS environment variables that the BATS script uses. The step command causes TAP to publish the BATS reports to Jenkins.

That’s it. If we need more tests, we can extend the BATS script. If we partitioned the tests into multiple scripts, we could add multiple stages.

The previous article shows all the steps needed to create a scripted pipeline. For this article, we just show to details of the script we created. Here’s where we set the description:

Here’s how we set the pipeline script:

We’ve accepted the defaults for all other settings.

Freestyle Project Test

Now that we’ve covered the details, let’s look at how we can implement the same process in a freestyle project. This project only implements the integration tests, so we’ve left the Source Code Management and Build Triggers portions empty.

Here’s the description:

For Source Code Management select None.

Here are the bindings:

These are equivalent to an environment section in a scripted pipeline. The withCredentials section in our example scripted pipeline provides a similar function with a limited scope. If we review the credential definitions, we’ll notice that we’re referring to the same credentials that we bound in the scripted pipeline.

Here are the Build and Post Build Actions sections:

The script is:

/var/lib/jenkins/openWeatherScript.sh $AWS_WEBSERVER_HOSTNAME $OPENWEATHER_API_KEY

We discussed openWeatherScript.sh previously:

#!/bin/bash

hostname=$1 apikey=$2 bats /var/lib/jenkins/openWeatherScripts.bats

This shows how to map the $AWS_WEBSERVER_HOSTNAME to the hostname BATS script environment variable, and how to map $OPENWEATHER_API_KEY to the apikey BATS script environment variable.

Summary

That’s it. We now have a working integration testing framework that uses secrets without exposing them and provides role-based access control using Conjur policies to ensure that only the processes and people that should have access to them do have access. The Jenkins binding mechanism makes it easy to use secrets. Just map an environment variable to a Jenkins credential, and Jenkins takes care of the rest.

To learn more about Conjur, check out Cyber Ark’s documentation at Securing Secrets Across the CI/CD Pipeline.