Secure CI/CD Pipelines: Best Practices for Managing CI/CD Secrets

For DevSecOps, there’s always a balancing act between the fast pace of development velocity and security. Developers want to move fast, but if you move too fast, there’s the risk of a window being left open for an attacker somewhere in your deployed applications or infrastructure. And with the growing number of automation tools available for DevOps teams and the explosion of machine identities that go hand-in-hand with automating development tasks, it can be hard to keep up with security tasks.

Automated continuous integration/continuous delivery (CI/CD) pipelines using tools like Jenkins, Concourse and CircleCI let you create highly configurable build, test and deployment systems. And you’re not limited to build and test: most CI/CD tools give you the ability to plug in functionality from other tools like performance profilers and error monitoring.

Taking DevOps a step further into operations and Infrastructure as Code (IaC) practices, many teams use CI/CD pipelines to manage configuration files for building out the infrastructure their applications run on. This means CI/CD tools are also handling change management, testing and deployment for Docker files, Kubernetes manifests, Helm charts and more.

While most discussion of CI/CD focuses on application code — avoiding bugs and crashes — the shift to managing IaC configuration with CI/CD tools highlights another potential problem: managing the secrets needed to run your applications and systems, secrets that can end up being exposed and putting your systems at risk. 

Automating build, test and deployment tasks takes much of the need for storing and keeping track of secrets off the shoulders of your developers and administrators — which is a valuable first step — but moves responsibility for those secrets into your CI/CD pipeline and associated tools.

In this article, we’ll look at some background concepts and best practices for making sure authentication and authorization secrets are used effectively and securely in CI/CD pipelines.

Identity, Authentication and Authorization

What do we mean by secrets? Secrets are digital credentials that are used to provide identity authentication and authorize access to privileged accounts, applications and services.

For example, users authenticate to have access to a source control system like GitHub to commit changes to a codebase, which kicks off test, build and deployment tasks in a CI/CD pipeline.

Here are some other examples:

• User or auto-generated passwords
• API, GitHub tokens and other application keys/credentials
• Hard-coded credentials in containerized applications 
• SSH keys
• Private certificates for secure communication, transmitting and receiving of data (TLS, SSL and so on)
• Private encryption keys for systems like PGP
• System-to-system passwords
• One-time password devices

Developers need to understand what authentication secrets they need to control, where they are necessary in the CI/CD pipeline and how they are configured, stored and managed within the pipeline and deployed infrastructure.

The Challenge of Secrets in a CI/CD Pipeline

Automated processes are a critical component of DevOps infrastructure. CI/CD orchestration and configuration tools such as Jenkins, Puppet, Ansible and Chef are increasingly deployed in DevOps processes to improve processes, facilitate faster deployment of software and product delivery and provide continuous cost reduction. 

However, CI/CD tools are the biggest consumers of secrets and have access to a lot of sensitive resources such as other apps and services and information like codebases and databases. As the number of secrets grows, it becomes harder to store, transmit and audit secrets securely.

Furthermore, secrets aren’t just for authentication between tools. Often, secrets need to be provided as part of the build and deployment process so that deployed resources have access. This is particularly important in hybrid cloud and microservices deployments, as well as with the automated scaling capabilities of tools like Kubernetes.

Compromised secrets mean someone could make unwanted changes or leak information. Furthermore, these secrets may be insecurely hard-coded or store insecure configuration files, which jeopardizes security for the entire automation process.

While tools, platforms, and infrastructure may have their own built-in capability to secure secrets, these may lack interoperability, leading to siloed security efforts, making it difficult to track, manage, and audit security effectiveness.

In the past few years, there have also been some high-profile breaches that occurred due to compromised CI/CD pipelines. SolarWinds, Codecov, Log4j, Uber and CircleCI – these are all examples of recent high-impact software supply chain attacks. With heightened scrutiny on the security of organizations’ software supply chains, it’s never been more important to protect the secrets within your CI/CD pipelines. For example, in the CircleCI breach, the code integration tool (the pipeline, essentially) was compromised, leading to the exposure of all the secrets and credentials that had access to other pieces of the software supply chain. You can see how severe these types of breaches can get when one part of the pipeline is compromised, as these tools have far-reaching and highly privileged access throughout your organization. And it’s really not a matter of if an attack will occur; it’s when. That’s why it’s critical to take additional steps to protect your secrets to mitigate that risk.

Common Risks to Secrets

Tools such as Jenkins interface with other systems and applications throughout DevOps environments, and there is the danger of exposed secrets in the clear in CI/CD config files. 

Let’s look at Jenkins as an example. Jenkins’ configurability and use of plugins make it challenging to securely determine who can use Jenkins at any one time, what Jenkins is allowed to do and where Jenkins can deploy its artifacts.

Jenkins communicates with a plethora of applications and systems across DevOps environments. Secrets can end up in configuration details within Jenkinsfiles, but Jenkins also includes a central credentials store to manage the credentials needed for each pipeline. Jenkins jobs and plugins can access these credentials at runtime, and the server obscures their usage in the output of each job. 

This creates the challenge of securing access to critical secrets such as passwords, source control and artifact deployment. This causes a security risk if developers leave Jenkins consoles in an insecure state within development environments. 

The Jenkins scripting console can be used to run arbitrary Groovy code and can be found under Manage Jenkins > Script Console or via /script from the root of the Jenkins install path. 

If developers store passwords within Jenkins, they may not be visible from within the web console, but are able to be extracted from the system itself.  

Any Jenkins users with Job/Configure permissions have the opportunity to run any executable on the Jenkins agents and request any set of credentials defined in their scope (global or otherwise) to be injected into agents using the credentials binding plugin.

The credentials binding plugin enables application secrets to be passed from job configurator users to the build agent. For example, users can add a withCredentials statement to their Groovy pipeline script with the credentials ID.

An example pipeline using withCredentials might be:

node {
  withCredentials( [usernamePassword( credentialsId: 'googlecompute', 
                                      usernameVariable: 'USERNAME', 
                                      passwordVariable: 'PASSWORD')]) {
        // build project
        sh 'mvn clean install'
    }
}

Here, the pipeline job instructs Jenkins to grab the Google Cloud credentials by using the “amazon” credentials ID, injecting the Google Cloud username and password into the process as variables USERNAME and PASSWORD and running a maven build. 

This exposes all credentials in the global scope to any pipeline and any job configurator user. Thus, an attacker can shift from low privilege to highly privileged with access to AWS secret keys, passwords and git credentials. 

These risks are not exclusive to Jenkins. Any system that allows user-defined build processes and the use of credentials in build processes faces the same potential issue of allowing the exposure of credentials. We need to protect and be confident in the artifacts produced from the CI/CD. 

Best Practices for CI/CD Security

The first steps for securing your team’s CI/CD pipeline include locking down configuration managers, systems that host repositories and the build servers. The pipeline should be monitored from end to end with watertight access control across the entire toolchain. Scripted builds need to be scanned for vulnerabilities, and source code needs to be regularly monitored for vulnerabilities prior to app deployment to production.

The security of secrets needs to apply both during transit and at rest. Best practices include the following:

• Remove hard-coded secrets from Jenkinsfiles and related CI/CD config files.
• Have rigorous security parameters, such as one-time passwords, for secrets regarding more sensitive tools and systems.
• Distribute secrets among Jenkinsfiles to reduce the potential attack target of each file.
• Use password managers, and rotate passwords after each use.
• Know who has access to what. Whether it’s role-based, time-based, or task-based, there should be a clear repository of access management. Another option to consider is segmenting secrets based on levels of access. 
• Machine identity is critical to secure non-human access in containers. Typically, an authenticator certifies that the client run-time container attributes of the requesting container match the native characteristics of the valid container. Once authenticated, the container can access multiple resources based on predefined role-based access control policies. You should destroy containers and virtual machines (VMs) after use.
• Ensure secrets are not inadvertently passed on during builds for pull requests via your CI/CD pipelines.
• Deploy the practice of least privilege. Give access only to requisite secrets. Besides employee access, least privilege also applies to applications, systems or connected devices that require privileges or permissions to perform tasks. You should regularly audit levels of access to maintain the level of least privilege. 

How CyberArk Helps Solve the Challenge of Exposed Secrets in Jenkinsfiles 

Fortunately, CyberArk provides a way to keep secrets out of your Jenkins master, off disk and out of source control. CyberArk has provided a Jenkins plugin that you can use to provide credentials to your Jenkins jobs at runtime. The plugin securely provides credentials that are stored in Conjur to Jenkins jobs. Take a look at our GitHub documentation and the article CI/CD Servers Know All Your Plumbing Secrets to learn how to set up the CyberArk’s Jenkins plugin. 

Securing Your CI/CD Pipeline Next Steps

Security needs to be a top priority in any developer team, especially when considering the heightened scrutiny on software supply chains these days. By following these best security practices you can ensure that digital authentication credentials are resistant to attacks by malicious agents and application identities remain secure.

Get started today in setting up a Conjur OSS environment and retrieving a secret from Conjur to your application. Learn how to keep secrets out of your Jenkins master, off disk and out of source control.

Be sure to check out our new hosted interactive tutorials for securing CI/CD pipelines, securing Ansible automation, and securing Kubernetes secrets, and take a look at the Conjur tutorials and blog to learn more. You can also sign up to the newsletter, which highlights interesting open-source community news, informative tutorials you can use, and the latest Conjur product news. Join the CyberArk commons community to connect with others to discuss security.