OpenShift Secrets Management

Securing OpenShift Secrets

Published February 22nd, 2021 by John Walsh

Almost every application needs to deal with secrets in one way or another, to authenticate with a backend database or an API, to get secure access to artifacts such as Docker images in a private registry, or for any number of other reasons. 

All container orchestration solutions let you manage such secrets in various ways. Red Hat OpenShift comes with robust, security-minded default settings, providing a more secure experience out of the box. In comparison with Kubernetes—on which OpenShift is built—the following settings and features in OpenShift provide this enhanced security:

  • By default, a container can’t be run as root.
  • Role-based access control (RBAC) is turned on by default.
  • The robust authentication and authorization model includes a built-in OAuth server, which can be connected to your organization’s Active Directory. 
  • You can also synchronize LDAP users for automated on- and offboarding. This ensures that former employees no longer have access to your OpenShift cluster.

It’s important to properly manage your secrets. Not doing so creates the risk of granting access to the wrong entities. Secrets can also be accidentally exposed through source code or logs. OpenShift’s security-minded default settings reduce these risks.

Secrets in OpenShift

OpenShift supports a number of different secret types to securely store sensitive data:

  • kubernetes.io/service-account-token uses a service account token.
  • kubernetes.io/dockercfg uses the dockercfg file for required Docker credentials.
  • kubernetes.io/dockerconfigjson uses the .docker/config.json file for required Docker credentials.
  • kubernetes.io/basic-auth is used with basic authentication.
  • kubernetes.io/ssh-auth Is used with SSH key authentication.
  • kubernetes.io/tls is used with TLS certificate authorities.

The advantage of these types is that it’s harder to accidentally enter incorrect data. For example, the kubernetes.io/basic-auth type will only accept secrets with the keys “username” and “password” or “token” set.

For any other type of secret, the opaque type, which is the default, can be used. This type doesn’t include any validation and can be used for any secrets with any key/value pairs that you want.

Secrets get created using the OpenShift CLI. For example, the following command creates a secret of type kubernetes.io/basic-auth that includes the key/value pair password=my-token:

oc create secret generic my-secret \

    –from-literal=password=my-token \

    –type=kubernetes.io/basic-auth

Instead of setting values directly on the command line, you can also take a value from the contents of a file. For example, the following command will put the contents of an SSH private key in an OpenShift secret of type kubernetes.io/ssh-auth:

oc create secret generic my-secret \

    –from-file=ssh-privatekey=./my-key \

    –type=kubernetes.io/ssh-auth

Keep in mind that, just as with Kubernetes, updating a secret won’t dynamically change the values in pods that are already using the secret. You’ll need to recreate the pods for the new secret value to take effect.

Making OpenShift Secrets More Robust

While OpenShift comes with a number of solid security policies set by default, there are steps you can take to make managing secrets even more robust and flexible. 

First, by default, OpenShift doesn’t encrypt secrets in its back end, which gives all cluster administrators access to all secrets. In addition, node administrators and anyone with shell access to pods can see all secrets used in a pod.

Moreover, secrets are used by applications and there’s always a risk the application will leak those secrets. Secrets set as environment variables can be exposed if the environment is printed in a debug log. 

Even with a sort of a secrets management solution, there could still be security issues if best practices aren’t followed related to handling the ultimate secret, secret zero.  Secret zero happens when a secret is needed to protect other secrets.

Best Practices for Secret Management

While the default secrets management options for OpenShift are a great start, keep in mind the following best practices while dealing with secrets:

  • Use the least-privilege principle: hand out only the exact permissions required. Share secrets only with containers that need them.
  • Don’t commit your plaintext secrets to source code.
  • Don’t store your secrets in the clear.
  • Never transmit a secret on an insecure connection. Use TLS.
  • Regularly rotate your secrets. Even if the risk of leaking a secret is minimized, automatic rotation of a secret ensures it will only be usable for a short time.

Conjur secures secrets for applications to use, but with the Secretless Broker feature, Conjur can also completely abstract secrets from the application development process, improving app development speed and security.   

When your application wants to reach a password-protected database, it proxies through the sidecar container, which is available on the pod’s local network. The secret is thus known only to the Secretless Broker, removing any risk from leaking the secret through the application. The sidecar container securely communicates with CyberArk Conjur using strong encryption and multi-factor authentication. 

All of this eliminates the Secret Zero problem, and makes it much easier to get your applications to interact with your secrets management solution.