Essentials to Securing Kubernetes Secrets with Secrets Management

Secret management is essential for ensuring an organization’s cybersecurity. In this era, when users share valuable information with service providers, social media applications, shopping applications, and more, privacy should be the highest priority. Secret management covers all the processes and tools to create, store, transmit, and manage digital credentials such as encryption keys, APIs, and passwords. A proper secret management technique safeguards user privacy.

Because of its robust and scalable nature, using Kubernetes to run enterprise applications has become significantly more popular in the last few years. However, Kubernetes security vulnerabilities force DevOps engineers to look for the best and most affordable options to secure their container manager. In this tutorial, we’ll elaborate on the four ways to secure Kubernetes secrets that we discussed in  a prior blog.

Monitor and Control Access to Kubernetes Containers and Pods

DevOps must manage and control an application correctly to maintain its reliability. Proper secrets management ensures secure applications by regulating access to containers and pods. Passwords, SSH keys, certificates, API tokens, and other secrets grant container access, so it’s crucial they do not end up in the wrong hands. Including secrets directly in code and exposing them publicly in source code management tools can turn into serious vulnerabilities.

Leaving secrets in code or configuration — or in application containers where they may be inadvertently discovered — can lead to unwanted access or privilege escalation — a main cause of the secret zero problem. Attackers can misuse this way to fetch the access and privilege and further expand a small breach into a big attack that breaches a whole network.

To reduce mishandled secret exposure and potential damage, DevOps engineers should always follow best practices while managing control and access. For example:

• Always start with a minimum set of permissions and grant additional permissions as necessary.
• Ensure to encrypt secrets and tightly control secrets access.
• Authorize secrets access based on a security policy suitable for a highly-flexible environment like Kubernetes.

One good practice when authorizing containers is to assign a role with only the least required set of permissions. Another best practice is to regularly rotate secrets to keep the access limited to required persons or identities. Every organization should carry out regular audits for visibility into access to critical systems.

You can control access by creating a role and applying it on kubectl. Make this role by giving access to a specific user for specific operations. In Role-Based Access Control (RBAC), you control creating, updating, deleting, editing, and viewing resources.

Perform the following steps to control access to your containers and pods:

1. Clone the demo application repo.
2. Clone the Conjur CLI from the Conjur Kubernetes integration demo.
3. From Conjur CLI, create a namespace using app_create_namespace.sh. This will create a namespace on Kubernetes and add different roles to the admin policy.
4. Further add or modify assigned roles by changing functions inside app_create_namespace.sh

By following these simple steps, you control access to containers and pods.

Strongly Authenticate Kubernetes Container Requests for Secrets

RBAC is a mechanism using security management policies that assure that no unauthorized person gets access. But there are many possibilities that the secret provider can provide secrets to the attacker itself. For example, attacker A tries to get secrets using the API and without RBAC, the secret provider will give all the secrets to the attacker and it can be a huge problem. So, how do we keep secrets confidential? Via another secret? If yes, then we return again to the secret zero problem.

In order to make it more secure, we have to implement stronger Kubernetes authentication of client secrets requests. There are many approaches to make it secure like single-factor

authentication, token authentication, multi-factor authentication, and many more. But the best way to implement more secure authentication is by running an underlying orchestration container to authenticate container requests by the multi-factor authentication method.

To authenticate the secret requests, we can implement a solution that uses multiple attributes such as certificates generated with the help of Kubernetes APIs, UUIDs, cryptographic keys, role access, token access, and many more.

Given that these attributes are only available to trusted containers in the orchestration system. The presence of one factor will not be enough to authenticate container requests as we will be using multi-factor authentication. So, we will be required to use multiple factors for authentication. And in any condition, the attacker gets credentials then also it will be of zero value if the attacker using it can’t be authenticated.

Conjur uses Secure Production Identity Framework for Everyone (SPIFFE) standards, using X.509 authentication. After initializing a Conjur instance, run the following commands to generate a self-signed certificate to authenticate Kubernetes requests:

ssl_cert=$(kubectl exec [conjur-pod-name] -- cat /opt/conjur/etc/ssl/conjur.pem)

Kubectl create configmap conjur-public-cert --from-file=ssl-certificate=<(echo "$ssl_cert")

After creating a configmap with an SSL certificate, we provide the SSL_CERTIFICATE in the environment so the application can be authenticated towards it.

Secretless Kubernetes Applications

Another way of keeping secrets secret is to isolate Kubernetes applications from the secrets, making them secretless. Implement this using secret providers. Leveraging secret providers stops the application from leaking secrets. Effectively, it is another application to manage secrets. Secretless Broker is one such open-source project.

When an application needs to access a resource securely, the app simply makes a local connection request to Secretless Broker. Secretless Broker then automatically authenticates the app, fetches the required credentials from the secrets provider, and establishes a connection to the database or other resource.

With Secretless Broker, the application no longer has access to credentials, preventing accidental leakage or exposure and reducing the attack surface.

This approach offers two significant simplifications for developers. First, it eliminates the developer’s need to write API calls to fetch the access credential or secret from Kubernetes secrets (or any other secrets store). Second, it removes the need for the application to handle secrets directly.

Secretless Broker proxies the application with secrets, ensuring the application works correctly without directly handling secrets. This mechanism reduces the hassle of the developer storing secrets and calling REST APIs.

To deploy Secretless Broker:

1. Load secrets with string literal using this command:

     kubectl create secret generic my-app-postgres \
       --from-literal=address=$POSTGRES_ADDRESS \
       --from-literal=username=$POSTGRES_USERNAME \
       --from-literal=password=$POSTGRES_PASSWORD

2. Create secretless.yml with the following configuration, which instructs it to listen on port 5432:

     version: "2"
     services:
        pg-db:
          connector: pg
          listenOn: tcp://0.0.0.0:5432
          credentials:
            host:
              from: kubernetes
              get: my-app-postgres#host
            port:
              from: kubernetes
              get: my-app-postgres#port
            username:
              from: kubernetes
              get: my-app-postgres#username
            password:
              from: kubernetes
              get: my-app-postgres#password

3. Update your application with database port 5432.
4. Add container dependency in the application’s configmap:

     - name: secretless-broker
           image: cyberark/secretless-broker:latest
           args: ["-f", "/etc/secretless/secretless.yml"]
           ports:
           - containerPort: 5432
             volumeMounts:
           - name: config
           mountPath: "/etc/secretless"
           readOnly: true

The Conjur SDK provides a simplified option for applications to access MySQL and PostgreSQL databases securely. It also enables developers to add support for additional databases and resources. You can try the interactive demo to explore the Conjur SDK further.

Centralize Secrets Management

It is possible to centrally manage secrets using several centralized tools available within various cloud platforms, like Amazon Web Services (AWS) and Google Cloud. DevOps tools like Ansible and Puppet also store secrets.

However, orchestration platforms like Kubernetes require more than one of the tools to manage all secrets. This creates blind spots within an organization’s access management picture and other difficulties since these tools are not interoperable or fully-featured for secrets management.

Centrally managing secrets is also key to fully auditing access, centralizing audit records in the process. Distributing secrets management responsibilities across multiple tools creates what is known as “security islands,” which every DevOps engineer should avoid.

Centralized secrets management solutions simplify audit, access control, and secrets management, providing a centralized view of the entire Kubernetes landscape. CyberArk Conjur’s seamless, open-source interface securely authenticates, controls, and audits non-human access across tools, applications, containers, and cloud environments via robust secrets management

Next Steps

In this article, we explored four ways to keep Kubernetes secrets secret.  Now you can try it for yourself with this interactive Kubernetes secrets management tutorial.  Join our community forum to ask questions and explore what other developers talking about and signup for the developer community newsletter to see the latest tutorials, blogs, and CyberArk developer news.