Locks and Keys

Enforcement at scale with Admission Controllers with Secrets Management

The ability to consume external resources “on-the-fly,” is a core capability of cloud-native architecture, one that comes with a prerequisite that you also have the ability to dynamically configure applications at runtime. Kubernetes achieves this with the help of plugins called Admission Controllers that, in addition to enabling validation of apps at runtime, also enable modification. This is achieved by intercepting authenticated API requests on their way to the etcd cluster, and then either accepting, modifying and accepting, or rejecting them altogether.

The different types of admission controllers that exist in Kubernetes are mutating controllers, validating controllers, and a combination of both in some cases. Admission control typically runs in two phases with the modification or “mutating” phase going first, followed by the validation phase. This is done so that objects can first be modified to enforce custom defaults and then be validated to check for “deal-breakers.” LimitRanger is an example of an admission controller that does both.

Security and Governance

Admission controllers increase security considerably by automatically rejecting requests that don’t meet standards. They also help ensure images aren’t pulled from unknown directories by using extensions like Request Payloads that perform image reviews and then reject requests where the image isn’t specified. For more granular control, the PodSecurityPolicy admission controller enables more specific pod configurations like limiting users that can launch specific processes, forbidding escalation of rights, and blocking access to host drives.

Good governance is also enforced by ensuring best practices are followed by automatically adding custom defaults like labels, annotations, and dynamic resource limits where necessary. Examples of admission controllers that add custom defaults include DefaultStorageClass that sets the storage class volume to default if none is specified, and PodNodeSelector that sets defaults to limit which node selectors can be used. Another good example in terms of governance is PodTolerationRestriction, which ensures pods are only scheduled on appropriate nodes.

Limitations with Plugins

Now while there’s no denying admission plugins are a pretty useful feature of Kubernetes, there are a couple of limitations we need to look at as well. The biggest problem here is that Admission Controllers need to first be compiled into an API server, and hence, can only be configured or modified at startup and not at “anytime.” Additionally, the lack of a proper management framework makes it difficult to factor in environmental variables like vulnerability test results while the application is running.

Kubernetes addressed these issues by introducing a new type of Admission Controller called Webhook Admission Controllers. Since admission webhooks run externally through a REST endpoint, also known as a webhook, they’re basically agnostic to the API server, and hence, can be modified or reconfigured at any time. Kubernetes ships with two admission webhooks: one for validation and one for modification, called ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks, respectively. Both need to be enabled in order to create and configure admission webhooks.

Webhooks in the Wild

In addition to admission webhooks that are used internally in Kubernetes, webhooks also exist in the wild; the most popular being the mutating admission webhook that Istio uses to inject sidecars into applicable Kubernetes pods. Istio also uses another webhook called Galley for configuration validation. Linkerd, another popular service mesh, recently underwent a complete rebuild to implement the sidecar approach, and now uses mutating admission webhooks for proxy injection. Similarly, Consul Connect also uses mutating webhooks to inject sidecars.

Other real-world use cases of webhooks include Github that uses webhooks to update applications about the current state of repositories, Twilio that uses webhooks to react to phone calls, and Foursquare Developers that use webhooks to reduced mobile network traffic as well as battery usage! An important point to remember, however, is that by default, these webhooks manage their own configuration and additional steps are required to securely manage configurations.

Secrets Management for Webhooks

The security issues with admission webhooks stem from the delivery of data to publicly-available URLs within your application, which means anyone with the URL is free to corrupt your data. While enforcing TLS connections is the advised way around this, TLS certificates and private keys need to then be adequately safeguarded in the process. As opposed to private keys being stored in plain text in command line arguments, Kubernetes secrets are used to store and manage such sensitive information.

Kubernetes secrets are considered the bare minimum, however, in terms of securing your admission webhooks for production clusters. This is because Kubernetes stores secrets as base64 encoded strings that reside in etcd, making them a bit too general and static to be used in production environments or at scale. This is why best practice dictates using an enterprise-level secrets management solution like CyberArk Conjur or HashiCorp Vault to properly manage secrets at production scale.

Abstract Admission Control

Abstraction – or the process of decoupling complexity – is a trend we are increasingly seeing throughout the enterprise. Admission webhooks are yet another form of abstraction where we are basically decoupling the admission controllers from the API server in a way that requests can be validated or modified anytime during the application lifecycle. This makes for a complex environment, to say the least, and one in which establishing and maintaining trust levels between microservices can become increasingly difficult.

In conclusion, while the Kubernetes ecosystem is all about extending capabilities and finding new use cases, security implications need to be considered right from the get-go, especially in dynamic environments where there are more machine credentials than human users.

Join the Conversation on the CyberArk Commons

If you’re interested in this and other open-source content, join the conversation on the CyberArk Commons Community. Secretless Broker, Conjur and other open-source projects are a part of the CyberArk Commons Community, an open community dedicated to developers, engineers, cybersecurity researchers, and other technically-minded people. To discuss Kubernetes, Secretless Broker, Conjur, CyberArk Threat Research, join me on the CyberArk Commons discussion forum.