GKE Workload Identity Federation for Kubernetes Principals

In this post, we’ll take a look at a new change to Workload Identity Federation on GKE that can reduce the amount of configuration and overhead required for IAM resources, and see it in action with cert-manager using Cloud DNS.

GKE Workload Identity enables a Kubernetes Service Account (KSA) to authenticate with Google Cloud APIs without needing to manage keys or credentials.

By using this feature, the KSA’s token is used to verify its identity and receive a Google Cloud Service Account (GSA) access token which can be used to authenticate and access Google Cloud APIs.

As recommended in the CIS GKE Benchmarks v1.5.0 (5.2.2), users should: ‘Prefer using dedicated Google Cloud Service Accounts and Workload Identity’.

This mitigates against generating, storing and rotating Google Service Account Keys, and adheres to the 5.2.1 recommendation of ‘Ensure GKE clusters are not running using the Compute Engine default service account’ which by default has overly permissive access.

Previously, Workload Identity on GKE used Service Account Impersonation to grant Kubernetes Service Accounts access to Google Cloud APIs by impersonating Google Cloud Service Accounts and inheriting their associated IAM permissions.

Now, Workload Identity Federation for GKE allows Kubernetes Service Accounts to be referenced directly using a principal identifier in IAM Policies without using impersonation.

Before:

$ gcloud iam service-accounts create my-gsa

$ gcloud projects add-iam-policy-binding jetstack-paul --member “serviceAccount:[email protected]” --role “roles/viewer”

$ gcloud iam service-accounts add-iam-policy-binding [email protected] --role roles/iam.workloadIdentityUser --member “serviceAccount:jetstack-paul.svc.id.goog[default/my-ksa]”

$ kubectl create serviceaccount my-ksa

$ kubectl annotate serviceaccount my-ksa iam.gke.io/[email protected]

After:

$ gcloud projects add-iam-policy-binding jetstack-paul --role=roles/viewer  --member=principal://iam.googleapis.com/projects/993897508389/locations/global/workloadIdentityPools/jetstack-paul.svc.id.goog/subject/ns/default/sa/my-ksa \
   --condition=None

The benefits of removing the need to impersonate Google Service Accounts are:

• Fewer IAM Policy bindings to manage
  • Previously, each KSA required an IAM Policy Binding to the GSA that granted the workloadIdentityUser IAM Role to impersonate
• No superfluous GSA
  • There is no longer the need for GSAs to impersonate as IAM policy bindings can name KSAs as principals
• No more annotating Kubernetes Service Accounts with the Google Service Account to impersonate
  • This can be especially painful when setting annotations for resources in templates, or consuming public templates that don’t support annotations in their inputs

Let’s look at a concrete example of where GKE Workload Identity Federation can come in useful for everyday applications.

To use GKE Workload Identity Federation, create a GKE cluster with a Workload Pool. This is created by default on Autopilot clusters.

$ gcloud container clusters create example \
   --location=europe-west2-a \
   --workload-pool=jetstack-paul.svc.id.goog

Next, let’s configure a Kubernetes Service Account to use with cert-manager and Google Cloud DNS.

Creating an IAM Policy Binding can grant a Kubernetes Service Account principal the desired Google Cloud IAM permissions. Previously, these permissions would have been granted to a Google Service Account which the Kubernetes Service Account would impersonate.

$ gcloud projects add-iam-policy-binding project/jetstack-paul \
   --role=roles/dns.admin \
--member=principal://iam.googleapis.com/projects/0123456789012/locations/global/workloadIdentityPools/jetstack-paul.svc.id.goog/subject/ns/cert-manager/sa/cert-manager \
   --condition=None

When using principal identifiers in IAM Policy Bindings, the Workload Identity Pool is shared across all clusters within a project. This means that if two clusters in the same Workload Identity Pool both contain a Kubernetes Service Account with the same name (in the same namespace), the principal identifier will match both Service Accounts and therefore they will each be granted the same permissions.

This identity sameness stems from there only being a single Workload Identity Pool per Google Cloud Project (at the time of writing) which includes all GKE clusters and therefore all workload identities in these clusters. IAM Policy Conditions can be used to restrict a policy to a specific principal, however, to isolate workload identities then separate Google Cloud Projects and Workload Identity Pools should be used to separate principals across clusters.

With the Kubernetes Service Account now having the necessary IAM Permissions, cert-manager can be deployed using the Kubernetes Service account which will be issued with a token with authorization to access the required Cloud DNS APIs.

helm upgrade --install cert-manager jetstack/cert-manager \
 --namespace cert-manager \
 --create-namespace \
 --set installCRDs=true \
 --set global.leaderElection.namespace=cert-manager \
 --set extraArgs={--issuer-ambient-credentials}

As normal, an Issuer can be deployed that uses Google CloudDNS to solve DNS01 ACME challenges for requested Certificates.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
 name: cloud-dns
spec:
 acme:
   email: [email protected]
   server: https://acme-staging-v02.api.letsencrypt.org/directory
   privateKeySecretRef:
     name: issuer-account-key
   solvers:
   - dns01:
       cloudDNS:
         project: jetstack-paul
–--
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: example-com
spec:
 secretName: example-com-tls
 issuerRef:
   name: cloud-dns
 dnsNames:
 - example.paul-gcp.jetstacker.net

All examples can be found in this repo: https://github.com/paulwilljones/gke-cert-manager-wi-fed

As we have seen, workloads in GKE can now access Google Cloud APIs by having a single IAM policy that directly references the Kubernetes Service Account as a principal.

To create this IAM allow policy, we used gcloud but it can easily be done with any IaC tool that manages Google Cloud resources. See these examples for how to administer Workload Identity Federation on GKE:

Terraform

resource "google_project_iam_custom_role" "cert_manager" {
 project     = data.google_project.project.project_id
 role_id     = "certmanagertf"
 title       = "Cert Manager" permissions = ["dns.resourceRecordSets.create", "dns.resourceRecordSets.list", "dns.resourceRecordSets.get", "dns.resourceRecordSets.update", "dns.resourceRecordSets.delete", "dns.changes.get", "dns.changes.create", "dns.changes.list", "dns.managedZones.list"]
}

resource "google_project_iam_member" "cert_manager" {
 project = data.google_project.project.project_id
 role    = google_project_iam_custom_role.cert_manager.name
 member  = "principal://iam.googleapis.com/projects/${data.google_project.project.number}/locations/global/workloadIdentityPools/${data.google_project.project.project_id}.svc.id.goog/subject/ns/cert-manager/sa/cert-manager"
}

https://github.com/paulwilljones/gke-cert-manager-wi-fed/tree/develop/terraform

Config Connector

apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMCustomRole
metadata:
 annotations:
   cnrm.cloud.google.com/project-id: jetstack-paul
 name: certmanagerkcc		#intentional naming to comply with IAM naming
spec:
 title: Cert Manager
 resourceID: certmanagerkcc
 permissions:
   - dns.resourceRecordSets.create
   - dns.resourceRecordSets.list
   - dns.resourceRecordSets.get
   - dns.resourceRecordSets.update
   - dns.resourceRecordSets.delete
   - dns.changes.get
   - dns.changes.create
   - dns.changes.list
   - dns.managedZones.list
 stage: GA
---
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicyMember
metadata:
 name: cert-manager-kcc
 namespace: cert-manager
 annotations:
   cnrm.cloud.google.com/project-id: jetstack-paul
spec:
 member: principal://iam.googleapis.com/projects/993897508389/locations/global/workloadIdentityPools/jetstack-paul.svc.id.goog/subject/ns/cert-manger/sa/cert-manager
 role: projects/jetstack-paul/roles/certmanagerkcc
 resourceRef:
   kind: Project
   external: projects/jetstack-paul

https://github.com/paulwilljones/gke-cert-manager-wi-fed/tree/develop/kcc

Crossplane

apiVersion: cloudplatform.gcp.upbound.io/v1beta1
kind: ProjectIAMCustomRole
metadata:
 name: certmanagerxp
spec:
 forProvider:
   permissions:
     - dns.resourceRecordSets.create
     - dns.resourceRecordSets.list
     - dns.resourceRecordSets.get
     - dns.resourceRecordSets.update
     - dns.resourceRecordSets.delete
     - dns.changes.get
     - dns.changes.create
     - dns.changes.list
     - dns.managedZones.list
   title: Cert Manager (Crossplane)
–--
apiVersion: cloudplatform.gcp.upbound.io/v1beta1
kind: ProjectIAMMember
metadata:
 name: cert-manager-xp
 namespace: cert-manager
spec:
 forProvider:
   project: jetstack-paul
   member: principal://iam.googleapis.com/projects/993897508389/locations/global/workloadIdentityPools/jetstack-paul.svc.id.goog/subject/ns/cert-manager/sa/cert-manager
   role: projects/jetstack-paul/roles/certmanagerxp

https://github.com/paulwilljones/gke-cert-manager-wi-fed/tree/develop/crossplane

Pulumi

cert_manager_role = gcp.projects.IAMCustomRole(
   "cert-manager",
   project=my_project.projects[0].project_id,
   role_id="certmanagerpulumi",
   title="Cert Manager (Pulumi)",
   permissions=[
       "dns.resourceRecordSets.create",
       "dns.resourceRecordSets.list",
       "dns.resourceRecordSets.get",
       "dns.resourceRecordSets.update",
       "dns.resourceRecordSets.delete",
       "dns.changes.get",
       "dns.changes.create",
       "dns.changes.list",
       "dns.managedZones.list",
   ],
)

project = gcp.projects.IAMMember(
   "project",
   project=my_project.projects[0].project_id,
   role=cert_manager_role.name,
   member=f"principal://iam.googleapis.com/projects/{my_project.projects[0].number}/locations/global/workloadIdentityPools/{my_project.projects[0].project_id}.svc.id.goog/subject/ns/cert-manager/sa/cert-manager",
)

manager = CertManager(
   "cert-manager",
   install_crds=True,
   helm_options=ReleaseArgs(namespace="cert-manager", create_namespace=True),
   extra_args=["--issuer-ambient-credentials"],
   global_=CertManagerGlobalArgs(
       leader_election=CertManagerGlobalLeaderElectionArgs(namespace="cert-manager")
   )
)

https://github.com/paulwilljones/gke-cert-manager-wi-fed/tree/develop/pulumi

For more information on GKE Workload Identity Federation, see the documentation here.