OpenBao Kubernetes auth across clusters, the secure way

#What goes wrong

The first cluster gets auth/kubernetes. The second cluster is added to the same idea with a copy of the role, and the policy says secret/data/app/*. Now a pod called app in namespace app gets the same secrets in both clusters. Whoever controls either cluster can create that pod.

Roles are written with bound_service_account_names="*" "for now". Any service account in the namespace can then log in.

Pods send their default service account token to OpenBao. That token is valid for the Kubernetes API too. Anything that sees it on the way can call the API as the pod.

For a remote cluster, the Kubernetes auth method must call that cluster's TokenReview API. Teams open the API server to the OpenBao network, and sometimes to the internet, to make that work.

#What the docs say

In general, Kubernetes applications should not share this JWT with other applications, as it allows API calls to be made on behalf of the Pod and can result in unintended access being granted to 3rd parties.

Source: OpenBao docs, Kubernetes auth method

List of service account names able to access this role. If set to "*" all names are allowed.

Source: OpenBao API docs, Kubernetes auth: create role

However, the client tokens cannot be revoked before their TTL expires, so it is recommended to keep the TTL short with that limitation in mind.

Source: OpenBao docs, Kubernetes auth method: use JWT auth

This method can be useful if Kubernetes' API is not reachable from OpenBao or if you would like a single JWT auth mount to service multiple Kubernetes clusters by chaining their public signing keys.

Source: OpenBao docs, JWT auth with Kubernetes as the provider

The docs offer one mount for several clusters as a convenience. They do not say that the same namespace and service account name then mean the same workload in every cluster, so each cluster needs its own mount and its own policy paths.

#The secure configuration

Two choices per cluster:

• OpenBao runs inside the cluster: use the kubernetes auth method with the pod's local token as reviewer. Kubernetes can revoke tokens early.
• The cluster is remote: use the jwt auth method with that cluster's service account public key. OpenBao needs no network path to the remote API server. Tokens cannot be revoked early, so keep their lifetime at the 10-minute minimum.

The configuration below is the remote case, one mount per cluster.

# cluster-a-app.hcl: the app in cluster A reads only cluster A's secrets.
path "secret/data/cluster-a/app/*" {
  capabilities = ["read"]
}

# Repeat for each cluster with its own name, key, issuer and policy.
bao policy write cluster-a-app cluster-a-app.hcl

bao auth enable -path=k8s-cluster-a jwt

# sa-a.pub: cluster A's service account signing public key
# (on kubeadm control planes, /etc/kubernetes/pki/sa.pub).
# During key rotation, list both the old and the new key.
bao write auth/k8s-cluster-a/config \
  [email protected] \
  bound_issuer=https://cluster-a.example.com     # the cluster's --service-account-issuer

bao write auth/k8s-cluster-a/role/app \
  role_type=jwt \
  user_claim=sub \
  bound_audiences=https://bao.example.com \
  bound_subject=system:serviceaccount:app:app \
  token_policies=cluster-a-app \
  token_no_default_policy=true \
  token_ttl=10m token_max_ttl=10m

In the cluster, the pod gets a token minted for OpenBao's audience only, and no API server token:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app
  namespace: app
automountServiceAccountToken: false   # no API-server token in the pod
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: app
spec:
  replicas: 1
  selector:
    matchLabels: {app: app}
  template:
    metadata:
      labels: {app: app}
    spec:
      serviceAccountName: app
      automountServiceAccountToken: false
      containers:
        - name: app
          image: registry.example.com/app@sha256:0000000000000000000000000000000000000000000000000000000000000000
          env:
            - name: BAO_ADDR
              value: https://bao.example.com:8200
            - name: BAO_AUTH_PATH
              value: auth/k8s-cluster-a      # this cluster's mount, never a shared one
          volumeMounts:
            - name: bao-token
              mountPath: /var/run/secrets/bao
              readOnly: true
      volumes:
        - name: bao-token
          projected:
            sources:
              - serviceAccountToken:
                  path: token
                  audience: https://bao.example.com   # only OpenBao accepts it
                  expirationSeconds: 600              # the minimum Kubernetes allows

For an OpenBao inside the cluster, the same rules apply to the kubernetes method: one mount per cluster, exact bound_service_account_names and bound_service_account_namespaces (never *), audience set on the role, and alias_name_source left at serviceaccount_uid.

#Prove it

The test in secure-tests/openbao-kubernetes-auth-clusters/ runs OpenBao in a container. No cluster is started. Two test RSA keys play the two clusters' signing keys, and the test signs tokens with the same claims a Kubernetes projected token carries (iss, aud, sub, kubernetes.io). The output below is from that container.

The app in cluster A logs in to its own mount and reads its own secret, and only that:

bao write auth/k8s-cluster-a/login role=app jwt=@token-a
bao kv get -field=password secret/cluster-a/app/db
bao kv get -field=password secret/cluster-b/app/db   # same token

{"policies":["cluster-a-app"],"ttl":600,"sub":{"role":"app"}}
example-a-only
Code: 403. Errors:
* permission denied

The same namespace and service account name from cluster B, presented to cluster A's mount:

Code: 400. Errors:
* error validating token: error verifying token signature: no known key successfully validated the token signature

A token with the API server's audience, which is what the default pod token carries:

Code: 400. Errors:
* error validating token: invalid audience (aud) claim: audience claim does not match any expected audience

Another service account in the same namespace:

Code: 400. Errors:
* error validating token: invalid subject (sub) claim

An expired token:

Code: 400. Errors:
* error validating token: invalid expiration time (exp) claim: token is expired

The manifest passes schema validation:

kubeconform -strict -summary -kubernetes-version 1.34.0 app.yaml

Summary: 2 resources found in 1 file - Valid: 2, Invalid: 0, Errors: 0, Skipped: 0

#With real tokens on a cluster

The same configuration on a lab cluster: OpenBao 2.7.0 in the cluster, jwt_validation_pubkeys from the control plane's /etc/kubernetes/pki/sa.pub, bound_issuer set to the cluster's issuer (kubectl get --raw /.well-known/openid-configuration | jq -r .issuer), and the Deployment above in a namespace baoapp, so the role's subject was system:serviceaccount:baoapp:app. A second mount, k8s-cluster-b, held a different key.

The token in the app pod, decoded:

{"iss":"https://kubernetes.default.svc.cluster.local","aud":["https://bao.example.com"],"sub":"system:serviceaccount:baoapp:app","ttl":600}
$ kubectl -n baoapp exec deploy/app -- ls /var/run/secrets/kubernetes.io/serviceaccount/
ls: /var/run/secrets/kubernetes.io/serviceaccount/: No such file or directory

Its logins, through the HTTP API:

app token -> k8s-cluster-a:   {"policies":["cluster-a-app"],"ttl":600,"sub":{"role":"app"}}
  read secret/cluster-a/app/db: example-a-only
  read secret/cluster-b/app/db: ["1 error occurred:\n\t* permission denied\n\n"]
app token -> k8s-cluster-b:   ["error validating token: error verifying token signature: no known key successfully validated the token signature"]

The other cases, each from a real pod:

service account app, default token (aud = the API server):
  {"aud":["https://kubernetes.default.svc.cluster.local"],"sub":"system:serviceaccount:baoapp:app"}
  ["error validating token: invalid audience (aud) claim: audience claim does not match any expected audience"]
service account other, OpenBao audience:
  ["error validating token: invalid subject (sub) claim"]

#Mistakes people make

#One mount for every cluster

A shared mount trusts every cluster's keys at once. Any cluster admin can then mint a login for any other cluster's workloads by creating the right namespace and name. One mount per cluster, one policy path per cluster.

#* in a role "for now"

A wildcard name means every service account in the namespace, including the one a CI job or a debug pod uses. Bind the exact subject.

#No audience, or the API server's audience

On the Kubernetes method, audience on the role is optional. Without it, OpenBao does not check the aud claim, so the default pod token, which also works against the API server, can log in. On the JWT method, a role with no bound_audiences rejects any token that carries an aud claim, but the OpenBao Kubernetes provider guide tells you to pick a value from the default audiences, which is the API server's. Use a dedicated audience, such as https://bao.example.com, on the role and on the projected token.

#Opening the API server so OpenBao can reach it

For remote clusters, the JWT method needs only the public key. No inbound path to the API server, no reviewer token stored in OpenBao.

#Long token lifetimes on the JWT method

OpenBao cannot ask the cluster whether a JWT was revoked. Keep expirationSeconds at 600 and token_max_ttl short.

#Forgetting key rotation

When the cluster rotates its service account key, logins fail. List the old and new public keys in jwt_validation_pubkeys during the overlap, then remove the old one.

#Checklist

• Create one auth mount per cluster.
• Configure each mount with that cluster's issuer and public key only.
• Scope each cluster's policies to its own path prefix.
• Bind every role to exact service account names and namespaces, or an exact sub.
• Set an audience on every role and on the projected token.
• Set automountServiceAccountToken: false on the ServiceAccount and the pod.
• Keep expirationSeconds at 600 and token_max_ttl at minutes, not hours.
• Keep alias_name_source at serviceaccount_uid for the Kubernetes method.
• Test that a token from cluster B fails on cluster A's mount.