Book a Security Review

Secrets and PKI

External Secrets Operator with OpenBao, the secure way

External Secrets Operator can read every Secret in the cluster and mint a token for any ServiceAccount. Install it with the defaults and one ClusterSecretStore, and that is the new shortest path to all your passwords.

The short answer

Turn off ClusterSecretStore, ClusterExternalSecret and PushSecret, and remove ESO's blanket right to create ServiceAccount tokens. Give each team a namespaced SecretStore that logs in to OpenBao as the team's own ServiceAccount, with an audience only OpenBao accepts, a pinned CA, and a read-only policy on the team's path.

Updated Houssam Hammoudi, CTOTested with External Secrets Operator v2.11.0, OpenBao 2.7.0, Kubernetes 1.34 (kind), kubeconform 0.8.0

On this page
  1. What goes wrong
  2. What the docs say
  3. The secure configuration
  4. Prove it
  5. Mistakes people make
  6. Checklist

What goes wrong

The provider example puts a static OpenBao token in a Kubernetes Secret. Teams then move it into one ClusterSecretStore so every namespace can use it. Any namespace can now reference that store and pull any secret the token can read. A developer with rights to create an ExternalSecret in their own namespace can read another team's database password.

The store logs in to OpenBao as the ESO controller's own ServiceAccount. Every team shares one OpenBao identity and one policy, which has to cover everyone.

The chart grants ESO serviceaccounts/token: create on every ServiceAccount. If the controller is compromised, it can mint a token for any workload in the cluster.

PushSecret is left on. It writes from the cluster into OpenBao, so the direction of trust flips: the cluster can now change what OpenBao holds.

What the docs say

External Secrets Operator integrates with OpenBao for secret management by using the HashiCorp Vault Provider.

Source: External Secrets Operator docs, OpenBao

For cluster-wide resources like ClusterSecretStore and ClusterExternalSecret, exercise caution since they have access to Secret resources across all namespaces.

Source: External Secrets Operator docs, Security best practices

However, this means ESO can create tokens for any ServiceAccount within its scope.

Source: External Secrets Operator docs, Security best practices

The OpenBao page sends you to the HashiCorp Vault provider page. That page starts with a static token called root "for the sake of simplicity", over plain http://. Neither page says that the policy and role on the OpenBao side are what actually limit a team. That part is yours.

The secure configuration

Helm values for the chart. No cluster-wide kinds, no PushSecret, no blanket token creation:

yaml
# values.yaml for the external-secrets chart (2.11.x)
crds:
  createClusterSecretStore: false
  createClusterExternalSecret: false
  createClusterPushSecret: false
  createPushSecret: false
processClusterStore: false
processClusterExternalSecret: false
processClusterPushSecret: false
processPushSecret: false
rbac:
  create: true
  # Token creation is granted per ServiceAccount with a Role that names it.
  serviceAccountTokenCreate: false

On the OpenBao side, one policy and one role per team:

hcl
# team-a-policy.hcl: read team A's values, nothing else.
# No list, no metadata, no write: ESO only needs read for explicit keys.
path "secret/data/team-a/*" {
  capabilities = ["read"]
}
# ESO checks its token with lookup-self. The role below sets
# token_no_default_policy, which removes the default policy that allows it.
path "auth/token/lookup-self" {
  capabilities = ["read"]
}
bash
bao policy write team-a-secrets team-a-policy.hcl

# One Kubernetes auth mount per cluster (see the cross-cluster page).
bao write auth/k8s-cluster-a/role/team-a-secrets \
  bound_service_account_names=team-a-secrets \
  bound_service_account_namespaces=team-a \
  audience=https://bao.example.com \
  token_policies=team-a-secrets \
  token_no_default_policy=true \
  token_ttl=10m token_max_ttl=10m

The mount calls the cluster's TokenReview API for every login. The reviewer needs the system:auth-delegator ClusterRole and a token the API server accepts: OpenBao's own ServiceAccount token when OpenBao runs in the cluster, or a token_reviewer_jwt set on the mount when it does not. The team's token cannot be the reviewer here. Its audience is OpenBao's, and the API server only accepts tokens bound to one of its own --api-audiences.

In the team's namespace: the ServiceAccount OpenBao trusts, the Role that lets ESO mint a token for that one ServiceAccount, the store, and an ExternalSecret.

yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: team-a-secrets
  namespace: team-a
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: eso-token-team-a-secrets
  namespace: team-a
rules:
  - apiGroups: [""]
    resources: ["serviceaccounts/token"]
    resourceNames: ["team-a-secrets"]        # this ServiceAccount only
    verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: eso-token-team-a-secrets
  namespace: team-a
subjects:
  - kind: ServiceAccount
    name: external-secrets
    namespace: external-secrets
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: eso-token-team-a-secrets
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: openbao-ca
  namespace: team-a
data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    (the CA that signed OpenBao's server certificate)
    -----END CERTIFICATE-----
---
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: openbao
  namespace: team-a
spec:
  provider:
    vault:                                   # OpenBao uses the Vault provider
      server: https://bao.example.com:8200
      path: secret
      version: v2
      caProvider:                            # verify OpenBao's certificate
        type: ConfigMap
        name: openbao-ca
        key: ca.crt
      auth:
        kubernetes:
          mountPath: k8s-cluster-a           # this cluster's auth mount
          role: team-a-secrets               # bound to this SA and namespace only
          serviceAccountRef:
            name: team-a-secrets             # the team's SA, not the ESO controller's
            audiences:
              - https://bao.example.com       # a token only OpenBao accepts
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-db
  namespace: team-a
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: SecretStore                        # namespaced; no ClusterSecretStore exists
    name: openbao
  target:
    name: app-db
    creationPolicy: Owner                    # the Secret is deleted with the ExternalSecret
  data:
    - secretKey: password
      remoteRef:
        key: team-a/app/db                   # read from secret/data/team-a/app/db
        property: password

Keep the right to create or edit SecretStores with the platform team. Teams may create ExternalSecrets in their own namespace.

Prove it

The test in secure-tests/external-secrets-operator-openbao/ builds JSON schemas from ESO v2.11.0's own CRDs, with unknown fields rejected, and runs kubeconform. This is real output:

bash
kubeconform -strict -summary -kubernetes-version 1.34.0 \
  -schema-location default \
  -schema-location 'schemas/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' team-a.yaml
text
Summary: 6 resources found in 1 file - Valid: 6, Invalid: 0, Errors: 0, Skipped: 0

The same file with caProvider misspelled as caprovider. If a tool that does not validate fields applied it, the API server would drop the unknown field, and ESO would verify OpenBao against the system roots instead of your CA:

text
typo.yaml - SecretStore openbao is invalid: problem validating schema. Check JSON formatting: jsonschema validation failed with 'file:///work/schemas/secretstore_v1.json#' - at '/spec/provider/vault': additional properties 'caprovider' not allowed
Summary: 6 resources found in 1 file - Valid: 5, Invalid: 1, Errors: 0, Skipped: 0

Every Helm key exists in the chart's values.yaml, and each one changes a default:

text
ok crds.createClusterSecretStore: chart default True -> False
ok crds.createClusterExternalSecret: chart default True -> False
ok crds.createClusterPushSecret: chart default True -> False
ok crds.createPushSecret: chart default True -> False
ok processClusterStore: chart default True -> False
ok processClusterExternalSecret: chart default True -> False
ok processClusterPushSecret: chart default True -> False
ok processPushSecret: chart default True -> False
ok rbac.create: chart default True -> True
ok rbac.serviceAccountTokenCreate: chart default True -> False

The OpenBao role, written and read back in an OpenBao 2.7.0 container:

bash
bao read -format=json auth/k8s-cluster-a/role/team-a-secrets | jq -c '.data | {bound_service_account_names, bound_service_account_namespaces, audience, alias_name_source, token_policies, token_ttl}'
text
{"bound_service_account_names":["team-a-secrets"],"bound_service_account_namespaces":["team-a"],"audience":"https://bao.example.com","alias_name_source":"serviceaccount_uid","token_policies":["team-a-secrets"],"token_ttl":600}

On a cluster

The chart installed with the values above, OpenBao 2.7.0 in the cluster behind a TLS listener signed by a lab CA, and a kubernetes auth mount (k8s-eso in the lab) whose reviewer is OpenBao's own ServiceAccount with system:auth-delegator. The team's resources as above, with the lab's server address, CA and mount.

text
$ kubectl api-resources --api-group=external-secrets.io
NAME              SHORTNAMES   APIVERSION               NAMESPACED   KIND
externalsecrets   es           external-secrets.io/v1   true         ExternalSecret
secretstores      ss           external-secrets.io/v1   true         SecretStore
$ kubectl auth can-i create serviceaccounts --subresource=token -n team-b --as=system:serviceaccount:external-secrets:external-secrets
no
$ kubectl auth can-i create serviceaccounts/team-a-secrets --subresource=token -n team-a --as=system:serviceaccount:external-secrets:external-secrets
yes
$ kubectl auth can-i create serviceaccounts/default --subresource=token -n team-a --as=system:serviceaccount:external-secrets:external-secrets
no

The first run failed. With only the secret/data/team-a/* rule in the policy, the login worked and then the store was rejected:

text
could not validate provider: invalid vault credentials: Error making API request.

URL: GET https://bao.openbao.svc.cluster.local:8200/v1/auth/token/lookup-self
Code: 403. Errors:

* 1 error occurred:
	* permission denied

ESO checks its token with lookup-self, which the default policy normally allows, and the role removes that policy. With the second rule in the policy above:

text
$ kubectl -n team-a get secretstore openbao
NAME      AGE   STATUS   CAPABILITIES   READY
openbao   3m    Valid    ReadWrite      True
$ kubectl -n team-a get externalsecret app-db
NAME     STORETYPE     STORE     REFRESH INTERVAL   STATUS         READY   LAST SYNC
app-db   SecretStore   openbao   1h                 SecretSynced   True    25s
$ kubectl -n team-a get secret app-db -o jsonpath='{.data.password}' | base64 -d
team-a-db-pass

ReadWrite is what the Vault provider supports, not what the token may do: the OpenBao policy allows reads only, and PushSecret is not installed.

Another team's path, from an ExternalSecret in team-a:

text
steal-b   SecretStore   openbao   1h                 SecretSyncedError   False
error processing spec.data[0] (key: team-b/app/db), err: cannot read secret data from Vault: Error making API request.

URL: GET https://bao.openbao.svc.cluster.local:8200/v1/secret/data/team-b/app/db
Code: 403. Errors:

No steal-b Secret was created.

The misspelled caprovider on the real API server:

text
$ kubectl apply -f typo.yaml
Error from server (BadRequest): error when creating "typo.yaml": SecretStore in version "v1" cannot be handled as a SecretStore: strict decoding error: unknown field "spec.provider.vault.caprovider"
$ kubectl apply --validate=false -f typo.yaml
secretstore.external-secrets.io/openbao-typo created
$ kubectl -n team-a get secretstore openbao-typo -o json | jq -c '.spec.provider.vault | {caProvider, caprovider}'
{"caProvider":null,"caprovider":null}

kubectl's default strict validation caught it. Any tool that turns that off stores the SecretStore without a CA. In the lab the store then failed with tls: failed to verify certificate, because the lab CA is private. If your OpenBao certificate chains to a public root, the same typo works silently with a weaker check.

Mistakes people make

A static token in a Secret

The docs' first example uses one "for the sake of simplicity". A root token can be set to never expire, and anyone who can read Secrets in that namespace has it. Use Kubernetes auth with a ServiceAccount.

One ClusterSecretStore for the whole cluster

Every namespace can reference it. The only thing between a team and another team's secrets is the store's OpenBao policy, which must cover everyone. Use a namespaced SecretStore per team, and turn the cluster kinds off.

Logging in as the controller

A store without serviceAccountRef uses the controller's own token. All teams share one OpenBao identity. Give each store the team's ServiceAccount and an OpenBao role bound to it.

Leaving serviceAccountTokenCreate on

The default lets ESO mint a token for any ServiceAccount. Turn it off and grant serviceaccounts/token per ServiceAccount with resourceNames.

list on secret/metadata/*

ESO needs it for dataFrom.find, which works by listing secret metadata. It also reveals every secret name under the path. Use explicit keys and grant read on secret/data/<team>/* only.

No CA on the store

Without caProvider or caBundle, ESO trusts the system roots for your private OpenBao. Pin the CA that signed OpenBao's certificate.

Checklist

  • Set the four crds.create* values and four process* values to false.
  • Set rbac.serviceAccountTokenCreate: false.
  • Create one ServiceAccount per team for OpenBao logins.
  • Grant ESO serviceaccounts/token create on that ServiceAccount only, with resourceNames.
  • Create one namespaced SecretStore per team with serviceAccountRef and an audience.
  • Pin OpenBao's CA with caProvider or caBundle.
  • Bind each OpenBao role to one ServiceAccount, one namespace and the audience.
  • Grant read on secret/data/<team>/* only.
  • Validate manifests with kubeconform -strict against the ESO CRDs you run.
  • Test that a team cannot read another team's path.

ESO is a very helpful courier. Make sure each team's packages come from its own mailbox, not from a master key the courier keeps in its pocket.

H2-CIAE

Learn it on a live range

Secrets and custody, in Identity and Access Engineering: a real host in your browser, and every objective checked on the machine.

Start free

The Secure Way

More on secrets and pki

OpenBao, External Secrets, internal certificate authorities and keeping secrets off the command line.

All secrets and pki guides