Signing keys no cluster can read, the secure way

#What goes wrong

cosign generate-key-pair k8s://ci/signing is one command, and the key pair is now a Kubernetes Secret. Every workload that can read Secrets in that namespace, every etcd backup, and every cluster admin has your release key.

The password on the key does not help. cosign stores it in the same Secret, as cosign.password, next to cosign.key.

When the build runner is compromised, the attacker copies the key and leaves. They can sign anything, from anywhere, for as long as the key is trusted. Rotating it means re-signing and re-distributing trust to every verifier.

#What the docs say

The secret will contain the private and public keys, as well as the password to decrypt the private key.

Source: Sigstore docs, cosign key management: Kubernetes Secret

This provider requires that the standard Vault or OpenBao environment variables ($VAULT_ADDR, $VAULT_TOKEN, $BAO_ADDR, $BAO_TOKEN) are set correctly.

Source: Sigstore docs, cosign key management: Hashicorp Vault and OpenBao

Enables keys to be exportable. This allows for all the valid keys in the key ring to be exported. Once set, this cannot be disabled.

Source: OpenBao API docs, Transit secrets engine: update key configuration

The cosign docs say the token needs "(at least)" a policy that also has hmac and verify paths. Signing works without them, as the test shows. Neither page says that a token allowed to update transit/keys/<name>/config can make the key exportable, for good.

#The secure configuration

Create the key once, as an OpenBao admin:

bao secrets enable transit
# exportable, allow_plaintext_backup and deletion_allowed stay false (the defaults).
bao write -f transit/keys/release type=ecdsa-p256

The signing policy. Sign with one key, read its public half, revoke yourself:

# cosign-sign.hcl
# No export, no config (which could make the key exportable), no other keys.
path "transit/sign/release/*" {
  capabilities = ["update"]
}
path "transit/sign/release" {
  capabilities = ["update"]
}
path "transit/keys/release" {
  capabilities = ["read"]      # public key and metadata only
}
path "auth/token/revoke-self" {
  capabilities = ["update"]
}

Give it to CI through a login bound to the pipeline's identity (for example a JWT auth role that trusts your CI's OIDC tokens for one repository and the release branch), with a short TTL:

bao policy write cosign-sign cosign-sign.hcl
bao write auth/ci-jwt/role/release-signer \
  role_type=jwt user_claim=sub \
  bound_audiences=https://bao.example.com \
  bound_subject="repo:example/app:ref:refs/heads/main" \
  token_policies=cosign-sign token_no_default_policy=true \
  token_ttl=15m token_max_ttl=15m

The signing step in the pipeline:

export BAO_ADDR=https://bao.example.com:8200
export BAO_TOKEN="$(bao write -field=token auth/ci-jwt/login role=release-signer jwt=@"$CI_OIDC_TOKEN_FILE")"
cosign sign-blob --yes --key openbao://release --bundle app.sigstore.json app-v1.0.0.tar
bao token revoke -self

Publish cosign.pub (from cosign public-key --key openbao://release) with your releases. Verifiers never need OpenBao.

Keep update on transit/keys/* and transit/keys/*/config out of every human day-to-day policy, and alert on any request to those paths in the audit log.

#Prove it

From secure-tests/signing-keys-no-cluster-can-read/. The key as created:

bao read -format=json transit/keys/release | jq -c '.data | {type, exportable, deletion_allowed, allow_plaintext_backup, latest_version}'

{"type":"ecdsa-p256","exportable":false,"deletion_allowed":false,"allow_plaintext_backup":false,"latest_version":1}

Nobody can take the private key out, and the CI token cannot change that or create a key of its own:

bao read transit/export/signing-key/release                       # admin
bao write transit/keys/release/config exportable=true             # CI token
bao write -f transit/keys/other type=ecdsa-p256                   # CI token

Code: 400. Errors:
* private key material is not exportable
Code: 403. Errors:
* 1 error occurred:
* permission denied
Code: 403. Errors:
* 1 error occurred:
* permission denied

cosign signs through OpenBao with the CI token:

cosign public-key --key openbao://release > cosign.pub
cosign sign-blob --key openbao://release --tlog-upload=false --use-signing-config=false --bundle app.sigstore.json app-v1.0.0.tar
cosign verify-blob --key cosign.pub --bundle app.sigstore.json --insecure-ignore-tlog=true app-v1.0.0.tar

-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEouM6edZ3dNn9aBINFiCnySnhaXqD
3V8RE/Vco6EfVPpmI3Idu34DExBhVc0OJokCrXHr15CHCWCLWRFbtVCo9g==
-----END PUBLIC KEY-----
Wrote bundle to file app.sigstore.json
MEQCIFNetUEnbksLSjroUABSY7kXRz8yr/gkem2mm5lkFHSAAiAQp6XyyN5se1w0OO4oyhA8WTlOBWR8wQN9q9e7rcQupA==
Verified OK

A changed artifact fails verification:

Error: failed to verify signature: could not verify message: invalid signature when validating ASN.1 encoded signature

After the job revokes its token, the same token cannot sign:

bao token revoke -self
cosign sign-blob --key openbao://release ...

Success! Revoked token (if it existed)
Error: signing app-v1.0.0.tar: getting keypair and token: creating signerverifier keypair: getting public key: public key: Error making API request.
Code: 403. Errors:
* permission denied

The test turns the transparency log off because it runs without network. In production, leave --tlog-upload on (or run your own Rekor) so every signature made with the key is on a public record you can watch.

#Mistakes people make

#k8s:// keys "because it is easy"

A key in a Secret is a file with extra steps. Anyone who can read the Secret can sign as you, from anywhere, forever.

#A signing token that can touch the key's config

update on transit/keys/release/config can set exportable=true, and that cannot be undone. From then on the key can be copied. Grant sign and a read on the key, nothing more.

#Long-lived CI tokens

A token stored as a CI variable is a key by another name. Log in from the job with the CI's own OIDC token, keep the TTL at minutes, and revoke at the end.

#Any branch can sign

Bind the login role to the repository and the release branch or tag. A pull request from a fork should never get a signing token.

#No record of what was signed

The key cannot leak, but a stolen token can still sign during its TTL. Keep the transparency log on, keep OpenBao's audit log, and compare the list of signatures with the list of releases.

#Rotating by deleting

Old signatures need the old public key. Rotate with transit/keys/release/rotate, publish the new public key, and keep the old one for verification until those releases are retired.

#Checklist

• Create signing keys in OpenBao transit, never with k8s:// or as files in CI.
• Leave exportable, allow_plaintext_backup and deletion_allowed false.
• Grant CI update on transit/sign/<key> and read on transit/keys/<key> only.
• Keep transit/keys/*/config and transit/export/* out of every policy used day to day.
• Issue CI tokens from an OIDC login bound to one repository and branch, TTL 15 minutes.
• Revoke the token as the last step of the job.
• Keep transparency log upload on in production.
• Alert on audit log entries for transit/keys/*/config, transit/export/* and transit/keys/* writes.
• Publish the public key and keep old ones after rotation.