Kuma certificate rotation and CA choices, the secure way

#What goes wrong

Kuma issues a certificate to every data plane proxy from the mesh CA and renews it automatically. Two lifetimes matter, and they fail differently:

• The workload certificate. Short means a stolen certificate is useful for hours, not weeks. Too short for your control plane's recovery time means proxies start failing if the control plane is down. Kuma's docs say the default is 30 days on one page and 24 hours on another; Kuma 2.14.3 issues 24-hour certificates when you set nothing.
• The CA. The builtin backend generates a root certificate and key and stores both as Kuma Secrets. On Kubernetes they sit in kuma-system; in multi-zone they sync to every zone. The key signs every identity in the mesh for as long as the root is valid.

The CA key is the real target. Anyone who can read it can mint a certificate for any service and pass every MeshTrafficPermission.

Then comes the day the CA must change. Kuma 2.14 accepts one CA backend per mesh and refuses to change it while mTLS is on. The only change it allows is new contents in the backend's Secrets. Done in one step, that splits the mesh: proxies that got a certificate from the new CA and proxies still on the old one reject each other, until every proxy has renewed.

#What the docs say

By default, the expiration time of a data plane proxy certificate is 30 days. Kuma rotates these certificates automatically after 4/5 of the certificate validity time

Source: Kuma docs, Mutual TLS

On mTLS enabled meshes, a data plane proxy may fail to refresh its client certificate prior to expiry (defaults to 24 hours), thus causing traffic from/to this data plane to fail.

Source: Kuma docs, Multi-zone deployment

We can have as many backends as we want, but only one at a time can be enabled via the enabledBackend property.

Source: Kuma docs, Mutual TLS

Using the inline modes in production presents a security risk since it makes the values of our CA root certificate and key more easily accessible from a malicious actor.

Source: Kuma docs, Mutual TLS

On Kuma 2.14.3 the 24-hour figure is the true one, and the four-fifths renewal is exact. The "as many backends as we want" sentence is not: a second backend is rejected, and so is any change of enabledBackend while mTLS is on (see Prove it). The docs do not describe a CA replacement at all.

#The secure configuration

1. Choose the CA backend, once. You cannot switch backends later without turning mTLS off for the whole mesh, so choose before you enable it.

2. Build an offline root and a mesh intermediate. Run this on an offline machine. Kuma requires the CA certificate to have CA:TRUE and keyCertSign, and rejects keyAgreement.

cat > root.cnf <<'CNF'
[req]
distinguished_name = dn
prompt = no
[dn]
CN = Example Mesh Root CA 2026
[v3_root]
basicConstraints = critical, CA:TRUE, pathlen:1
keyUsage = critical, keyCertSign, cRLSign
subjectKeyIdentifier = hash
[v3_int]
basicConstraints = critical, CA:TRUE, pathlen:0
keyUsage = critical, keyCertSign, cRLSign
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
CNF

# Root: ten years, key stays offline.
openssl req -config root.cnf -x509 -new -newkey rsa:4096 -nodes -sha256 \
  -days 3650 -extensions v3_root -keyout root.key -out root.crt

# Intermediate for one mesh: one year.
openssl req -new -newkey rsa:4096 -nodes -sha256 \
  -subj "/CN=Example Mesh default CA 2026" -keyout mesh-ca.key -out mesh-ca.csr
openssl x509 -req -in mesh-ca.csr -CA root.crt -CAkey root.key -CAcreateserial \
  -sha256 -days 365 -extfile root.cnf -extensions v3_int -out mesh-ca.crt

# Kuma wants the intermediate first, then the root.
cat mesh-ca.crt root.crt > mesh-ca-chain.pem

3. Store them as Kuma Secrets, labelled with the mesh when you create them. Kuma puts an unlabelled Secret in the default mesh and refuses to move it to another mesh afterwards. Use names without a year: the backend will point at these names for the life of the mesh.

put() {   # put <secret-name> <file>: create or replace a Kuma Secret for mesh "default"
  kubectl -n kuma-system create secret generic "$1" --type=system.kuma.io/secret \
    --from-file=value="$2" --dry-run=client -o yaml \
    | kubectl label --local -f - kuma.io/mesh=default -o yaml \
    | kubectl apply -f -
}
put mesh-ca-cert mesh-ca-chain.pem
put mesh-ca-key mesh-ca.key
shred -u mesh-ca.key            # the only copy left is the Secret

In multi-zone, create them on the global control plane; they sync to the zones. The root key never leaves the offline machine.

4. Point the Mesh at the provided backend, with an explicit lifetime.

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca
    backends:
      - name: ca                     # one backend for the life of the mesh
        type: provided
        mode: STRICT
        dpCert:
          rotation:
            expiration: 24h          # renewed after about 19 hours
        conf:
          cert:
            secret: mesh-ca-cert     # intermediate + root chain
          key:
            secret: mesh-ca-key      # intermediate key only

Kuma renews after four fifths of the lifetime. With 24 hours, a proxy has about five hours of margin if its control plane is unreachable. If your zone control plane can be down longer, raise the lifetime and alert on it; do not jump to 30 days by default.

5. Rotate the CA in three stages. A proxy trusts exactly the certificates in mesh-ca-cert, as they were when its own certificate was issued, and sends only its leaf certificate. So every proxy must trust the new CA before any proxy gets a certificate from it, and must keep trusting the old one until none is left.

# Stage 1: old intermediate still signs; everyone learns to trust the new one.
cat mesh-ca.crt root.crt mesh-ca-2027.crt > stage1.pem
put mesh-ca-cert stage1.pem

# Stage 2: the new intermediate signs; the old one stays trusted.
cat mesh-ca-2027.crt root.crt mesh-ca.crt > stage2.pem
put mesh-ca-cert stage2.pem
put mesh-ca-key mesh-ca-2027.key

# Stage 3: drop the old intermediate.
cat mesh-ca-2027.crt root.crt > stage3.pem
put mesh-ca-cert stage3.pem

After each stage, every proxy must be reissued before the next one. Either wait longer than one certificate lifetime, or force it: wait a minute (the control plane does not reread the Secret at once), then change dpCert.rotation.expiration (for example 24h to 23h, and back at the next stage). The control plane then logs generating certificate ... "reason": "Mesh mTLS settings have changed" for every proxy. Check the result with the script in Prove it before you go on.

The same three stages replace the root: put the new intermediate and the new root in stage 1 (mesh-ca.crt root.crt new-int.crt new-root.crt), sign with the new intermediate in stage 2, and keep only the new pair in stage 3.

#Prove it

Run on a lab cluster with Kuma 2.14.3, in a separate test mesh rot with a provided backend, an nginx server, a curl client, and a second curl client (the watcher) polling the server twice a second during every rotation.

1. The CA meets Kuma's requirements, and Kuma rejects one that does not. The page's commands, with OpenSSL 3.5.8 in alpine:3.22:

mesh-ca.crt: OK
subject=CN=Example Mesh default CA 2026
issuer=CN=Example Mesh Root CA 2026
notAfter=Sep 24 21:08:42 2027 GMT
X509v3 Basic Constraints: critical
    CA:TRUE, pathlen:0
X509v3 Key Usage: critical
    Certificate Sign, CRL Sign

An intermediate with digitalSignature, keyAgreement added:

The Mesh "rotbad" is invalid: spec.mtls.backends[0].conf.cert[0]: key usage extension 'keyAgreement' must NOT be set (see X509-SVID: Appendix A. X.509 Field Reference)

2. The Secret must be labelled at creation:

$ kubectl -n kuma-system label secret rot-ca-2026-cert kuma.io/mesh=rot
Error from server (Invalid): admission webhook "secret.validator.kuma-admission.kuma.io" denied the request: metadata.labels["kuma.io/mesh"]: cannot change mesh of the Secret. Delete the Secret first and apply it again.

3. The default lifetime is 24 hours, and renewal comes at four fifths. The lab's default mesh has a builtin backend with no dpCert (kumactl inspect dataplanes, trimmed):

MESH      NAME                           CERT REGENERATED AGO   CERT EXPIRATION       CERT BACKEND
default   backend-7d59fcd4cd-gwtzk.app   15m                    2026-09-26 07:53:36   ca-1

Issued at 07:53:36, expiring at 07:53:36 the next day. With expiration: 5m in the test mesh, a certificate issued at 08:09:20 (expiring 08:14:20) was replaced at 08:13:20 by one expiring 08:18:20: four minutes of five.

4. Kuma refuses a second backend and a backend change:

The Mesh "rot" is invalid: spec.mtls.backends: cannot have more than 1 backends
The Mesh "rot" is invalid: spec.mtls.enabledBackend: Changing CA when mTLS is enabled is forbidden. Disable mTLS first and then change the CA

5. Replacing the CA in one step breaks traffic. The Secrets were changed to a new intermediate in one go, then only the client was restarted, so it got a certificate from the new CA while the server kept its old one:

client -> server: 0/20 ok, failures: 503 ...
after reissuing all proxies:
client -> server: 20/20 ok

In production, with 24-hour certificates renewing over about 19 hours, that split lasts for hours.

6. The three stages do not. The same test, with the client restarted in stage 2 so that the two sides hold certificates from different intermediates:

== stage 1: old int (signs) + new int + root, old key; reissue all
  client: identity issued by [Example Mesh rot CA 2026], trusts [Example Mesh rot CA 2026,Example Mesh rot CA 2027,Example Mesh Root CA 2026]
  server: identity issued by [Example Mesh rot CA 2026], trusts [Example Mesh rot CA 2026,Example Mesh rot CA 2027,Example Mesh Root CA 2026]
== stage 2: new int (signs) + old int + root, new key; wait, reissue the client only
  client: identity issued by [Example Mesh rot CA 2027], trusts [Example Mesh rot CA 2027,Example Mesh rot CA 2026,Example Mesh Root CA 2026]
  server: identity issued by [Example Mesh rot CA 2026], trusts [Example Mesh rot CA 2026,Example Mesh rot CA 2027,Example Mesh Root CA 2026]
  client -> server: 20/20 ok
== stage 3: new int + root; reissue all
  client: identity issued by [Example Mesh rot CA 2027], trusts [Example Mesh rot CA 2027,Example Mesh Root CA 2026]
  server: identity issued by [Example Mesh rot CA 2027], trusts [Example Mesh rot CA 2027,Example Mesh Root CA 2026]
  client -> server: 20/20 ok

The watcher sent 605 requests during a full intermediate rotation: 605 returned 200. A rotation to a new root and a new intermediate, in the same three stages, with the client under the new root and the server under the old one in stage 2: 20/20, and the watcher got 781 of 781.

A proxy restarted about 20 seconds after a Secret change still got a certificate from the old contents; after a one-minute wait, from the new. That is why each stage waits before it forces the reissue.

7. Check every proxy between stages. This script reads each proxy's identity issuer and trusted CAs from its config dump:

#!/bin/sh
# For every proxy in a mesh: who issued its identity, and which CAs it trusts.
MESH=${1:-default}
for dp in $(kumactl get dataplanes --mesh "$MESH" -o json | jq -r '.items[].name'); do
  kumactl inspect dataplane "$dp" --mesh "$MESH" --type=config-dump > /tmp/dump.json
  sec() { jq -r --arg n "$1:secret:$MESH" '.configs[] | select(."@type"|test("SecretsConfigDump"))
      | .dynamic_active_secrets[] | select(.name==$n) | .secret
      | (.tls_certificate.certificate_chain.inline_bytes // .validation_context.trusted_ca.inline_bytes)' /tmp/dump.json | base64 -d; }
  issuer=$(sec identity_cert | openssl x509 -noout -issuer | sed 's/.*CN *= *//')
  trusts=$(sec mesh_ca | awk '/BEGIN CERT/{c=""} {c=c $0 "\n"} /END CERT/{printf "%s", c | "openssl x509 -noout -subject"; close("openssl x509 -noout -subject")}' | sed 's/.*CN *= *//' | paste -sd, -)
  echo "$dp | issued by: $issuer | trusts: $trusts"
done

client-59b5797769-2f5wz.kuma-rot | issued by: Example Mesh rot CA under 2036 root | trusts: Example Mesh rot CA under 2036 root,Example Mesh Root CA 2036
server-786f9b64bb-vfk5s.kuma-rot | issued by: Example Mesh rot CA under 2036 root | trusts: Example Mesh rot CA under 2036 root,Example Mesh Root CA 2036
watcher-6f9975bf46-z8cp4.kuma-rot | issued by: Example Mesh rot CA under 2036 root | trusts: Example Mesh rot CA under 2036 root,Example Mesh Root CA 2036

Go to the next stage only when every line shows the expected issuer and trust list.

8. No inline CA material and no builtin key in production:

kubectl get meshes -o yaml | grep -n -E 'inline|type: builtin'
kubectl -n kuma-system get secrets | grep ca-builtin

On the lab, whose default mesh is builtin, both commands find it:

20:        type: builtin
default.ca-builtin-cert-ca-1            system.kuma.io/secret          1      10h
default.ca-builtin-key-ca-1             system.kuma.io/secret          1      10h

On a production mesh set up as above, both print nothing.

#Mistakes people make

#Accepting the default lifetime

The docs state two different defaults (24 hours and 30 days). Put dpCert.rotation.expiration in every backend so the value is one you chose.

#Planning a rotation with a second backend

The docs describe several backends with one enabled. Kuma 2.14 rejects a second backend and rejects any change of enabledBackend while mTLS is on. Rotate through the Secrets.

#Replacing the CA in one step

New Secret contents take effect per proxy, as each one renews. Until the last proxy has renewed, old and new proxies reject each other. Use the three stages.

#Year-stamped backend and Secret names

The backend name and the Secret names it points to cannot change while mTLS is on. A name like ca-2026 will still be there in 2030.

#Starting with builtin in production

Moving from builtin to provided is a backend change. The only way is to turn mTLS off for the whole mesh first.

#Inline certificates and keys

Inline CA material ends up in the Mesh object, in Git and in every backup of it. Kuma's docs call it a security risk. Use Secrets.

#Putting the root key in the cluster

If the Secret holds the root key, every zone holds it, and a leak means building a new trust root for the whole mesh. With an intermediate, a leak means a new intermediate from the same offline root.

#Forgetting the CA has an expiry date

A ten-year builtin CA or a one-year intermediate both end. When they do, every proxy fails at once. Monitor the CA's notAfter, not only the workload certificates.

#Checklist

• Production meshes use a provided backend with an intermediate from the start.
• The root key is offline and never stored in a cluster.
• The intermediate has CA:TRUE, pathlen:0 and key usage keyCertSign, cRLSign only.
• CA material is stored in Kuma Secrets, labelled with the mesh at creation, never inline.
• Backend and Secret names carry no date.
• dpCert.rotation.expiration is set explicitly in every backend.
• The workload lifetime leaves margin for your control plane's recovery time.
• Monitoring alerts on workload certificate expiry and on the intermediate's notAfter.
• CA rotation runs in three stages, with every proxy checked between stages, and has been rehearsed in a test mesh.