Kuma multi-zone mTLS with a builtin CA, the secure way

#What goes wrong

A multi-zone Kuma mesh has one global control plane and one control plane per zone. Services in one zone reach services in another through the zone ingress. The zone ingress routes on the TLS Server Name Indication, so cross-zone traffic only works with mTLS on.

That forces the decision, and it is often made in a hurry:

• Allow-all permissions. With mTLS on, Kuma denies all traffic that no MeshTrafficPermission allows. The quick fix is one allow-all policy, which gives every workload in every zone access to every other workload.
• Permissive mode. Servers accept plaintext as well as mTLS. Any pod that skips its sidecar, or any host that reaches a pod IP, talks to the service with no identity at all.
• An unguarded CA key. The builtin CA generates a root certificate and a private key and stores both as Kuma Secrets. In multi-zone, Secrets sync from global to every zone. On Kubernetes that means a Secret in kuma-system on every zone cluster. Anyone who can read it can issue a certificate for any service in the mesh, in any zone.
• Long certificates or a fragile zone. Workload certificates are issued by the zone control plane. If they last long, a stolen one stays useful. If they are short and the zone control plane is down for longer than the remaining lifetime, traffic stops.

#What the docs say

mTLS is mandatory to enable cross-zone service communication.

Source: Kuma docs, Deploy a multi-zone global control plane

Always make sure that a MeshTrafficPermission resource is present before enabling mTLS in a Mesh in order to avoid unexpected traffic interruptions caused by a lack of authorization between proxies.

Source: Kuma docs, Mutual TLS

The Secrets are synced from global to zones, not the other way around as this would risk exposing sensitive information.

Source: Kuma docs, Manage secrets

On mTLS enabled meshes, a data plane proxy may fail to refresh its client certificate prior to expiry (defaults to 24 hours), thus causing traffic from/to this data plane to fail.

Source: Kuma docs, Multi-zone deployment

The docs never put the second and third quotes together: the builtin CA's private key is one of the Secrets that syncs, so every zone cluster holds the key that can mint any identity in the mesh. The docs also disagree on the default certificate lifetime: the multi-zone page says 24 hours, and the Mutual TLS page says 30 days. The 2.14 source uses 24 hours when dpCert.rotation.expiration is empty. Set it explicitly.

#The secure configuration

1. Install the control planes with verified KDS TLS. The zone control plane must verify the global control plane's certificate. The install guide sets skipVerify=true; do not copy that. Securing this link in full is covered in Securing Kuma zone-to-global traffic.

# global-values.yaml (Helm, global cluster)
controlPlane:
  mode: global
  globalZoneSyncService:
    type: LoadBalancer
    loadBalancerSourceRanges:       # only the zone control planes' egress IPs
      - 198.51.100.10/32
      - 198.51.100.20/32
  tls:
    kdsGlobalServer:
      secretName: kds-server-tls    # certificate from your own CA

# zone-values.yaml (Helm, one per zone cluster)
controlPlane:
  mode: zone
  zone: zone-1
  kdsGlobalAddress: grpcs://kds.example.com:5685
  tls:
    kdsZoneClient:
      secretName: kds-ca-certs      # ca.crt that signed the global KDS certificate
      skipVerify: false             # verify the global control plane
ingress:
  enabled: true                     # cross-zone traffic arrives here, mTLS only
egress:
  enabled: true                     # cross-zone and external traffic leaves here

2. Apply permissions first, on the global control plane. One narrow permission per real caller, never an allow-all. Default-deny patterns are in Default-deny MeshTrafficPermission.

apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  name: allow-frontend-to-backend
  namespace: kuma-system            # system namespace on global: syncs to all zones
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: backend
      k8s.kuma.io/namespace: app
  from:
    - targetRef:
        kind: MeshSubset
        tags:
          # kuma.io/service is derived from the Service name, namespace and
          # port on Kubernetes; workloads cannot set it to impersonate.
          kuma.io/service: frontend_app_svc_8080
      default:
        action: Allow

3. Enable strict mTLS with the builtin CA, on the global control plane.

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
      - name: ca-1
        type: builtin
        mode: STRICT                  # never PERMISSIVE across zones
        dpCert:
          rotation:
            expiration: 24h           # short-lived workload certificates
        conf:
          caCert:
            RSAbits: 4096
            expiration: 10y
  routing:
    zoneEgress: true                  # cross-zone traffic leaves through the zone egress
    localityAwareLoadBalancing: true  # prefer the local zone, fail over to others
  networking:
    outbound:
      passthrough: false              # no traffic to destinations the mesh does not know
---
apiVersion: kuma.io/v1alpha1
kind: MeshTLS
metadata:
  name: strict-tls13
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  rules:
    - default:
        mode: Strict
        tlsVersion:
          min: TLS13
          max: TLS13

Kuma renews a workload certificate after four fifths of its lifetime. With 24 hours, a zone control plane can be down for a little under five hours before certificates start to expire. Size the lifetime to your real recovery time for a zone control plane, and alert well before it.

4. Guard the CA key on every cluster. The key lives in default.ca-builtin-key-ca-1 in kuma-system, on global and on each zone. Remove broad Secret read access in kuma-system (for example, cluster-wide get secrets granted to operators or CI) and keep a narrow break-glass binding instead:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kuma-ca-breakglass
  namespace: kuma-system
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["default.ca-builtin-cert-ca-1", "default.ca-builtin-key-ca-1"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kuma-ca-breakglass
  namespace: kuma-system
subjects:
  - kind: Group
    name: mesh-breakglass
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: kuma-ca-breakglass
  apiGroup: rbac.authorization.k8s.io

If the key must never sit in a zone cluster at all, the builtin CA is the wrong choice; see Kuma certificate rotation and CA choices.

#Prove it

Run on three lab clusters (kind, Kubernetes 1.34) with Kuma 2.14.3 from the Helm chart: a global control plane and zones zone-1 and zone-2. The global cluster ran Cilium 1.20.2 so that its LoadBalancer Service got an address and honoured loadBalancerSourceRanges; the zone clusters used a NodePort Service for the zone ingress because they had no load balancer. The KDS certificate came from cert-manager with the Certificate on Securing Kuma zone-to-global traffic, and the zones resolved kds.example.com to the global address. frontend and reports ran in zone 1, backend in zone 2, all in the namespace app.

1. The mesh has mTLS on, with the right backend (on global):

$ kumactl get meshes
NAME      mTLS           LOCALITY   ZONEEGRESS   AGE
default   builtin/ca-1   on         on           13m

The global control plane created no allow-all permission for the mesh; allow-frontend-to-backend was the only MeshTrafficPermission.

2. Every data plane proxy holds a current certificate:

kumactl inspect dataplanes --mesh default

The certificate columns, for the three proxies (from -o json):

backend-bd5678f9c-wt66m-...    ca-1  expires 2026-09-26T10:54:20Z  regenerated 2026-09-25T10:54:20Z
frontend-7fdf95ff69-dz4ld-...  ca-1  expires 2026-09-26T10:54:17Z  regenerated 2026-09-25T10:54:17Z
reports-6f95d86c5d-nhw4n-...   ca-1  expires 2026-09-26T10:54:18Z  regenerated 2026-09-25T10:54:18Z

A proxy with never under CERT REGENERATED AGO and - under CERT EXPIRATION has no identity.

3. Every zone uses the same root, and only the control plane can read it:

kubectl -n kuma-system get secret default.ca-builtin-cert-ca-1   -o jsonpath='{.data.value}' | base64 -d   | openssl x509 -noout -fingerprint -sha256 -enddate

global  sha256 Fingerprint=50:F6:CC:89:AC:DB:3D:71:...:89:76:FF:20 notAfter=Sep 22 10:54:14 2036 GMT
zone1   sha256 Fingerprint=50:F6:CC:89:AC:DB:3D:71:...:89:76:FF:20 notAfter=Sep 22 10:54:14 2036 GMT
zone2   sha256 Fingerprint=50:F6:CC:89:AC:DB:3D:71:...:89:76:FF:20 notAfter=Sep 22 10:54:14 2036 GMT

default.ca-builtin-key-ca-1 existed on global and on both zones. On a zone, with the break-glass Role above applied:

$ kubectl auth can-i get secrets -n kuma-system [email protected]
no
$ kubectl auth can-i get secrets -n kuma-system --as=system:serviceaccount:ci:deployer
no
$ kubectl auth can-i get secrets -n kuma-system --as=system:serviceaccount:kuma-system:kuma-control-plane
yes
$ kubectl auth can-i get secret/default.ca-builtin-key-ca-1 -n kuma-system [email protected] --as-group=mesh-breakglass
yes
$ kubectl auth can-i get secret/kuma-tls-cert -n kuma-system [email protected] --as-group=mesh-breakglass
no

4. Services report certificates from the right backend:

$ kubectl get serviceinsight all-services-default -o json | jq -c '.spec.services | to_entries[] | ...'
{"service":"backend_app_svc_8080","issuedBackends":{"ca-1":1},"status":"online"}
{"service":"frontend_app_svc_8080","issuedBackends":{"ca-1":1},"status":"online"}
{"service":"reports_app_svc_8080","issuedBackends":{"ca-1":1},"status":"online"}

5. Cross-zone traffic works, a denied caller is refused, and a label cannot impersonate:

frontend (zone 1) -> http://backend.app.svc.8080.mesh:80/ : 200
frontend (zone 1) -> http://backend_app_svc_8080.mesh:80/ : 200
reports  (zone 1) -> backend (zone 2)                    : 403

A pod labelled kuma.io/service: frontend_app_svc_8080, with no Service of its own:

imposter dataplane service tags: ["imposter_app_svc"]
imposter -> backend: 403

Kuma ignored the label and derived the name itself.

6. Only TLS 1.3 between proxies. The backend proxy's config dump, and a TLS 1.2 and a TLS 1.3 client against its inbound port:

{"tls_minimum_protocol_version":"TLSv1_3","tls_maximum_protocol_version":"TLSv1_3"}
tls1.2: alert protocol version
tls1.3: Protocol: TLSv1.3

#Mistakes people make

#The allow-all that was only for today

An allow-all MeshTrafficPermission makes mTLS encryption without authorization. Write the narrow permissions first; Kuma's AllowWithShadowDeny action logs what a deny would block while it still allows, which helps you find callers you forgot.

#Permissive as a permanent state

Permissive accepts plaintext on every server. In a multi-zone mesh that means any pod on any node can talk to a service without a certificate. Use it during a migration window only, then switch to STRICT and check.

#Creating the Mesh on a zone

The Mesh resource syncs only from global to the zones, and a zone control plane does not accept it from anywhere else. Create the Mesh on global, and create the mesh-wide MeshTLS and MeshTrafficPermissions in kuma-system on global too, so every zone gets the same configuration.

#Forgetting the key is in every zone

People harden the global cluster and leave zone clusters with broad Secret access for CI and operators. The builtin CA key is there too. Audit kuma-system Secret access on every cluster, not only global.

#Matching on labels anyone can set

A permission that allows from tags such as env: prod trusts pod labels, and anyone who can create a pod can set those. On Kubernetes, Kuma derives kuma.io/service from the Service and says it cannot be customized for security reasons. Match on that.

#Checklist

• MeshTrafficPermissions for every real caller exist on global before mTLS is enabled.
• No allow-all MeshTrafficPermission exists in the mesh.
• The Mesh is created on the global control plane with a builtin backend in mode: STRICT.
• dpCert.rotation.expiration is set explicitly and is shorter than your zone control plane recovery time allows.
• A mesh-wide MeshTLS sets mode: Strict and TLS 1.3.
• networking.outbound.passthrough is false.
• Zone control planes verify the global KDS certificate (skipVerify: false).
• The global zone sync service accepts only the zone control planes' addresses.
• Only the Kuma control plane and one break-glass group can read Secrets in kuma-system on every cluster.
• The CA fingerprint matches across all zones.
• kumactl inspect dataplanes shows a valid certificate for every proxy.
• A caller without permission is refused across zones.