Securing Kuma zone-to-global traffic, the secure way

#What goes wrong

In a multi-zone Kuma deployment, each zone control plane keeps a gRPC stream open to the global control plane on port 5685. Kuma calls this the Kuma Discovery Service (KDS). Global sends policies, zone ingress addresses and Secrets down to the zones; zones send their data plane inventory, and the policies created in the zone, up.

The Secrets include the builtin CA certificate and private key for each mesh. Whoever reads this stream can issue workload certificates for any service.

Three defaults make that stream weaker than it should be:

• skipVerify. The global control plane starts with an autogenerated, self-signed certificate. The install guide tells zones to skip verification. A zone that does not verify the server will send its connection, and trust the data it receives, from anyone who can intercept the address.
• An open port. The global zone sync Service is a LoadBalancer on port 5685 with no source restriction by default.
• No zone authentication. Kuma's documentation for this link lists firewall rules as the way to decide which zones may connect. Assume that anything that reaches port 5685 and speaks KDS can ask for what zones receive.

#What the docs say

Set --set controlPlane.tls.kdsZoneClient.skipVerify=true because the default global control plane’s certificate is self-signed.

Source: Kuma docs, Deploy a multi-zone global control plane

It’s recommended that the zone control plane verifies the identity of the global control plane.

Source: Kuma docs, Secure access across services

Define firewall rules on the global control plane to only accept connections from known IPs of the zone control planes.

Source: Kuma docs, Secure access across services

The Secrets are synced from global to zones, not the other way around as this would risk exposing sensitive information.

Source: Kuma docs, Manage secrets

The install guide and the security page disagree, and the install guide is the one people copy. The docs also never say what travels over KDS in one place; put the Secrets page next to the multi-zone page and it is clear the link carries the mesh CA.

#The secure configuration

1. Issue the KDS server certificate from your CA. With cert-manager and a private CA issuer:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kds-server-tls
  namespace: kuma-system              # on the global cluster
spec:
  secretName: kds-server-tls
  issuerRef:
    kind: ClusterIssuer
    name: internal-ca
  dnsNames:
    - kds.example.com                 # the name zones will dial
  duration: 2160h                     # 90 days
  renewBefore: 720h                   # 30 days
  privateKey:
    algorithm: ECDSA
    size: 384
    rotationPolicy: Always
  usages:
    - server auth

The Secret has tls.crt and tls.key, which is what controlPlane.tls.kdsGlobalServer.secretName expects.

2. Global control plane.

# global-values.yaml
controlPlane:
  mode: global
  globalZoneSyncService:
    type: LoadBalancer
    loadBalancerSourceRanges:           # zone control planes' egress addresses only
      - 198.51.100.10/32
      - 198.51.100.20/32
  tls:
    kdsGlobalServer:
      secretName: kds-server-tls        # issued by your CA for kds.example.com
    apiServer:
      secretName: api-server-tls        # HTTPS API on 5682 with your certificate
  envVars:
    KUMA_API_SERVER_HTTP_INTERFACE: "127.0.0.1"   # plain HTTP API only on localhost

Binding the plain HTTP API to localhost comes from the same Kuma page; check first that nothing in the cluster calls the control plane on 5681. kubectl port-forward still works, because it connects inside the pod.

3. Each zone control plane. Put only the CA certificate in the zone:

kubectl -n kuma-system create secret generic kds-ca-certs --from-file=ca.crt=internal-ca.crt

# zone-values.yaml
controlPlane:
  mode: zone
  zone: zone-1
  kdsGlobalAddress: grpcs://kds.example.com:5685   # name must match the certificate
  tls:
    kdsZoneClient:
      secretName: kds-ca-certs          # ca.crt of the CA that signed kds-server-tls
      skipVerify: false
ingress:
  enabled: true

These render as KUMA_MULTIZONE_ZONE_GLOBAL_ADDRESS and KUMA_MULTIZONE_ZONE_KDS_ROOT_CA_FILE on the zone control plane, with no KUMA_MULTIZONE_ZONE_KDS_TLS_SKIP_VERIFY.

4. Network. Besides loadBalancerSourceRanges, allow TCP 5685 in the cloud firewall only from the zone control planes' egress addresses. A pod-level policy on the control plane is harder to get right: the load balancer may replace the client address with a node address, and a policy that selects the control plane pod switches all its other ports to default deny too.

5. On Universal zone proxies, short-lived zone tokens. Zone ingress and egress on Kubernetes authenticate with their service account token. On Universal they use a zone token; give it a short validity and keep the revocation list current:

kumactl generate zone-token --zone=zone-1 --scope ingress --valid-for 720h > zone-ingress-token

#Prove it

Run on three lab clusters (kind, Kubernetes 1.34) with Kuma 2.14.3 from the Helm chart: a global control plane and zones zone-1 and zone-2. The global cluster ran Cilium 1.20.2 so that its LoadBalancer Service got an address and honoured loadBalancerSourceRanges; the zone clusters used a NodePort Service for the zone ingress because they had no load balancer. The KDS certificate came from cert-manager with the Certificate on Securing Kuma zone-to-global traffic, and the zones resolved kds.example.com to the global address.

1. The global endpoint presents your certificate, and it verifies. The name is in the SAN (the Certificate above sets no common name, so subject= is empty):

openssl s_client -connect kds.example.com:5685 -servername kds.example.com \
  -verify_hostname kds.example.com -CAfile internal-ca.crt -alpn h2 </dev/null 2>/dev/null \
  | grep -E 'issuer=|Verify return code|^Protocol|Negotiated TLS1.3 group'

issuer=CN=Example Internal CA
Negotiated TLS1.3 group: X25519MLKEM768
Protocol: TLSv1.3
Verify return code: 0 (ok)

With a different CA file, Verify return code: 21 (unable to verify the first certificate); by IP address, Verify return code: 64 (IP address mismatch). The KDS link negotiated the post-quantum hybrid X25519MLKEM768 without any setting.

2. No zone skips verification, and a zone refuses a wrong name:

$ kubectl -n kuma-system get deploy kuma-control-plane -o yaml \
    | grep -A1 -E 'KUMA_MULTIZONE_ZONE_KDS_(TLS_SKIP_VERIFY|ROOT_CA_FILE)|KUMA_MULTIZONE_ZONE_GLOBAL_ADDRESS'
        - name: KUMA_MULTIZONE_ZONE_GLOBAL_ADDRESS
          value: grpcs://kds.example.com:5685
        - name: KUMA_MULTIZONE_ZONE_KDS_ROOT_CA_FILE
          value: /var/run/secrets/kuma.io/kds-client-tls-cert/ca.crt

The same on both zones. With the global address changed to the IP, the zone control plane logged:

ERROR	kds-zone.kds-mux-client	component terminated with an error	{"peer": "grpcs://172.18.251.10:5685", ..., "error": "... authentication handshake failed: tls: failed to verify certificate: x509: cannot validate certificate for 172.18.251.10 because it doesn't contain any IP SANs"}

3. The port is closed to everyone else. First with the placeholder 198.51.100.x ranges still in the values, then with the two zone nodes' addresses:

placeholder ranges:  zone 1 node -> 5685: timeout
                     zone CP log: kds-zone.kds-mux-client component terminated with an error ... Error while dialing
zone addresses:      zone 1 node -> 5685: connected
                     zone 2 node -> 5685: connected
                     another host on the same network -> 5685: timeout

4. Only your zones are connected:

$ kumactl inspect zones
NAME     STATUS   LAST CONNECTED AGO   LAST UPDATED AGO   TOTAL UPDATES   TOTAL ERRORS   ZONE-CP VERSION   BACKEND
zone-1   Online   1m                   1m                 46              0              2.14.3            kubernetes
zone-2   Online   1m                   1m                 45              0              2.14.3            kubernetes

5. The plain HTTP API answers only inside the pod. From another pod on the global cluster:

5681 http:  refused (curl exit 7)
5682 https: 200

kumactl through kubectl port-forward to 5681 kept working.

#Mistakes people make

#Copying skipVerify from the install guide

It is there so the quick start works with a self-signed certificate. In production it removes the only check the zone makes on the global control plane. Replace the certificate, then remove the flag.

#Dialing an IP address

The zone verifies the name in kdsGlobalAddress against the certificate. An IP address in the address and a DNS name in the certificate fail verification, and the "fix" is skipVerify again. Use the DNS name.

#Leaving the zone sync LoadBalancer open to the internet

The Service defaults to LoadBalancer without source ranges. Set loadBalancerSourceRanges and a cloud firewall rule, and check both after every cluster rebuild.

#Treating the KDS link as configuration only

It carries Secrets, including the mesh CA key when you use the builtin CA. Protect it like access to the CA itself.

#Long-lived zone tokens on Universal

Kuma's docs give a zone token a ten-year expiration when none is specified. kumactl generate zone-token in 2.14 requires --valid-for, so choose a short value there. If a token may have leaked, add it to the zone-token-revocations Secret, or rotate the signing key.

#Checklist

• The global KDS server certificate comes from your CA, not the autogenerated one.
• The certificate names the DNS name that zones dial.
• Every zone has controlPlane.tls.kdsZoneClient.secretName with the CA, and skipVerify: false.
• kdsGlobalAddress uses the DNS name, with grpcs://.
• The zone sync Service has loadBalancerSourceRanges limited to zone control planes.
• A cloud firewall allows TCP 5685 only from zone control planes.
• openssl s_client verifies the KDS certificate against your CA.
• kumactl inspect zones lists only your zones.
• Zone tokens on Universal have an explicit, short validity.