Service mesh

Default-deny MeshTrafficPermission, the secure way

Your Kuma mesh has mTLS, certificates rotate, the padlock icons are all green, and any pod in any namespace can still call the payments API. Somewhere in kuma-system there is a policy called allow-all, and it was only supposed to be there for the demo.

The short answer

With mTLS on, Kuma denies traffic that no MeshTrafficPermission allows. Keep it that way: delete any allow-all, add an explicit mesh-wide Deny in the system namespace, and let each service allow its real callers by kuma.io/service in its own namespace. Roll out removals with AllowWithShadowDeny, and restrict who may write these policies.

Updated Houssam Hammoudi, CTOTested with Kuma 2.14.3 on Kubernetes 1.34.0 (kind), strict builtin mTLS

On this page
  1. What goes wrong
  2. What the docs say
  3. The secure configuration
  4. Prove it
  5. Mistakes people make
  6. Checklist

What goes wrong

mTLS gives every workload an identity. MeshTrafficPermission decides what that identity may call. Encryption without authorization means an attacker in any meshed pod can reach every service over a perfectly encrypted connection.

Kuma's default is actually secure: once mTLS is enabled, traffic that no MeshTrafficPermission allows is denied. Meshes lose that default in three common ways:

  • The allow-all. Kuma 2.14 creates no permission when it creates a Mesh (the legacy allow-all-<mesh> TrafficPermission appears only when KUMA_DEFAULTS_CREATE_MESH_ROUTING_RESOURCES is true; the default is false). So the first time mTLS is enabled, traffic stops, and the fix is a policy that allows Mesh to Mesh. It stays.
  • Permissive mode. MeshTrafficPermission works on mTLS identities. Kuma's docs state that mTLS has to be enabled for the policy to work. A client that sends plaintext brings no certificate to check.
  • Anyone can write the policy. In Kuma 2.x, a MeshTrafficPermission created in an application namespace is a workload-owner policy. Kuma ranks policies first by the top-level targetRef (a Dataplane target outranks Mesh), then by origin, then by role, and a workload-owner policy outranks a system policy in kuma-system. Your mesh-wide deny is the lowest layer. Whoever can create MeshTrafficPermissions in a namespace decides who can call the workloads in it.

What the docs say

Also remember that by default when mTLS is enabled all traffic is denied unless a MeshTrafficPermission policy is being configured to explicitly allow traffic across proxies.

Source: Kuma docs, Mutual TLS

Mutual TLS has to be enabled to make MeshTrafficPermission work.

Source: Kuma docs, MeshTrafficPermission

AllowWithShadowDeny - same as Allow but will log as if request is denied, this is useful for rolling new restrictive policies without breaking things.

Source: Kuma docs, MeshTrafficPermission

For security reasons it’s not possible to customize the kuma.io/service in Kubernetes.

Source: Kuma docs, Deploy a multi-zone global control plane

The policy page shows allow-all and deny-all side by side as equal examples. The priority rules that make a namespace policy beat a system policy are on the policies introduction page, and neither page says that this makes Kubernetes RBAC on meshtrafficpermissions part of your authorization model.

The secure configuration

1. Strict mTLS on the mesh. Permissions need identities. Set the Mesh backend to mode: STRICT and do not leave Permissive in any MeshTLS (see Kuma multi-zone mTLS with a builtin CA).

2. Remove every allow-all.

bash
kubectl get meshtrafficpermissions -A -o yaml \
  | grep -B20 'action: Allow' | grep -E 'name:|namespace:|kind: Mesh$'

Delete any policy that allows kind: Mesh in from without a narrower targetRef on top. If you need it during a migration, change its action to AllowWithShadowDeny first and watch the logs.

3. An explicit mesh-wide deny. Kuma already denies by default with mTLS on; this policy writes that down, so a future allow-all at the same level is a visible change instead of a quiet one.

yaml
# System policy (kuma-system, on the global control plane in multi-zone).
apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  name: deny-all
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  from:
    - targetRef:
        kind: Mesh
      default:
        action: Deny

4. Each service allows its callers, in its own namespace.

yaml
# Workload-owner policy in the server's namespace: who may call backend.
apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  name: backend-inbound
  namespace: app
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: backend
  from:
    - targetRef:
        kind: MeshSubset
        tags:
          # Derived from the Service name, namespace and port on Kubernetes;
          # a workload cannot set it to impersonate another service.
          kuma.io/service: frontend_app_svc_8080
      default:
        action: Allow
    - targetRef:
        kind: MeshSubset
        tags:
          kuma.io/service: reports_app_svc_8080
      default:
        action: AllowWithShadowDeny   # being removed: logs as denied, still allows

The order inside from matters: a later rule wins over an earlier one for a client that matches both. Put narrow denies after broad allows.

5. Control who writes permissions. Only the platform team, or a reviewed GitOps pipeline, gets write access to meshtrafficpermissions:

yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: mesh-traffic-permission-admin
rules:
  - apiGroups: ["kuma.io"]
    resources: ["meshtrafficpermissions"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: mesh-traffic-permission-admin
  namespace: app
subjects:
  - kind: Group
    name: mesh-platform
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: mesh-traffic-permission-admin
  apiGroup: rbac.authorization.k8s.io

Check that no broad role also grants it, for example a custom ClusterRole aggregated into edit, or a CI role with kuma.io and * resources.

Prove it

Run on a lab cluster (Kubernetes 1.34, Kuma 2.14.3, strict builtin mTLS) with the two policies and the RBAC above, and four meshed services in app: backend, frontend, billing and reports. Real output:

1. Only the deny-all and the explicit allows exist:

bash
kubectl get meshtrafficpermissions -A \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,TARGET:.spec.targetRef.kind
text
NS            NAME              TARGET
app           backend-inbound   Dataplane
kuma-system   deny-all          Mesh

2. Both apply to the backend's sidecar:

bash
kumactl inspect dataplane backend-7d59fcd4cd-gwtzk.app --type=policies
text
DATAPLANE:
  MeshCircuitBreaker
    mesh-circuit-breaker-all-default.kuma-system
  MeshRetry
    mesh-retry-all-default.kuma-system
  MeshTimeout
    mesh-timeout-all-default.kuma-system
  MeshTrafficPermission
    deny-all.kuma-system
backend-inbound.app

3. The allowed caller gets through; everyone else gets 403:

bash
kubectl -n app exec deploy/frontend -c app -- \
  curl -s -o /dev/null -w '%{http_code}\n' http://backend.app.svc.cluster.local:8080/
kubectl -n app exec deploy/billing -c app -- \
  curl -s -o /dev/null -w '%{http_code}\n' http://backend.app.svc.cluster.local:8080/
kubectl -n app exec deploy/reports -c app -- \
  curl -s -o /dev/null -w '%{http_code}\n' http://backend.app.svc.cluster.local:8080/
text
200
403
200

reports still gets 200 because its rule is AllowWithShadowDeny: allowed while you watch the logs for who still depends on it, then removed.

4. Only the platform group can write the policies:

bash
kubectl auth can-i create meshtrafficpermissions.kuma.io -n app [email protected]
kubectl auth can-i create meshtrafficpermissions.kuma.io -n app [email protected] --as-group=mesh-platform
kubectl auth can-i create meshtrafficpermissions.kuma.io -n app \
  --as=system:serviceaccount:ci:deployer
text
no
yes
no

Mistakes people make

The demo policy that became production

An allow-all created "to get traffic flowing" is the most common reason a mesh with mTLS has no authorization. Search for it by action and target, not by name.

Relying on the system-namespace deny alone

A workload-owner policy in an application namespace outranks the system policy. If a team can write MeshTrafficPermissions, the deny in kuma-system is a default, not a guarantee. RBAC is the guarantee.

Matching on free-form tags

from tags such as team: payments come from pod labels, which the pod's creator controls. Match callers on kuma.io/service, which Kuma derives on Kubernetes and does not let workloads customize.

Allowing a whole namespace by habit

Allowing every service in a namespace to call a database because "they are all ours" turns one compromised pod into access to the database. Allow the callers you saw, one by one.

Removing access without a shadow period

Switching a caller from Allow to Deny in one step breaks the caller you did not know about. AllowWithShadowDeny logs what would be denied while it still allows; switch to Deny when the logs are quiet.

Checklist

  • The Mesh has mTLS in mode: STRICT and no MeshTLS sets Permissive.
  • No MeshTrafficPermission allows Mesh to Mesh.
  • A deny-all MeshTrafficPermission exists in the system namespace.
  • Every service has a workload-owner policy that allows named callers by kuma.io/service.
  • Callers being removed pass through AllowWithShadowDeny before Deny.
  • Only the platform team or a reviewed pipeline can write meshtrafficpermissions.
  • kumactl inspect dataplane --type=policies shows the expected permissions on each proxy.
  • A caller without permission gets 403 or a reset connection.

mTLS tells every service who is calling. MeshTrafficPermission is where you finally get to say "no".

H2-CSPE

Learn it on a live range

Service mesh and gateways, in Secure Platform Engineering: a real host in your browser, and every objective checked on the machine.

Start free

The Dome

Want it run for you?

The Dome puts post-quantum TLS, a WAF that blocks, signed DNS and a zero-trust mesh in front of your application. Tell us what you run.

See the Dome