Default-deny MeshTrafficPermission, the secure way
Your Kuma mesh has mTLS, certificates rotate, the padlock icons are all green, and any pod in any namespace can still call the payments API. Somewhere in kuma-system there is a policy called allow-all, and it was only supposed to be there for the demo.
The short answer
With mTLS on, Kuma denies traffic that no MeshTrafficPermission allows. Keep it that way: delete any allow-all, add an explicit mesh-wide Deny in the system namespace, and let each service allow its real callers by kuma.io/service in its own namespace. Roll out removals with AllowWithShadowDeny, and restrict who may write these policies.
On this page
What goes wrong
mTLS gives every workload an identity. MeshTrafficPermission decides what that identity may call. Encryption without authorization means an attacker in any meshed pod can reach every service over a perfectly encrypted connection.
Kuma's default is actually secure: once mTLS is enabled, traffic that no MeshTrafficPermission allows is denied. Meshes lose that default in three common ways:
- The allow-all. Kuma 2.14 creates no permission when it creates a Mesh
(the legacy
allow-all-<mesh>TrafficPermission appears only whenKUMA_DEFAULTS_CREATE_MESH_ROUTING_RESOURCESistrue; the default isfalse). So the first time mTLS is enabled, traffic stops, and the fix is a policy that allowsMeshtoMesh. It stays. - Permissive mode. MeshTrafficPermission works on mTLS identities. Kuma's docs state that mTLS has to be enabled for the policy to work. A client that sends plaintext brings no certificate to check.
- Anyone can write the policy. In Kuma 2.x, a MeshTrafficPermission
created in an application namespace is a workload-owner policy. Kuma ranks
policies first by the top-level
targetRef(aDataplanetarget outranksMesh), then by origin, then by role, and a workload-owner policy outranks a system policy inkuma-system. Your mesh-wide deny is the lowest layer. Whoever can create MeshTrafficPermissions in a namespace decides who can call the workloads in it.
What the docs say
Also remember that by default when mTLS is enabled all traffic is denied unless a MeshTrafficPermission policy is being configured to explicitly allow traffic across proxies.
Source: Kuma docs, Mutual TLS
Mutual TLS has to be enabled to make MeshTrafficPermission work.
Source: Kuma docs, MeshTrafficPermission
AllowWithShadowDeny - same as Allow but will log as if request is denied, this is useful for rolling new restrictive policies without breaking things.
Source: Kuma docs, MeshTrafficPermission
For security reasons it’s not possible to customize the kuma.io/service in Kubernetes.
Source: Kuma docs, Deploy a multi-zone global control plane
The policy page shows allow-all and deny-all side by side as equal examples.
The priority rules that make a namespace policy beat a system policy are on
the policies introduction page, and neither page says that this makes
Kubernetes RBAC on meshtrafficpermissions part of your authorization model.
The secure configuration
1. Strict mTLS on the mesh. Permissions need identities. Set the Mesh
backend to mode: STRICT and do not leave Permissive in any MeshTLS (see
Kuma multi-zone mTLS with a builtin CA).
2. Remove every allow-all.
kubectl get meshtrafficpermissions -A -o yaml \
| grep -B20 'action: Allow' | grep -E 'name:|namespace:|kind: Mesh$'Delete any policy that allows kind: Mesh in from without a narrower
targetRef on top. If you need it during a migration, change its action to
AllowWithShadowDeny first and watch the logs.
3. An explicit mesh-wide deny. Kuma already denies by default with mTLS on; this policy writes that down, so a future allow-all at the same level is a visible change instead of a quiet one.
# System policy (kuma-system, on the global control plane in multi-zone).
apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
name: deny-all
namespace: kuma-system
labels:
kuma.io/mesh: default
spec:
targetRef:
kind: Mesh
from:
- targetRef:
kind: Mesh
default:
action: Deny4. Each service allows its callers, in its own namespace.
# Workload-owner policy in the server's namespace: who may call backend.
apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
name: backend-inbound
namespace: app
labels:
kuma.io/mesh: default
spec:
targetRef:
kind: Dataplane
labels:
app: backend
from:
- targetRef:
kind: MeshSubset
tags:
# Derived from the Service name, namespace and port on Kubernetes;
# a workload cannot set it to impersonate another service.
kuma.io/service: frontend_app_svc_8080
default:
action: Allow
- targetRef:
kind: MeshSubset
tags:
kuma.io/service: reports_app_svc_8080
default:
action: AllowWithShadowDeny # being removed: logs as denied, still allowsThe order inside from matters: a later rule wins over an earlier one for a
client that matches both. Put narrow denies after broad allows.
5. Control who writes permissions. Only the platform team, or a
reviewed GitOps pipeline, gets write access to meshtrafficpermissions:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: mesh-traffic-permission-admin
rules:
- apiGroups: ["kuma.io"]
resources: ["meshtrafficpermissions"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: mesh-traffic-permission-admin
namespace: app
subjects:
- kind: Group
name: mesh-platform
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: mesh-traffic-permission-admin
apiGroup: rbac.authorization.k8s.ioCheck that no broad role also grants it, for example a custom ClusterRole
aggregated into edit, or a CI role with kuma.io and * resources.
Prove it
Run on a lab cluster (Kubernetes 1.34, Kuma 2.14.3, strict builtin mTLS) with
the two policies and the RBAC above, and four meshed services in app:
backend, frontend, billing and reports. Real output:
1. Only the deny-all and the explicit allows exist:
kubectl get meshtrafficpermissions -A \
-o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,TARGET:.spec.targetRef.kindNS NAME TARGET
app backend-inbound Dataplane
kuma-system deny-all Mesh2. Both apply to the backend's sidecar:
kumactl inspect dataplane backend-7d59fcd4cd-gwtzk.app --type=policiesDATAPLANE:
MeshCircuitBreaker
mesh-circuit-breaker-all-default.kuma-system
MeshRetry
mesh-retry-all-default.kuma-system
MeshTimeout
mesh-timeout-all-default.kuma-system
MeshTrafficPermission
deny-all.kuma-system
backend-inbound.app3. The allowed caller gets through; everyone else gets 403:
kubectl -n app exec deploy/frontend -c app -- \
curl -s -o /dev/null -w '%{http_code}\n' http://backend.app.svc.cluster.local:8080/
kubectl -n app exec deploy/billing -c app -- \
curl -s -o /dev/null -w '%{http_code}\n' http://backend.app.svc.cluster.local:8080/
kubectl -n app exec deploy/reports -c app -- \
curl -s -o /dev/null -w '%{http_code}\n' http://backend.app.svc.cluster.local:8080/200
403
200reports still gets 200 because its rule is AllowWithShadowDeny: allowed
while you watch the logs for who still depends on it, then removed.
4. Only the platform group can write the policies:
kubectl auth can-i create meshtrafficpermissions.kuma.io -n app [email protected]
kubectl auth can-i create meshtrafficpermissions.kuma.io -n app [email protected] --as-group=mesh-platform
kubectl auth can-i create meshtrafficpermissions.kuma.io -n app \
--as=system:serviceaccount:ci:deployerno
yes
noMistakes people make
The demo policy that became production
An allow-all created "to get traffic flowing" is the most common reason a mesh with mTLS has no authorization. Search for it by action and target, not by name.
Relying on the system-namespace deny alone
A workload-owner policy in an application namespace outranks the system
policy. If a team can write MeshTrafficPermissions, the deny in kuma-system
is a default, not a guarantee. RBAC is the guarantee.
Matching on free-form tags
from tags such as team: payments come from pod labels, which the pod's
creator controls. Match callers on kuma.io/service, which Kuma derives on
Kubernetes and does not let workloads customize.
Allowing a whole namespace by habit
Allowing every service in a namespace to call a database because "they are all ours" turns one compromised pod into access to the database. Allow the callers you saw, one by one.
Removing access without a shadow period
Switching a caller from Allow to Deny in one step breaks the caller you did
not know about. AllowWithShadowDeny logs what would be denied while it still
allows; switch to Deny when the logs are quiet.
Checklist
- The Mesh has mTLS in
mode: STRICTand no MeshTLS setsPermissive. - No MeshTrafficPermission allows
MeshtoMesh. - A
deny-allMeshTrafficPermission exists in the system namespace. - Every service has a workload-owner policy that allows named callers by
kuma.io/service. - Callers being removed pass through
AllowWithShadowDenybeforeDeny. - Only the platform team or a reviewed pipeline can write
meshtrafficpermissions. kumactl inspect dataplane --type=policiesshows the expected permissions on each proxy.- A caller without permission gets
403or a reset connection.
mTLS tells every service who is calling. MeshTrafficPermission is where you finally get to say "no".
H2-CSPE
Learn it on a live range
Service mesh and gateways, in Secure Platform Engineering: a real host in your browser, and every objective checked on the machine.
Start freeThe Dome
Want it run for you?
The Dome puts post-quantum TLS, a WAF that blocks, signed DNS and a zero-trust mesh in front of your application. Tell us what you run.
See the Dome