Mesh telemetry over OTLP, the secure way

#What goes wrong

A mesh sees every request, so its telemetry is a copy of your traffic's metadata: who called whom, when, how often, on which path. Three things make that copy leak:

• Secrets in paths. Kuma's default HTTP access log format logs %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%, the full path with its query string. Password reset tokens, API keys in URLs and signed download links all land in the log store, where retention is long and access is broad.
• Secrets in traces. Fixing the access log format is not enough. Every span Envoy sends carries an http.url attribute with the full URL, query string included, and MeshTrace has no setting to drop it.
• Redirectable destinations. A MeshOpenTelemetryBackend reads the standard OTEL_EXPORTER_OTLP_* environment variables of the sidecar by default, and they win over the configured address. Whoever can set environment on the sidecar can send that workload's mesh telemetry somewhere else.
• Plaintext on the way out. The sidecar's own export to the collector is not mesh traffic. If the collector is across the network, that stream travels on its own terms. In Kuma 2.14 neither MeshOpenTelemetryBackend nor the policies' OpenTelemetry backends have a TLS field; certificates can be supplied only through the sidecar's OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE and OTEL_EXPORTER_OTLP_CLIENT_KEY environment variables.

#What the docs say

The default env policy is mode: Optional plus precedence: EnvFirst plus allowSignalOverrides: true, so an empty backend reuses those values.

Source: Kuma docs, MeshOpenTelemetryBackend

When environment input must be ignored (regulated backends), set mode: Disabled.

Source: Kuma docs, MeshOpenTelemetryBackend

Inline endpoint fields on those three policies still work in 2.14 but are deprecated and will be removed in 3.0. New deployments should use backendRef.

Source: Kuma docs, MeshOpenTelemetryBackend

NQ: The output will be the request path without the query parameters.

Source: Envoy docs, Substitution Formatter, %PATH(X:Y):Z%

Kuma documents mode: Disabled for "regulated backends", which undersells it: it is the setting that stops a workload from choosing where its mesh telemetry goes. The MeshAccessLog page shows the default format with the full path and never mentions query strings.

#The secure configuration

1. One backend, on the node, not redirectable. With no address, kuma-dp sends to the node's HOST_IP on 4317, where a collector agent runs as a DaemonSet with hostPort: 4317. Telemetry leaves the pod but not the node.

Kuma 2.14 has no TLS setting for this hop in the resource itself, and mode: Disabled also ignores the OTEL_EXPORTER_OTLP_* certificate variables, so the sidecar-to-agent export is plaintext. That is why the agent runs on the same node, and why TLS starts at the agent (step 4).

apiVersion: kuma.io/v1alpha1
kind: MeshOpenTelemetryBackend
metadata:
  name: node-agent
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  endpoint:
    port: 4317
  protocol: grpc
  env:
    mode: Disabled            # pod environment variables cannot redirect mesh telemetry

2. Access logs without query strings, and without headers you did not choose.

apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: access-logs-otel
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  to:
    - targetRef:
        kind: Mesh
      default:
        backends:
          - type: OpenTelemetry
            openTelemetry:
              backendRef:
                kind: MeshOpenTelemetryBackend
                labels:
                  kuma.io/display-name: node-agent
              body:
                kvlistValue:
                  values:
                    - key: method
                      value:
                        stringValue: "%REQ(:METHOD)%"
                    - key: path
                      value:
                        stringValue: "%PATH(NQ:ORIG_OR_PATH)%"   # path without the query string
                    - key: status
                      value:
                        stringValue: "%RESPONSE_CODE%"
                    - key: source
                      value:
                        stringValue: "%KUMA_SOURCE_SERVICE%"
                    - key: destination
                      value:
                        stringValue: "%KUMA_DESTINATION_SERVICE%"
                    - key: duration_ms
                      value:
                        stringValue: "%DURATION%"
                    - key: request_id
                      value:
                        stringValue: "%REQ(X-REQUEST-ID)%"

Never add %REQ(AUTHORIZATION)%, %REQ(COOKIE)% or a full-header dump.

3. Traces sampled, metrics on the same backend.

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: traces-otel
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  default:
    backends:
      - type: OpenTelemetry
        openTelemetry:
          backendRef:
            kind: MeshOpenTelemetryBackend
            labels:
              kuma.io/display-name: node-agent
    sampling:
      overall: 10
---
apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
  name: metrics-otel
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  default:
    backends:
      - type: OpenTelemetry
        openTelemetry:
          backendRef:
            kind: MeshOpenTelemetryBackend
            labels:
              kuma.io/display-name: node-agent
          refreshInterval: 30s

4. The node agent strips query strings from spans and exports with TLS 1.3 and a client certificate.

The transform processor is in the contrib distribution (otel/opentelemetry-collector-contrib); the core otel/opentelemetry-collector image rejects this config with unknown type: "transform".

# OpenTelemetry Collector agent configuration (DaemonSet, hostPort 4317)
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: ${env:MY_POD_IP}:4317     # exposed on the node through hostPort 4317
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15
  batch: {}
  transform/strip-query:
    trace_statements:
      - 'replace_pattern(span.attributes["http.url"], "\\?.*$", "")'   # Envoy puts the full URL, query included, in every span
exporters:
  otlp_grpc/central:
    endpoint: otel-gateway.example.com:4317
    tls:
      ca_file: /etc/otel/tls/ca.crt           # verify the central gateway
      cert_file: /etc/otel/tls/tls.crt        # client certificate: the gateway knows who sends
      key_file: /etc/otel/tls/tls.key
      min_version: "1.3"
    headers:
      X-Scope-OrgID: mesh-default             # tenant, if the backend is multi-tenant
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, transform/strip-query, batch]
      exporters: [otlp_grpc/central]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp_grpc/central]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp_grpc/central]

5. Only cluster pods can send to the agent. hostPort 4317 answers on the node's address, to anyone who can reach the node.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: otel-agent-cluster-only
  namespace: otel
spec:
  podSelector:
    matchLabels:
      app: otel-agent
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector: {}       # any pod in the cluster; nothing from outside
      ports:
        - {protocol: TCP, port: 4317}

Keep a firewall rule that blocks 4317 on node addresses from outside as well. memory_limiter keeps a telemetry flood from taking the node's memory.

The agent is a single point per node: while it restarts, sidecars on that node get connection refused and that telemetry is lost. Roll agent changes out slowly and do not rely on the mesh telemetry stream as your only audit record.

#Prove it

Run on a lab cluster with Kuma 2.14.3 and the resources above, in a test mesh. The central gateway was a second collector with an OTLP receiver requiring TLS 1.3 and a client certificate, printing what it received. A meshed client sent 60 requests like this one:

curl -H "Authorization: Bearer secret-token-123" \
  "http://server.kuma-rot.svc.cluster.local:8080/checkout?card=4111111111111111"

1. What arrives at the gateway, before and after the transform. Counts from the gateway's output for one batch of 60 requests:

without transform/strip-query:  log records: 60  spans: 12  lines with the card number: 12  lines with the token: 0
with transform/strip-query:     log records: 60  spans: 4   lines with the card number: 0   lines with the token: 0

The access log body, as received:

Body: Map({"destination":"server_kuma-rot_svc_8080","duration_ms":"2","method":"GET","path":"/checkout","request_id":"3271578e-17c4-42fb-8ca6-06e0597b45d5","source":"client_kuma-rot_svc","status":"404"})

A span without the transform, then with it:

     -> http.url: Str(http://server.kuma-rot.svc.cluster.local:8080/checkout?card=4111111111111111)
     -> http.url: Str(http://server.kuma-rot.svc.cluster.local:8080/checkout)

All 12 lines with the card number came from spans. The access log was clean from the start.

2. The environment cannot redirect telemetry. A ContainerPatch set OTEL_EXPORTER_OTLP_ENDPOINT on one sidecar to a listener pod, standing in for someone who can change a workload's sidecar environment. The same 30 requests in each mode:

env.mode Disabled:            bytes received by the listener pod: 0
env.mode Optional (default):  bytes received by the listener pod: 132

The 132 bytes start with PRI * HTTP/2.0, the gRPC connection preface: the sidecar was exporting to the address from its environment. Kuma reports the decision in the data plane insight:

kubectl get dataplaneinsight -n <namespace> <pod-name> -o jsonpath='{.spec.openTelemetry}'

Disabled: {"backends":[{"logs":{"blockedReasons":["EnvDisabledByPolicy"],"enabled":true,"envInputPresent":true,"state":"ready"}, ...
Optional: {"backends":[{"logs":{"enabled":true,"envAllowed":true,"envInputPresent":true,"state":"ready"}, ...

envAllowed: true together with envInputPresent: true means a pod's environment is steering its telemetry. With Disabled, look for EnvDisabledByPolicy.

3. The sidecar-to-agent hop is plaintext. A capture on the node, port 4317, during 20 requests:

packets: 88  readable service name: 141  readable /checkout: 24  TLS records: 0

That is why the agent runs on the same node and TLS starts at the agent.

4. The agent's export is TLS 1.3 with a client certificate:

openssl s_client -connect otel-gateway.example.com:4317 -alpn h2 \
  -cert tls.crt -key tls.key -CAfile ca.crt </dev/null 2>/dev/null \
  | grep -E 'Protocol|Verify return code'
openssl s_client -connect otel-gateway.example.com:4317 -alpn h2 \
  -CAfile ca.crt </dev/null 2>&1 | grep -iE 'alert|error' | head -3

Protocol: TLSv1.3
Verify return code: 0 (ok)
...:error:0A00045C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required:...:SSL alert number 116

With -tls1_2: tlsv1 alert protocol version.

5. The node port is closed to the outside:

no policy:            from outside the cluster to node:4317: open     from a pod: open   metric batches in 60s: 6
with the policy:      from outside the cluster to node:4317: blocked  from a pod: open   metric batches in 60s: 7

#Mistakes people make

#Logging the default format

The default HTTP format logs the full path, query included. Build the body yourself with %PATH(NQ:ORIG_OR_PATH)% and only the fields you need.

#Cleaning the access log and not the traces

%PATH(NQ:...)% fixes the access log only. Envoy's spans carry the full URL in http.url. Strip it in the node agent before anything leaves the node.

#Leaving env mode on Optional

With the default EnvFirst, the sidecar's environment overrides the configured address. Set mode: Disabled unless you rely on environment input, and then prefer ExplicitFirst so the address stays yours.

#A central collector across the network, in clear

The sidecar's export is not mesh traffic. Keep the first hop on the node and put TLS on the hop that crosses the network.

#Inline endpoints in every policy

Inline endpoints are deprecated, and a copy per policy means one of them is always out of date. Use one MeshOpenTelemetryBackend and backendRef.

#Tracing everything forever

100 percent sampling multiplies storage and the amount of request metadata you keep. Sample traces; keep access logs for audit, with sensible retention.

#Checklist

• MeshMetric, MeshTrace and MeshAccessLog use backendRef to one MeshOpenTelemetryBackend.
• The backend sets env.mode: Disabled (or ExplicitFirst when env input is required).
• A collector agent runs on every node, and the backend sends to the node address.
• The access log body uses %PATH(NQ:ORIG_OR_PATH)% and no authorization or cookie headers.
• The agent strips the query string from http.url in every span.
• Traces are sampled.
• The agent exports with TLS 1.3, a client certificate and a tenant header.
• The agent has memory_limiter and a network policy that limits who can send to it.
• Stored logs and traces contain no query strings.