Hubble for network forensics, the secure way

#What goes wrong

Hubble records every flow Cilium handles, with Kubernetes identities attached: namespace, pod, labels, DNS names, verdict. For an investigation that is better than any packet capture. But by default it is a live tool, not a record:

• A ring buffer, per node. Each agent keeps the most recent flows in memory (4095 events by default). On a busy node that is minutes. After an agent restart it is nothing.
• No export by default. The exporter that writes flows to a file is off until you set a path, and its default rotation keeps 10 MB times 5 files per node.
• Sensitive content. With L7 visibility, flows can carry URLs, query strings and headers. Evidence you keep can itself leak tokens.
• A valuable API. The Hubble API shows who talks to whom across the cluster. It listens on every node, and it is protected by mutual TLS only as long as nobody turns that off to fix a certificate problem.

#What the docs say

Hubble Exporter is a feature of cilium-agent that lets you write Hubble flows to a file for later consumption as logs.

Source: Cilium docs, Configuring Hubble exporter

hubble.export.static.fileMaxBackups: number of rotated Hubble export files to keep. (default 5)

Source: Cilium docs, Configuring Hubble exporter

Setting this value to false is highly discouraged as the Hubble API provides access to potentially sensitive network flow metadata and is exposed on the host network.

Source: Cilium docs, Helm Reference, hubble.tls.enabled

Enables redacting URL query (GET) parameters.

Source: Cilium docs, Helm Reference, hubble.redact.http.urlQuery

The exporter page treats export as a logging feature and suggests filters and field masks for performance. For forensics the priorities are different: keep the fields you will need to prove what happened, and get them off the node quickly.

#The secure configuration

1. Cilium Helm values for Hubble as a recorder.

hubble:
  enabled: true
  eventBufferCapacity: "65535"          # largest ring buffer: more history for live queries
  tls:
    enabled: true                       # mutual TLS on the per-node Hubble API (the default; never turn off)
    auto:
      enabled: true
      method: cronJob
  relay:
    enabled: true
    tls:
      server:
        enabled: true                   # TLS for clients of Hubble Relay too
  redact:
    enabled: true
    http:
      urlQuery: true                    # drop query strings from L7 flows
      userInfo: true                    # drop user:password@ from URLs
      headers:
        deny:
          - Authorization
          - Proxy-Authorization
          - Cookie
          - Set-Cookie
  export:
    static:
      enabled: true
      filePath: /var/run/cilium/hubble/events.log
      fileMaxSizeMb: 50
      fileMaxBackups: 10                # a buffer for the shipper, not the archive
      fieldMask:
        - time
        - uuid
        - verdict
        - drop_reason_desc
        - IP
        - l4
        - source
        - destination
        - destination_names
        - source_service
        - destination_service
        - traffic_direction
        - is_reply
        - node_name
        - Type
        - event_type                    # the CLI needs it to print the event type
        - l7
        - Summary

These render as hubble-export-file-path, hubble-export-fieldmask, hubble-redact-enabled and hubble-disable-tls: "false" in the cilium-config ConfigMap.

Keep source and destination whole in the mask: they carry the labels and identities that let you say which workload did it, even after its pods are gone.

2. Ship the file off the node. Run a log agent (Grafana Alloy, Fluent Bit, Vector) as a DaemonSet that tails /var/run/cilium/hubble/events.log* on each node and sends it to your log store or SIEM. Retention and immutability are set there, not on the node.

3. Restrict who can query. The Hubble API is on the host network behind mutual TLS; Relay aggregates it. Give Relay access to investigators only (through the tailnet or a bastion, never a public Service), and keep the Hubble UI behind single sign-on if you deploy it.

4. Keep time right. Evidence from many nodes is only useful if the clocks agree. Keep NTP healthy on every node and record times in UTC.

#Prove it

Run on a lab cluster after helm upgrade cilium cilium/cilium --version 1.20.2 --reset-then-reuse-values -f hubble-values.yaml.

$ kubectl -n kube-system get cm cilium-config -o json | jq -r '.data | ...'
hubble-disable-tls: false
hubble-event-buffer-capacity: 65535
hubble-export-fieldmask: time uuid verdict drop_reason_desc IP l4 source destination destination_names source_service destination_service traffic_direction is_reply node_name Type event_type l7 Summary
hubble-export-file-path: /var/run/cilium/hubble/events.log
hubble-redact-enabled: true
hubble-redact-http-headers-deny: Authorization Proxy-Authorization Cookie Set-Cookie
hubble-redact-http-urlquery: true
hubble-redact-http-userinfo: true

1. Offline queries work on exported flows. First the two flow lines from the "Configuring Hubble exporter" page of the Cilium docs, saved as flow-from-cilium-docs.json:

$ hubble observe --input-file flow-from-cilium-docs.json --time-format RFC3339
2023-08-21T12:12:13Z: fe80::64d8:8aff:fe72:fc14 (unknown) <> ff02::2 (unknown) UNKNOWN DROPPED ()
2023-08-21T12:12:18Z: default/xwing:44916 (unknown) <> default/deathstar-7848d6c4d5-th9v2:80 (unknown) UNKNOWN DROPPED ()
$ hubble observe --input-file flow-from-cilium-docs.json --verdict DROPPED --from-pod default/xwing -o dict
  TIMESTAMP: Aug 21 12:12:18.510
     SOURCE: default/xwing:44916
DESTINATION: default/deathstar-7848d6c4d5-th9v2:80
       TYPE: UNKNOWN
    VERDICT: DROPPED
    SUMMARY:

The docs' example was exported with a mask that drops event_type and the identities, so the CLI prints UNKNOWN and (unknown). The same query on this lab's own export, with the mask above:

$ hubble observe --input-file events-worker2.log --verdict DROPPED --from-pod pay/payments -o dict
  TIMESTAMP: Sep 25 01:33:33.652
     SOURCE: pay/payments-595db8dd7f-cl56d:33706
DESTINATION: 172-18-0-1.sslip.io:8443
       TYPE: policy-verdict:none EGRESS
    VERDICT: DENIED
    SUMMARY: TCP Flags: SYN

Destination name, direction and verdict survive, because the mask keeps destination_names, traffic_direction and event_type. The exporter also writes agent_event lines (policy updates, endpoint changes). The CLI skips them with a warning, unknown field detected ... "agent_event"; your log pipeline can keep them as a record of policy changes.

2. The exporter is writing, on every node:

pod/cilium-4545f 2
{"flow":{"time":"2026-09-25T01:31:47.625726584Z","uuid":"6e3a69f9-05d3-47d9-9560-eb913f17e98b","verdict":"FORWARDED","IP
pod/cilium-rrqrl 2
{"flow":{"time":"2026-09-25T01:31:47.925189756Z","uuid":"609d2229-3c7d-4c92-9bd9-d1f0baa4de70","verdict":"FORWARDED","IP
pod/cilium-w6hnt 2
{"flow":{"time":"2026-09-25T01:31:47.890615468Z","uuid":"f556dd44-e06d-43a3-a4ca-c9387672014c","verdict":"FORWARDED","IP

Rotated copies appear next to events.log once it reaches fileMaxSizeMb.

3. The TLS requirement holds:

$ kubectl -n kube-system get configmap cilium-config -o jsonpath='{.data.hubble-disable-tls}{"\n"}'
false
$ kubectl -n kube-system port-forward svc/hubble-relay 4245:443 &
$ hubble observe --server localhost:4245 --last 1
rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
$ hubble observe --server localhost:4245 --tls --tls-ca-cert-files hubble-ca.crt \
    --tls-server-name ui.hubble-relay.cilium.io --last 1
Sep 25 01:32:22.040: kube-system/hubble-relay-749757b597-qkw47:40584 (ID:19626) <- 172.18.0.4:4244 (host) to-endpoint FORWARDED (TCP Flags: SYN, ACK)

The second call had the CA (from the hubble-relay-server-certs Secret, whose certificate is for *.hubble-relay.cilium.io) and no client certificate, and it worked. That is hubble.relay.tls.server.mtls at its default of false. Set mtls: true if Relay must require client certificates, and give the Hubble UI and CLI their own.

4. Redaction works on L7 flows. A request with a password in the URL, a query string, a bearer token and a cookie, through an HTTP-aware policy:

$ curl -H "Authorization: Bearer s3cr3t-token" -H "Cookie: session=abc123" \
    "http://admin:[email protected]/login?password=hunter2&user=alice"
$ hubble observe --namespace edge-test --type l7 --protocol http --last 4 -o json \
    | jq -c '.flow.l7 | {type, url: .http.url, headers: .http.headers}'
{"type":"REQUEST","url":"http://web.edge-test.svc.cluster.local/login","headers":[{"key":":scheme","value":"http"},{"key":"Accept","value":"*/*"},{"key":"Authorization","value":"HUBBLE_REDACTED"},{"key":"Cookie","value":"HUBBLE_REDACTED"},{"key":"User-Agent","value":"curl/8.14.1"},...]}

No admin:hunter2@, no query string, and both secrets replaced. The exported file on the node agrees:

$ grep -c hunter2 /var/run/cilium/hubble/events.log
0
$ grep -c s3cr3t /var/run/cilium/hubble/events.log
0

5. A forensic query, live or from the export. One line per destination, with a count:

hubble observe --from-pod pay/payments-595db8dd7f-cl56d \
  --since 2026-09-25T01:24:00Z --until 2026-09-25T01:34:00Z \
  --not --to-namespace pay -o jsonpb > payments-egress.json
jq -r '.flow | select((.is_reply // false) == false)
  | [(.destination.namespace // .IP.destination), (.destination_names // [] | join(",")),
     (.l4.TCP.destination_port // .l4.UDP.destination_port), .verdict] | @tsv' \
  payments-egress.json | sort | uniq -c | sort -rn

     32 kube-system		53	FORWARDED
      8 kube-system		53	REDIRECTED
      6 172.18.0.1	172-18-0-1.sslip.io	8443	DROPPED
      5 104.20.23.154	example.com	443	FORWARDED

The third line is the one an investigator wants: an attempt to reach a private address through a public name, blocked. Leave the timestamp out of the columns you deduplicate on, or sort -u keeps every packet.

#Mistakes people make

#Treating the ring buffer as history

It is a buffer. Export, ship, and query the store; use live Hubble for the last few minutes.

#Exporting with a field mask that throws away identity

A mask with only IPs and ports leaves you mapping addresses to pods by hand, after the pods are gone. Keep source, destination, destination_names, Type and event_type.

#Keeping evidence only on the node

A compromised node can delete its own logs, and a replaced node takes them with it. Ship continuously.

#Storing secrets in flow logs

L7 flows can include URLs and headers. Turn on redaction for query strings, user info and authorization headers before you enable export.

#Turning off Hubble TLS to fix a certificate error

The API exposes the cluster's communication map. Fix the certificates (the chart can generate and rotate them); do not set hubble.tls.enabled: false.

#Checklist

• hubble.export.static.enabled is on, with a file path, size and backup count.
• The field mask keeps source, destination, destination_names, verdict, IP, l4, Type, event_type and time.
• A log agent ships the export files from every node to a store with retention you set.
• hubble.redact removes URL queries, user info and authorization and cookie headers.
• hubble.tls.enabled is true and Relay serves TLS.
• Relay and the UI are reachable only by investigators, over private access.
• Node clocks are synchronized.
• An offline hubble observe --input-file query has been practiced on real exports.