Blocking the cloud metadata endpoint from pods, the secure way

#What goes wrong

Every major cloud runs an instance metadata service on the link-local address 169.254.169.254. A VM asks it for its identity, its network settings, its user-data and, on most clouds, temporary credentials for the role attached to the VM.

Pods on a Kubernetes node route to that address through the node. Unless something blocks it, any pod can read what the node can read. The usual path in is a server-side request forgery (SSRF): an application that fetches a URL for a user is asked to fetch http://169.254.169.254/....

What leaks depends on the cloud: role credentials, bootstrap data in user-data, and details that make the next step easier. With node credentials, the attacker acts as the node's cloud identity, outside Kubernetes and outside your network policy.

Two things make the usual fixes weaker than they look:

• An allow can undo it. If the block is an ordinary allow-list rule, one broad policy (toEntities: [world]) re-opens it.
• hostNetwork pods are the node. A pod with hostNetwork: true uses the node's network namespace. Pod policies do not select it.

#What the docs say

Droplets can access the metadata service using the special, static, link-local IP address 169.254.169.254.

Source: DigitalOcean docs, How to Access Information about a Droplet using the Metadata API

Deny policies take precedence over allow policies, regardless of whether they are a Cilium Network Policy, a Clusterwide Cilium Network Policy or even a Kubernetes Network Policy.

Source: Cilium docs, Deny Policies

Rules with EnableDefaultDeny disabled are ignored when determining the default mode.

Source: Cilium docs, Policy Enforcement Modes

The hop limit is the number of network hops that the PUT response is allowed to make.

Source: AWS docs, Configure instance metadata options

None of these pages connect the pieces for Kubernetes. The Cilium docs give you a deny that beats allows and a switch that keeps a deny-only policy from turning on default deny everywhere; the cloud docs describe the endpoint and the hop limit. Put together, they give a block that later policies cannot undo, without breaking any other traffic.

#The secure configuration

1. A cluster-wide deny for pod traffic to metadata.

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: deny-cloud-metadata
spec:
  description: "No pod reaches the cloud metadata service, except pods labelled for it"
  endpointSelector:
    matchExpressions:
      - key: security.example.com/metadata-access
        operator: NotIn
        values: ["allowed"]
  # This policy only denies. It must not switch every pod into default deny.
  enableDefaultDeny:
    egress: false
    ingress: false
  egressDeny:
    - toCIDR:
        - 169.254.169.254/32    # AWS, GCP, Azure, DigitalOcean, OpenStack metadata
        - fd00:ec2::254/128     # AWS IMDS over IPv6

Deny rules win over allow rules, so no namespace policy can re-open the path. Exceptions are made by label on the pod, and only in the selector. Who may set that label is now a security decision: restrict it with an admission policy, and keep it off application workloads.

Some platform components legitimately read metadata. Many run with hostNetwork: true and are not affected. On GKE, Workload Identity serves pod credentials from the metadata address; if your pods use it, test before you roll out, and scope the label to the pods that need it.

2. Forbid hostNetwork where applications run. hostNetwork pods are not selected by pod policies. Pod Security Admission baseline rejects them:

apiVersion: v1
kind: Namespace
metadata:
  name: app
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/enforce-version: latest

3. On AWS, add the cloud-side control. Require IMDSv2 (session tokens) and a hop limit of 1, so the token response cannot travel past the node's own network namespace into a pod:

aws ec2 modify-instance-metadata-options \
  --instance-id i-0123456789abcdef0 \
  --http-tokens required \
  --http-put-response-hop-limit 1 \
  --http-endpoint enabled

Set the same in your launch template or node group so new nodes get it. A hop limit of 1 also breaks pods that rely on metadata for credentials; move those to a workload identity mechanism that does not use the node's role.

#Prove it

The lab is a kind cluster on a DigitalOcean droplet, so 169.254.169.254 is a real metadata service. Only HTTP status codes are shown.

0. Before the policy, any pod reads metadata:

$ kubectl -n edge-test exec probe -- curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://169.254.169.254/metadata/v1/id
200

1. After the policy, the same pod cannot:

$ kubectl apply -f deny-cloud-metadata.yaml
$ kubectl -n edge-test exec probe -- curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://169.254.169.254/metadata/v1/id
000
command terminated with exit code 28

000 with exit code 28 is curl's timeout: the packets were dropped. Any HTTP status, even 401 or 404, would mean the pod reached the service.

2. Cilium dropped it, with the deny verdict:

$ hubble observe --from-namespace edge-test --to-ip 169.254.169.254 --verdict DROPPED --print-policy-names --last 2
edge-test/probe:39434 (ID:2919) <> 169.254.169.254:80 (ID:16777218) policy-verdict:L3-Only EGRESS DENIED BY deny-cloud-metadata (CiliumClusterwideNetworkPolicy) (TCP Flags: SYN)
edge-test/probe:39434 (ID:2919) <> 169.254.169.254:80 (ID:16777218) Policy denied by denylist DROPPED (TCP Flags: SYN)

Use --from-namespace or --from-pod. Hubble rejects --namespace together with --to-ip: filters --to-ip and --namespace cannot be combined.

3. A broad allow does not re-open it. With this test policy applied, the probe still got 000, and normal traffic still worked:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: test-allow-everything
  namespace: edge-test
spec:
  endpointSelector:
    matchLabels:
      run: probe
  egress:
    - toEntities:
        - all

metadata:              000
web.edge-test (a Service in the cluster):  200

4. The label is the only way through. Labelling the running pod opened metadata again after Cilium recomputed the pod's identity:

$ kubectl -n edge-test label pod probe security.example.com/metadata-access=allowed
t+5s 000
t+10s 000
t+15s 200

So anyone who can patch pods in a namespace can exempt a pod. Restrict that label with an admission policy.

5. hostNetwork walks around the policy. A hostNetwork: true pod, with the deny policy in place:

$ kubectl -n hostnet-test exec hn -- curl -s -m 5 -o /dev/null -w '%{http_code}\n' http://169.254.169.254/metadata/v1/id
200

Pod Security baseline stops new ones:

$ kubectl label ns hostnet-test pod-security.kubernetes.io/enforce=baseline pod-security.kubernetes.io/enforce-version=latest
Warning: existing pods in namespace "hostnet-test" violate the new PodSecurity enforce level "baseline:latest"
Warning: hn: host namespaces
$ kubectl -n hostnet-test run hn2 --image=curlimages/curl:8.14.1 --overrides='{"spec":{"hostNetwork":true}}' ...
Error from server (Forbidden): pods "hn2" is forbidden: violates PodSecurity "baseline:latest": host namespaces (hostNetwork=true)

Note the warning: the label does not remove the hostNetwork pod that already runs. Find and delete those yourself (the jq query below).

6. Only the pods you meant are exempt:

kubectl get pods -A -l security.example.com/metadata-access=allowed
kubectl get pods -A -o json \
  | jq -r '.items[] | select(.spec.hostNetwork==true) | "\(.metadata.namespace)/\(.metadata.name)"'

7. On AWS, the node settings are in place (not run in this lab):

aws ec2 describe-instances --instance-ids i-0123456789abcdef0 \
  --query 'Reservations[].Instances[].MetadataOptions'

What you should see: "HttpTokens": "required" and "HttpPutResponseHopLimit": 1.

#Mistakes people make

#Blocking it with an allow-list that someone widens later

A default-deny egress policy blocks metadata only until a team adds toEntities: [world] or toCIDR: 0.0.0.0/0. An egressDeny survives that.

#Turning the whole cluster into default deny by accident

Cilium's deny docs warn that pods enter default-deny mode as soon as a single policy selects them, deny policies included. A cluster-wide deny that selects every pod can switch them all to default-deny egress. Set enableDefaultDeny to false for both directions on a deny-only policy.

#Forgetting IPv6

AWS also serves IMDS on fd00:ec2::254 when the IPv6 endpoint is enabled. Deny it even if you think IPv6 is off; it costs one line.

#Trusting the label

The exception label is as strong as the controls on who can set it. If any developer can label a pod metadata-access=allowed, the block is advisory.

#Leaving hostNetwork open in app namespaces

A hostNetwork pod is the node for Cilium's pod policies. Enforce Pod Security baseline or stricter in every application namespace.

#Checklist

• A CiliumClusterwideNetworkPolicy denies egress to 169.254.169.254/32 and fd00:ec2::254/128.
• The policy sets enableDefaultDeny to false for ingress and egress.
• Exceptions use a pod label, and admission policy controls who can set it.
• Application namespaces enforce Pod Security baseline or restricted.
• On AWS, nodes require IMDSv2 with a hop limit of 1, in the launch template too.
• A probe pod gets no response from 169.254.169.254.
• A test allow-all policy does not re-open the path.
• Hubble shows the probe's packets dropped by policy.