IP blocklists with Envoy Gateway SecurityPolicy, the secure way

#What goes wrong

Envoy Gateway runs behind something: a cloud load balancer, a CDN, another proxy. Each of them opens its own TCP connection to Envoy. Unless you tell Envoy Gateway how to find the real client, the client address it sees is the load balancer's.

A deny rule on the attacker's range then never matches, because no request seems to come from it. The same mistake the other way round is worse: an allow rule for an office range never matches either, so someone "fixes" it by allowing the load balancer's range, which is everyone.

The opposite error is trusting too much. X-Forwarded-For (XFF) is a request header, and the client writes the first part of it. If Envoy trusts one hop more than really exists, it reads an address that the attacker typed.

#What the docs say

The client IP is inferred from the X-Forwarded-For header, a custom header, or the proxy protocol. You can use the ClientIPDetection or the ProxyProtocol field in the ClientTrafficPolicy to configure how the client IP is detected.

Source: Envoy Gateway API reference, Principal.clientCIDRs

If NumTrustedHops is set to N, the client IP is taken from the Nth address from the right end of the XFF header.

Source: Envoy Gateway API reference, XForwardedForSettings

If use_remote_address is true and xff_num_trusted_hops is set to a value N that is greater than zero, the trusted client address is the Nth address from the right end of XFF.

Source: Envoy docs, x-forwarded-for

The docs describe the counting. They do not say what happens when the count is wrong, which is the part that decides whether your blocklist works.

#The secure configuration

Count the proxies in front of Envoy that append to XFF. With one cloud load balancer in HTTP mode, that number is 1.

# How Envoy Gateway finds the client address.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: client-ip
  namespace: edge
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: edge
  clientIPDetection:
    xForwardedFor:
      # Exactly the number of proxies in front that append to XFF.
      # Take the address the load balancer appended, never one further left.
      numTrustedHops: 1
---
# The blocklist.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: ip-blocklist
  namespace: edge
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: edge
  authorization:
    defaultAction: Allow           # a blocklist: allow unless listed
    rules:
    - name: blocklist
      action: Deny
      principal:
        clientCIDRs:
        - 203.0.113.0/24
        - 2001:db8:bad::/48

If the load balancer works at TCP level (it does not add XFF), use PROXY protocol instead of XFF; see real client IPs with PROXY protocol v2.

Envoy Gateway attaches one ClientTrafficPolicy per Gateway. If the Gateway already has one (for TLS settings, for example), add clientIPDetection to that policy instead of creating a second; a second one is refused with Accepted=False (see ML-KEM-1024 on Envoy Gateway).

Restrict the Envoy Service so only the load balancer can reach it. A trusted hop count means nothing if a client can connect to Envoy directly and write its own XFF. With Cilium, a policy on the Envoy pods that Envoy Gateway creates (the lab's load balancer was a pod in edge; for a cloud load balancer, allow its addresses with fromCIDR):

# Only the load balancer may connect to Envoy's listener.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: envoy-only-from-lb
  namespace: envoy-gateway-system
spec:
  endpointSelector:
    matchLabels:
      gateway.envoyproxy.io/owning-gateway-name: edge
      gateway.envoyproxy.io/owning-gateway-namespace: edge   # the name alone also matches same-named Gateways elsewhere
  ingress:
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: edge
            app.kubernetes.io/name: lb
      toPorts:
        - ports:
            - port: "10080"
              protocol: TCP

Select by both the Gateway's name and its namespace. In the lab, a second Gateway also called edge, in another namespace, was locked out by the name-only selector as well.

The port is Envoy's container port, not the Gateway's: in the lab, Envoy Gateway ran the listener for port 80 on 10080 inside the pod. Check yours with kubectl -n envoy-gateway-system get endpointslices for the Envoy Service.

#Prove it

#What Envoy Gateway renders (offline, no cluster)

The test in secure-tests/ip-blocklists-envoy-gateway-securitypolicy/run.sh renders both policies with egctl experimental translate:

== Envoy Gateway v1.9.1: SecurityPolicy deny + ClientTrafficPolicy numTrustedHops 1, rendered offline by egctl
  ClientTrafficPolicy Accepted="True": Policy has been accepted.
  SecurityPolicy Accepted="True": Policy has been accepted.
  - name: envoy.extensions.http.original_ip_detection.xff
  useRemoteAddress: false
  action: DENY
  - addressPrefix: 203.0.113.0
  prefixLen: 24
  '@type': type.googleapis.com/envoy.extensions.matching.common_inputs.network.v3.SourceIPInput

The deny rule matches SourceIPInput: the client address after XFF processing, not the TCP peer.

#The rendered filters on Envoy 1.39.1

The same script runs that RBAC matcher and client IP setup on a standalone Envoy, with three values of numTrustedHops. The test client plays the load balancer: it connects from 127.0.0.1 and sends the XFF a load balancer would send. client seen is the address Envoy used.

== Envoy Gateway numTrustedHops: 0 (no ClientTrafficPolicy)
  blocked client 203.0.113.9                   XFF="203.0.113.9"                -> 200 (client seen: 127.0.0.1)
  blocked client forging XFF 198.51.100.1      XFF="198.51.100.1, 203.0.113.9"  -> 200 (client seen: 127.0.0.1)
  normal client 198.51.100.20                  XFF="198.51.100.20"              -> 200 (client seen: 127.0.0.1)

== Envoy Gateway numTrustedHops: 1
  blocked client 203.0.113.9                   XFF="203.0.113.9"                -> 403 (client seen: n/a)
  blocked client forging XFF 198.51.100.1      XFF="198.51.100.1, 203.0.113.9"  -> 403 (client seen: n/a)
  normal client 198.51.100.20                  XFF="198.51.100.20"              -> 200 (client seen: 198.51.100.20)

== Envoy Gateway numTrustedHops: 2
  blocked client 203.0.113.9                   XFF="203.0.113.9"                -> 200 (client seen: 127.0.0.1)
  blocked client forging XFF 198.51.100.1      XFF="198.51.100.1, 203.0.113.9"  -> 200 (client seen: 198.51.100.1)
  normal client 198.51.100.20                  XFF="198.51.100.20"              -> 200 (client seen: 127.0.0.1)

With 0, every request is the load balancer and the blocklist is dead. With 1, the blocked range gets 403 even when it forges an XFF entry. With 2, the attacker's forged 198.51.100.1 becomes the client and walks through.

#On a cluster

Both policies, unchanged except for one extra blocked address (the test pod's own), on a Gateway in a lab cluster. In front of Envoy, an nginx proxy played the cloud load balancer: it appends the peer address to X-Forwarded-For, like one in HTTP mode.

clienttrafficpolicy/client-ip: Accepted=True Policy has been accepted.
securitypolicy/ip-blocklist: Accepted=True Policy has been accepted.

probe is on the blocklist; other is not.

probe  through the LB   (no XFF)                   -> 403
probe  through the LB   forged XFF 198.51.100.1    -> 403
other  through the LB   forged XFF 203.0.113.9     -> 200
other  through the LB   (no XFF)                   -> 200
probe  direct to Envoy  forged XFF 198.51.100.20   -> 200
probe  direct to Envoy  (no XFF)                   -> 403

Through the load balancer the blocklist holds, whatever the client writes in the header. The fifth line is the hole: a blocked client that reaches Envoy directly and writes one XFF entry is trusted as if the load balancer had written it. With envoy-only-from-lb applied:

probe  through the LB   (no XFF)                   -> 403
other  through the LB   (no XFF)                   -> 200
probe  direct to Envoy  forged XFF 198.51.100.20   -> 000   (curl exit 28: timeout)
probe  direct to Envoy  (no XFF)                   -> 000   (curl exit 28: timeout)

The Envoy pod stayed Running and ready: kubelet probes from the node are not blocked by this policy.

On your own cluster, check the status as above, then send a request through the load balancer with a forged header:

curl -s -o /dev/null -w "%{http_code}\n" -H "X-Forwarded-For: 203.0.113.9" https://www.example.com/

What you should see: 200, because the load balancer appends your real address after the forged one and Envoy Gateway reads the real one. A 403 here means Envoy trusts one hop too many. Then try the Envoy Service address directly from a pod or host that is not the load balancer: it should time out.

#Mistakes people make

#Skipping client IP detection

Without a ClientTrafficPolicy, the client is the TCP peer. Behind a load balancer that is the load balancer, for every request.

#Counting hops generously

One extra trusted hop hands the client IP to whoever writes the XFF header. Count only proxies you run or pay for, and recount when the path changes (adding a CDN in front adds a hop).

#Letting clients reach Envoy directly

If the Envoy Service has a public address that bypasses the load balancer, a client can send any XFF it likes. Restrict the source ranges on the Service or the cloud firewall to the load balancer.

#Maintaining the list by hand

A blocklist that someone edits after an incident is stale by the next one. Feed it from WAF decisions with an expiry; see WAF auto-ban across edge PoPs.

#Using a blocklist as access control

Blocklists slow down known bad sources. Admin paths need an allowlist or authentication instead; see protecting an identity-provider admin console.

#Checklist

• Count the XFF-appending proxies in front of Envoy.
• Set clientIPDetection.xForwardedFor.numTrustedHops to exactly that count, or use PROXY protocol.
• Write the SecurityPolicy with defaultAction: Allow and a Deny rule of clientCIDRs.
• Check both policies read Accepted=True.
• Send a forged XFF through the load balancer and expect a normal response.
• Restrict direct access to the Envoy Service to the load balancer.
• Recount hops whenever a CDN or proxy is added in front.