Real client IPs with Proxy Protocol v2, the secure way

#What goes wrong

A TCP (layer 4) load balancer opens its own connection to Envoy. It cannot add an X-Forwarded-For header, because it does not speak HTTP, and with TLS passthrough it could not read the request anyway. So Envoy sees the load balancer's address as the client, for every request.

PROXY protocol fixes this. The load balancer writes a short header at the start of each connection with the original client address, and Envoy reads it. Version 2 is a binary format; version 1 is a text line.

The hole is that the header is just bytes at the start of a connection. Envoy believes whatever header arrives. If anyone other than the load balancer can open a connection to Envoy's port, they can write a header that says they are any address they like: an allowlisted office, or a different address for every request to dodge a rate limit.

#What the docs say

The receiver MUST be configured to only receive the protocol described in this specification and MUST not try to guess whether the protocol header is present or not.

Source: HAProxy, The PROXY protocol specification

Allow requests through that don’t use proxy protocol. Defaults to false.

Source: Envoy docs, Proxy Protocol listener filter

Optional breaks conformance with the specification. Only enable if ALL traffic to the listener comes from a trusted source.

Source: Envoy Gateway API reference, ProxyProtocolSettings

This is always the physical peer even if the remote_ip is inferred from the x-forwarder-for header, the proxy protocol, etc.

Source: Envoy docs, RBAC Principal direct_remote_ip

The docs warn about optional mode. They do not say that required mode is just as spoofable when the port is reachable from anywhere but the load balancer.

#The secure configuration

Envoy Gateway, on the Gateway behind a TCP load balancer with PROXY protocol v2 turned on:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: proxy-protocol
  namespace: edge
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: edge
  # Required mode: a connection without a valid PROXY header is closed.
  # Do not set optional: true.
  proxyProtocol: {}

Envoy Gateway attaches one ClientTrafficPolicy per Gateway. If the Gateway already has one, merge these fields into it; a second policy is refused with Accepted=False (see ML-KEM-1024 on Envoy Gateway).

Turn on PROXY protocol v2 on the load balancer in the same change. With it on only one side, every connection fails.

Standalone Envoy, the listener with the filter and a guard that only the load balancer may connect (replace 192.0.2.10 with its address):

listeners:
- name: edge
  address: {socket_address: {address: 0.0.0.0, port_value: 443}}
  listener_filters:
  - name: envoy.filters.listener.proxy_protocol
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol
      disallowed_versions: [V1]        # the load balancer sends v2
  filter_chains:
  - filters:
    # Checked against the TCP peer, not the address in the PROXY header.
    - name: envoy.filters.network.rbac
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.filters.network.rbac.v3.RBAC
        stat_prefix: only_lb
        rules:
          action: ALLOW
          policies:
            load-balancer:
              permissions: [{any: true}]
              principals: [{direct_remote_ip: {address_prefix: 192.0.2.10, prefix_len: 32}}]
    - name: envoy.filters.network.http_connection_manager
      # ... use_remote_address: true, routes, filters ...

On Kubernetes, do the same with a NetworkPolicy on the Envoy pods: allow the listener port only from the load balancer's source addresses (for most cloud load balancers, the node or subnet range they connect from), not from other pods.

#Prove it

The test in secure-tests/real-client-ips-proxy-protocol-v2/run.sh puts Envoy, an HAProxy load balancer (send-proxy-v2) and a client on a throwaway Docker network, so each has its own address. Envoy returns the client address it used in a response header. Each block tries four ways in: through the load balancer, direct without a header, direct with a forged v1 header (curl --haproxy-clientip), direct with a forged v2 header (raw bytes through nc).

Envoy Gateway renders proxyProtocol: {} as the plain listener filter:

== Envoy Gateway v1.9.1: ClientTrafficPolicy proxyProtocol, rendered offline by egctl
  ClientTrafficPolicy Accepted="True": Policy has been accepted.
  listenerFilters:
  - name: envoy.filters.listener.proxy_protocol
  typedConfig:
  '@type': type.googleapis.com/envoy.extensions.filters.listener.proxy_protocol.v3.ProxyProtocol
  - name: envoy.filters.listener.tls_inspector

The four Envoy settings:

== Envoy proxy_protocol listener filter, defaults (header required, v1 and v2)
  client address: 172.21.0.4, load balancer: 172.21.0.2
  via the load balancer (PROXY v2 added by HAProxy):      200 (client seen: 172.21.0.4)
  direct to Envoy, no PROXY header:                       no response
  direct to Envoy, forged PROXY v1 claiming 203.0.113.50: 200 (client seen: 203.0.113.50)
  direct to Envoy, forged PROXY v2 claiming 203.0.113.60: 200 (client seen: 203.0.113.60)

== disallowed_versions: [V1]
  client address: 172.21.0.4, load balancer: 172.21.0.2
  via the load balancer (PROXY v2 added by HAProxy):      200 (client seen: 172.21.0.4)
  direct to Envoy, no PROXY header:                       no response
  direct to Envoy, forged PROXY v1 claiming 203.0.113.50: no response
  direct to Envoy, forged PROXY v2 claiming 203.0.113.60: 200 (client seen: 203.0.113.60)

== allow_requests_without_proxy_protocol: true (optional mode)
  client address: 172.21.0.4, load balancer: 172.21.0.2
  via the load balancer (PROXY v2 added by HAProxy):      200 (client seen: 172.21.0.4)
  direct to Envoy, no PROXY header:                       200 (client seen: 172.21.0.4)
  direct to Envoy, forged PROXY v1 claiming 203.0.113.50: 200 (client seen: 203.0.113.50)
  direct to Envoy, forged PROXY v2 claiming 203.0.113.60: 200 (client seen: 203.0.113.60)

== defaults, plus a network RBAC filter that admits only the load balancer's address
  client address: 172.21.0.4, load balancer: 172.21.0.2
  via the load balancer (PROXY v2 added by HAProxy):      200 (client seen: 172.21.0.4)
  direct to Envoy, no PROXY header:                       no response
  direct to Envoy, forged PROXY v1 claiming 203.0.113.50: no response
  direct to Envoy, forged PROXY v2 claiming 203.0.113.60: no response

Through the load balancer, Envoy sees the real client (172.21.0.4) in every setting. Directly, required mode stops a plain request but believes any forged header. Disallowing v1 stops only the easy forgery. The peer check stops all of them.

#Mistakes people make

#Thinking required mode means authenticated

Required mode only means "a header must be present". Anyone who can reach the port can supply one. The protection is who can reach the port.

#Turning on optional mode during a migration and leaving it

Optional mode accepts both kinds of connection, so a direct client can choose to be itself or anyone. Switch the load balancer and Envoy together, in a maintenance window if you must, and keep required mode.

#Enabling it on one side only

PROXY protocol on Envoy without the load balancer sending it closes every connection. On the load balancer without Envoy expecting it, Envoy reads the header as garbage at the start of the TLS handshake. Change both at once.

#Adding XFF trust on top

With PROXY protocol, the client address comes from the header. Do not also trust X-Forwarded-For hops from the same load balancer; it does not set XFF, but clients can.

#Leaving Envoy reachable from inside the cluster

Other pods can connect to the Envoy pod directly and forge a header. A NetworkPolicy on the Envoy pods closes that path.

#Checklist

• Turn on PROXY protocol v2 on the TCP load balancer.
• Set proxyProtocol: {} in a ClientTrafficPolicy (required mode), never optional: true.
• Change the load balancer and Envoy in the same window.
• Allow Envoy's listener port only from the load balancer's source addresses.
• On standalone Envoy, add a network RBAC rule with direct_remote_ip.
• Send a forged header from outside the load balancer and expect no response.
• Check the access log shows real client addresses, not the load balancer's.