WAF auto-ban across edge PoPs, the secure way

#What goes wrong

A WAF judges one request at a time. A scanner that fails a thousand times still gets its thousand-and-first try, and with several points of presence (PoPs) it gets a fresh start in each region. Banning repeat offenders at the edge saves your backends, your logs and your WAF CPU.

Doing it naively goes wrong in predictable ways. The ban uses the address the WAF logged, which behind a load balancer is the load balancer, so the first ban takes down the whole site. Bans never expire, so the list grows until it contains half of a mobile carrier. A policy is written without defaultAction: Allow, and Envoy Gateway's default for an authorization block is deny, for everyone.

And one trap is specific to Envoy Gateway: a SecurityPolicy on a single route replaces the Gateway-level policy for that route. Your ban list silently stops covering every route that has its own policy.

#What the docs say

Note: if neither Rules nor DefaultAction is specified, the default action is to deny all requests.

Source: Envoy Gateway API reference, Authorization

Route-specific policies take precedence first

Source: Envoy Gateway docs, SecurityPolicy precedence

MergeType determines how this configuration is merged with existing SecurityPolicy configurations targeting a parent resource.

Source: Envoy Gateway API reference, SecurityPolicySpec

The precedence page describes which policy wins. It does not spell out that "wins" means the Gateway-level ban is gone for that route unless the route policy sets mergeType.

#The secure configuration

The loop has four parts.

1. Detection, per PoP. Coraza with CRS blocks requests and logs the client address. That address must be the real client: set client IP detection first (see IP blocklists).
2. Collection. Ship the Envoy logs of every PoP to one store (Loki, an Elasticsearch-compatible store, a SIEM).
3. Decision, once, centrally. A scheduled job counts blocking decisions (CRS rules 949110 and 949111) per client address over the last 24 hours, removes a never-ban list, caps the list size, and renders the policy.
4. Distribution. The job commits the policy to the repository your GitOps tool syncs to every PoP. Every PoP enforces the same list.

The decision script, reading Coraza log lines on stdin. Envoy Gateway attaches only one SecurityPolicy to a Gateway, so the script renders that one policy: your static blocklist from a file, plus the addresses Coraza keeps blocking.

#!/bin/sh
# ban-candidates.sh THRESHOLD STATIC_LIST < coraza.log
# Renders the Gateway's one SecurityPolicy: the static blocklist (CIDRs, one per
# line in STATIC_LIST) plus every address with THRESHOLD or more WAF blocks.
th=${1:-5}; static=${2:-static-blocklist.txt}
ips=$(grep -E '\[id "94911[01]"\]' | sed -nE 's/.*\[client "([0-9a-fA-F:.]+)"\].*/\1/p' \
      | sort | uniq -c | awk -v t="$th" '$1 >= t {print $2}' | sort -V)
cat <<YAML
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: ip-blocklist               # one SecurityPolicy per Gateway: static list and bans together
  namespace: edge
  annotations:
    example.com/generated-at: "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: edge
  authorization:
    defaultAction: Allow          # a ban list: everyone else is allowed
    rules:
    - name: blocklist
      action: Deny
      principal:
        clientCIDRs:
YAML
grep -vE '^[[:space:]]*(#|$)' "$static" | sed 's/^/        - /'
[ -z "$ips" ] && exit 0
cat <<YAML
    - name: waf-autoban
      action: Deny
      principal:
        clientCIDRs:
YAML
for ip in $ips; do
  case $ip in *:*) echo "        - $ip/128" ;; *) echo "        - $ip/32" ;; esac
done

In production, add before rendering:

# Never ban these: your monitors, office egress, partners, the load balancer.
grep -v -x -F -f never-ban.txt
# Cap the list so a bug or a flood cannot ban the internet.
head -n 5000

Every route that needs its own SecurityPolicy (JWT, CORS, OIDC) must merge with the Gateway policy, or the ban does not reach it:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: app-cors
  namespace: edge
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: app
  mergeType: StrategicMerge       # keep the Gateway-level ban on this route
  cors:
    allowOrigins: ["https://www.example.com"]

The merge has a limit. If the route's policy sets authorization itself (an allowlist on an admin route, for example), the route's rules replace the Gateway's, even with StrategicMerge. Copy the ban rules into that policy, before its allow rules, or the ban does not apply there (Prove it, step 4).

Because the list is recomputed from a sliding window at every run, a ban expires by itself when the address stops attacking. No state, no cleanup job.

#Prove it

1. Detection, on a standalone Envoy. The test in secure-tests/waf-auto-ban-edge-pops/run.sh runs Coraza with CRS on Envoy 1.39.1 behind one trusted proxy hop, sends attacks from two addresses and normal traffic from a third, and runs the script on the log:

== Coraza with CRS, xff_num_trusted_hops: 1 (one load balancer in front)
  2 GET  /.env                                        -> 403
  1 GET  /?id=1%27%20OR%20%271%27%3D%271              -> 403
  1 GET  /?id=2%27%20OR%20%271%27%3D%271              -> 403
  1 GET  /?id=3%27%20OR%20%271%27%3D%271              -> 403
  1 GET  /?id=4%27%20OR%20%271%27%3D%271              -> 403
  1 GET  /?id=5%27%20OR%20%271%27%3D%271              -> 403
  1 GET  /?id=6%27%20OR%20%271%27%3D%271              -> 403
  3 GET  /search?q=shoes                              -> 200

== Client addresses Coraza logged for blocked requests
  2 198.51.100.23
  6 203.0.113.7

== ban-candidates.sh 5 static-blocklist.txt
    rules:
    - name: blocklist
      action: Deny
      principal:
        clientCIDRs:
        - 203.0.113.0/24
        - 2001:db8:bad::/48
    - name: waf-autoban
      action: Deny
      principal:
        clientCIDRs:
        - 203.0.113.7/32

== The generated SecurityPolicy, checked by Envoy Gateway v1.9.1 (egctl, offline)
  SecurityPolicy Accepted="True": Policy has been accepted.

Only the address with six blocks is banned; the one with two is not.

2. The loop on a cluster. The WAF from Coraza with OWASP CRS v4 on a lab Gateway, behind a proxy that appends X-Forwarded-For. other sent six SQL injection probes, other2 two, other3 only normal requests:

403 403 403 403 403 403  <- other (6 attacks)
403 403  <- other2 (2 attacks)
200 200 200  <- other3 (normal)
$ kubectl -n envoy-gateway-system logs <envoy pod> -c envoy --since=3m > envoy-coraza.log
$ grep -E '\[id "94911[01]"\]' envoy-coraza.log | sed -nE 's/.*\[client "([0-9a-fA-F:.]+)"\].*/\1/p' | sort | uniq -c
      6 10.244.2.207
      2 10.244.2.219

Coraza logged each pod's own address, not the proxy's: client IP detection works end to end. The first version of this page rendered the ban as a second SecurityPolicy, waf-autoban, next to the static blocklist. On the cluster:

ip-blocklist: Accepted=True Policy has been accepted.
waf-autoban: Accepted=False Unable to target Gateway edge, another SecurityPolicy has already attached to it
other   www.example.com /   -> 200

The attacker was not banned at all. With the script above, one policy:

ip-blocklist: Accepted=True Policy has been accepted.
other   www.example.com /   -> 403   (auto-banned)
other3  www.example.com /   -> 200
probe   www.example.com /   -> 403   (static blocklist)

3. The ban expires by itself. With no blocks in the window, the script renders the static list alone and drops the waf-autoban rule.

4. Routes with their own SecurityPolicy. A route app for /api with the CORS policy above:

without mergeType:           other  /api -> 404   (reached the backend: not banned)
with mergeType StrategicMerge: other  /api -> 403
app-cors: Merged=True Merged with policy edge/ip-blocklist

And a route with its own authorization: the admin route from protecting an IdP admin console, whose allowlist includes other:

with mergeType StrategicMerge: other  id.example.com/admin/master/console/ -> 200

Merged=True in the status, yet the route's rendered RBAC held only its own rules (the admin-network ALLOW and a default DENY). Authorization is replaced, not merged. With the ban rule copied into the route's policy, first:

$ kubectl get securitypolicy idp-admin-allowlist -n edge -o jsonpath='{range .spec.authorization.rules[*]}{.name}:{.action} {end}'
waf-autoban:Deny admin-network:Allow
other  id.example.com/admin/master/console/ -> 403

5. Across PoPs (not run here): after the GitOps sync, the same request from a banned address returns 403 at every PoP address.

#Mistakes people make

#Banning the load balancer

If client IP detection is not set, every logged address is the load balancer. The first ban blocks everyone. Check the addresses in the log before you enable the job.

#Forgetting defaultAction: Allow

An authorization block without defaultAction denies everything that no rule allows. For a ban list, that is the whole internet.

#Route policies that do not merge

A SecurityPolicy on a route takes precedence over the Gateway policy. Set mergeType: StrategicMerge on route policies, and alert on the "is being overridden" status message.

#Bans that never expire

Addresses change owners: mobile carriers, cloud providers, VPN exits. Compute the list from a sliding window so bans end on their own.

#No never-ban list and no cap

Your uptime monitor trips a rule once a day for a year, and one day it is banned. Keep a never-ban list, and cap the number of entries.

#Deciding per PoP

A ban decided in one region leaves the others open, and different lists per region make incidents hard to read. Decide centrally, distribute the same list everywhere.

#Checklist

• Configure client IP detection on every PoP before enabling bans.
• Ship Coraza logs from all PoPs to one store.
• Count CRS 949110/949111 blocks per address over a sliding window.
• Remove a never-ban list and cap the number of entries.
• Render one SecurityPolicy with defaultAction: Allow and a Deny rule.
• Distribute it to every PoP through GitOps.
• Set mergeType: StrategicMerge on every route-level SecurityPolicy.
• Alert on "overridden" policy status and on ban list size jumps.