Fail-closed WASM filters pinned by digest, the secure way

#What goes wrong

A Wasm filter is code that runs inside Envoy on every request. For a WAF it is the security control itself. Two things can go wrong with it at runtime: the module you get is not the module you tested, or the module does not load.

Referencing a module by tag or by URL without a checksum covers neither. A re-pushed tag or a replaced file changes the code in your edge without a change in your config. Envoy Gateway does not verify an image or HTTP module when sha256 is not set.

When a module fails, the proxy chooses. Fail closed returns 503. Fail open skips the filter and passes the request on, uninspected. Fail open is often switched on after the first outage caused by a module, because 503s are loud and a missing WAF is quiet. That trade is exactly backwards for a security filter: an attacker who can break the module's download or configuration gets a WAF-free edge.

#What the docs say

If not specified, Envoy Gateway will not verify the downloaded Wasm code.

Source: Envoy Gateway API reference, HTTPWasmCodeSource sha256

SHA256 checksum that will be used to verify the OCI image. It must match the digest of the OCI image.

Source: Envoy Gateway API reference, ImageWasmCodeSource sha256

FAIL_OPEN All plugins associated with the VM will be ignored and the filter chain will continue. This makes sense when the plugin is optional.

Source: Envoy docs, Wasm FailurePolicy

The Envoy docs say fail-open is for optional plugins. A WAF is not optional. The docs do not say that for a multi-architecture image, the digest Envoy Gateway checks is the platform image manifest, not the index digest that docker pull prints.

#The secure configuration

Find the digest of the platform manifest (linux/amd64 here) of the image:

crane manifest ghcr.io/corazawaf/coraza-proxy-wasm:0.6.0 \
  | jq -r '.manifests[] | select(.platform.os=="linux" and .platform.architecture=="amd64") | .digest'
# sha256:65d6009b9da2e8965e592a08b74a86725435fc01aa39c756dce0bd5ea64b3f4e

Envoy Gateway:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: coraza-waf
  namespace: edge
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: edge
  wasm:
  - name: coraza-waf
    rootID: coraza
    failOpen: false               # fail closed: 5xx if the module is not running
    code:
      type: Image
      image:
        # Digest in the URL: the registry cannot hand you anything else.
        url: ghcr.io/corazawaf/coraza-proxy-wasm@sha256:65d6009b9da2e8965e592a08b74a86725435fc01aa39c756dce0bd5ea64b3f4e
        # And Envoy Gateway checks what it pulled against the same digest.
        sha256: 65d6009b9da2e8965e592a08b74a86725435fc01aa39c756dce0bd5ea64b3f4e
    config:
      directives_map:
        default: ["Include @recommended-conf", "SecRuleEngine On", "Include @crs-setup-conf", "Include @owasp_crs/*.conf"]
      default_directives: default

Standalone Envoy, a remote module (the sha256 field is required here):

- name: envoy.filters.http.wasm
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
    config:
      name: coraza
      root_id: coraza
      fail_open: false                     # newer Envoy: failure_policy: FAIL_CLOSED
      vm_config:
        runtime: envoy.wasm.runtime.v8
        vm_id: coraza
        code:
          remote:
            http_uri:
              uri: https://artifacts.example.com/coraza-proxy-wasm-0.6.0.wasm
              cluster: artifacts
              timeout: 10s
            # sha256 of the .wasm file itself (not of the zip or image)
            sha256: f36e710529167482df820b790981230092da098d26d21aebb16f1d9c8c4e26d5
            retry_policy:
              num_retries: 10
              retry_back_off: {base_interval: 1s, max_interval: 4s}

Better still for standalone Envoy: ship the verified module in the image or a read-only volume and load it with local, so startup does not depend on a download at all.

#Prove it

The test in secure-tests/fail-closed-wasm-filters-pinned-digest/run.sh serves coraza-proxy-wasm.wasm from a throwaway HTTP file server and starts Envoy 1.39.1 with a remote Wasm source. Each case sends a normal request and a SQL injection.

== 1. Remote module, sha256 pinned to the real digest
envoy: running (exit 0)
GET  /                                    -> 200
GET  /?id=1%27%20OR%20%271%27%3D%271      -> 403

== 2. Remote module, sha256 pinned to a different digest (the file changed), fail_open: false
envoy: running (exit 0)
GET  /                                    -> 503
GET  /?id=1%27%20OR%20%271%27%3D%271      -> 503
  log: Plugin configured to fail closed failed to load
  log: Plugin coraza failed to load
  log: Retry limit exceeded for fetching data from remote data source

== 2. Remote module, sha256 pinned to a different digest (the file changed), fail_open: true
envoy: running (exit 0)
GET  /                                    -> 200
GET  /?id=1%27%20OR%20%271%27%3D%271      -> 200
  log: Plugin coraza failed to load
  log: Retry limit exceeded for fetching data from remote data source

== 3. Broken plugin configuration (a typo in one directive) at startup
envoy: exited (exit 1)
  log: Failed to parse directives: invalid WAF config from string

Case 2 with fail-open is the one to remember: the WAF is gone, the SQL injection gets a 200, and the only trace is a log line. Note that Envoy reports a digest mismatch as a failed fetch; the reason (data is invalid) appears only at debug log level.

#Mistakes people make

#Pinning the tag and calling it pinned

:0.6.0 is a name, not content. Put the digest in the URL and in sha256.

#Copying the index digest

docker pull and many registries show the digest of the multi-platform index. Envoy Gateway compares against the digest of the image it pulled for its platform. Use the platform manifest digest, or the check fails.

#Turning on fail-open after an outage

It trades a visible outage for an invisible security gap. Fix what made the module fail: pin it, mirror it, load it locally, alert on load failures.

#Depending on a public registry at startup

If the registry is slow or down, every new Envoy pod waits or fails. Mirror the image into your own registry and reference it there, by digest.

#Not alerting on "failed to load"

With fail-closed, a failed module shows up as 503s. Alert on the log line and on the 503 rate of the edge so you hear about it before customers do.

#Checklist

• Reference Wasm images by @sha256: digest in the URL.
• Set sha256 to the platform manifest digest in Envoy Gateway.
• Set sha256 of the .wasm file for Envoy remote sources, or load the module locally.
• Keep failOpen: false (FAIL_CLOSED); never enable fail-open for a WAF.
• Mirror the module to a registry you control.
• Alert on "failed to load" and on edge 503 rates.
• After every module update, send a known attack and expect it blocked.