Upstream TLS from the edge with a pinned CA, the secure way

#What goes wrong

The edge terminates TLS from the client and opens a new connection to the backend. That second hop crosses the node network, sometimes the cloud network between zones, sometimes the internet between regions. It deserves the same protection as the first hop.

Turning on TLS is half the job. Authenticating the backend is the other half. In Envoy, an upstream TLS context without a trusted CA verifies nothing: any certificate is accepted, including one an attacker generated a minute ago. The connection is encrypted, to whoever answered.

Two quieter mistakes: trusting the system CA store, which lets any public CA vouch for your backend, and trusting your internal CA without checking the name, which lets any service with a certificate from that CA pose as any other.

#What the docs say

If not specified and a peer certificate is presented it will not be verified.

Source: Envoy docs, CertificateValidationContext trusted_ca

Hostname defines the server name indication (SNI) the Gateway should use in order to connect to the backend, and must match the certificate served by the backend pod.

Source: Gateway API docs, BackendTLSPolicy

InsecureSkipVerify indicates whether the upstream’s certificate verification should be skipped.

Source: Envoy Gateway API reference, Backend BackendTLSSettings

The Gateway API page explains the fields. It does not say that wellKnownCACertificates: System trusts every public CA for your internal backends, which is rarely what you want for a service you issue certificates for yourself.

#The secure configuration

Envoy Gateway, with the internal CA certificate in a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: internal-ca
  namespace: edge                  # same namespace as the Service
data:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...your internal CA certificate (public part only)...
    -----END CERTIFICATE-----
---
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
  name: app-tls
  namespace: edge
spec:
  targetRefs:
  - group: ""
    kind: Service
    name: app
  validation:
    # Pin your own CA, not the system store.
    caCertificateRefs:
    - group: ""
      kind: ConfigMap
      name: internal-ca
    # SNI sent to the backend, and the name its certificate must carry.
    hostname: app.internal.example

Standalone Envoy, the cluster's transport socket:

transport_socket:
  name: envoy.transport_sockets.tls
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
    sni: app.internal.example
    common_tls_context:
      tls_params:
        tls_minimum_protocol_version: TLSv1_3
        tls_maximum_protocol_version: TLSv1_3   # client default max is TLSv1_2
      validation_context:
        trusted_ca: {filename: /etc/envoy/tls/internal-ca.pem}   # pinned CA
        match_typed_subject_alt_names:                           # and the name
        - san_type: DNS
          matcher: {exact: app.internal.example}

Issue backend certificates from that CA with the exact DNS name, for example with cert-manager and a CA issuer; see an offline internal CA with cert-manager.

#Prove it

The test in secure-tests/upstream-tls-edge-pinned-ca/run.sh puts Envoy 1.39.1 in front of three OpenSSL backends: the real app (internal CA, app.internal.example), an impostor (same internal CA, certificate for billing.internal.example), and a rogue backend with a self-signed certificate for app.internal.example. The Envoy counters show why a connection failed.

== No validation_context (encrypted, not authenticated)
  real      -> 200 
  impostor  -> 200 
  rogue     -> 200 

== Pinned internal CA only
  real      -> 200 
  impostor  -> 200 
  rogue     -> 503 (envoy: fail_verify_error: 1 )

== Pinned internal CA plus SAN match
  real      -> 200 
  impostor  -> 503 (envoy: fail_verify_san: 1 )
  rogue     -> 503 (envoy: fail_verify_error: 1 )

Without a validation context, the rogue backend gets the traffic. With the CA alone, any certificate from your CA is accepted for any backend. With both, only the real backend is.

Envoy Gateway renders the BackendTLSPolicy above with both checks:

== Envoy Gateway v1.9.1: BackendTLSPolicy with a pinned CA, rendered offline by egctl
  BackendTLSPolicy Accepted="True": Policy has been accepted.
  matchTypedSubjectAltNames:
  exact: app.internal.example
  sanType: DNS
  validationContextSdsSecretConfig:
  sni: app.internal.example

On a cluster, check the policy status and watch Envoy's counters for the backend cluster (ssl.fail_verify_error, ssl.fail_verify_san) through egctl x stats envoy-proxy or the admin endpoint.

#Mistakes people make

#Treating "TLS on" as "verified"

An UpstreamTlsContext without trusted_ca encrypts to anyone. Every upstream TLS context needs a validation context.

#Trusting the system store for internal backends

wellKnownCACertificates: System lets every public CA vouch for your backend. Use caCertificateRefs with your internal CA.

#Pinning the CA and skipping the name

With one internal CA for many services, every service certificate is valid for every backend. Set hostname (Gateway API) or match_typed_subject_alt_names (Envoy) so the edge checks the name.

#Setting insecureSkipVerify to get a deploy through

It turns the check off for good, and nobody remembers to turn it back on. Fix the certificate name or the CA instead.

#Forgetting Envoy's client TLS default

Envoy's upstream (client) TLS defaults to a maximum of TLS 1.2. Set the maximum to TLS 1.3 when you set the minimum, or the handshake fails.

#Checklist

• Put a BackendTLSPolicy on every backend Service the edge reaches over TLS.
• Use caCertificateRefs with your internal CA, not System.
• Set hostname to the DNS name in the backend certificate.
• Never set insecureSkipVerify.
• On standalone Envoy, set trusted_ca and match_typed_subject_alt_names.
• Set TLS 1.3 as both minimum and maximum on upstream contexts.
• Watch ssl.fail_verify_error and ssl.fail_verify_san for the backend clusters.