Let's Encrypt DNS-01 over RFC 2136 and TSIG, the secure way

#What goes wrong

The quick setup is allow-update { key acme; }; on the main zone. That key can now change every record: the MX, the A record for the login page, the NS set. It sits in a Kubernetes Secret, readable by anyone with get secrets in the cert-manager namespace.

With that key, an attacker does not need to break TLS. They point login.example.com at their own server and ask Let's Encrypt for a certificate. The DNS-01 challenge passes, because they control the DNS.

Some teams leave updates open to an address range instead of a key. Anyone on that network, or anyone who can send spoofed UDP from it, can then update the zone.

#What the docs say

Since DNS servers are commonly exposed to the public internet, being able to push an unauthenticated update to any server that responds to queries would be immediately untenable.

Source: cert-manager docs, RFC-2136

By default, cert-manager will not follow CNAME records pointing to subdomains.

Source: cert-manager docs, DNS01: delegated domains

The update-policy clause allows more fine-grained control over which updates are allowed.

Source: BIND 9 reference manual, update-policy

Some sites choose to keep all dynamically updated DNS data in a subdomain and delegate that subdomain to a separate zone.

Source: BIND 9 reference manual, Dynamic Update Security

cert-manager's RFC 2136 page shows a key and a solver, and links to outside tutorials for the server side. It does not say that what the key can change is decided on the DNS server, and that this is the part that matters.

#The secure configuration

Generate the key on the DNS server:

tsig-keygen -a hmac-sha256 acme-cert-manager. > /etc/bind/keys/acme-cert-manager.key
chown root:bind /etc/bind/keys/acme-cert-manager.key
chmod 0640 /etc/bind/keys/acme-cert-manager.key

The BIND configuration. The main zone is static; the ACME zone takes TXT updates from one key:

// named.conf
options {
    directory "/var/cache/bind";
    recursion no;
    allow-transfer { none; };            // or TSIG-protected transfers to your secondaries
};

include "/etc/bind/keys/acme-cert-manager.key";   // key "acme-cert-manager." (hmac-sha256)

zone "example.com" {
    type primary;
    file "/etc/bind/zones/example.com.zone";
    allow-update { none; };              // never touched by ACME
};

zone "acme.example.com" {
    type primary;
    file "/var/cache/bind/acme.example.com.zone";
    update-policy {
        grant acme-cert-manager. zonesub TXT;   // TXT only, only in this zone
    };
};

In the main zone, delegate the ACME zone and point each challenge name at it:

; example.com.zone
acme                 NS    ns1.example.com.
_acme-challenge.app  CNAME app.acme.example.com.

; acme.example.com.zone: empty apart from SOA and NS; cert-manager fills it.
$TTL 60
@  SOA  ns1.example.com. hostmaster.example.com. 1 3600 600 86400 60
@  NS   ns1.example.com.

In the cluster, the secret is created from the key file, and the solver follows the CNAME:

kubectl -n cert-manager create secret generic acme-tsig \
  --from-literal=secret="$(awk -F'"' '/secret/ {print $2}' acme-cert-manager.key)"

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: [email protected]               # use a real address; Let's Encrypt refuses @example.com
    privateKeySecretRef:
      name: letsencrypt-account-key
    solvers:
      - selector:
          dnsZones:
            - example.com
        dns01:
          cnameStrategy: Follow          # write the TXT where the CNAME points
          rfc2136:
            nameserver: 192.0.2.53:53    # the primary that accepts updates
            tsigKeyName: acme-cert-manager
            tsigAlgorithm: HMACSHA256    # the default is HMACMD5; always set it
            tsigSecretSecretRef:
              name: acme-tsig
              key: secret

Add a CAA record that names your ACME account, so a certificate for your names can only come from it (see CAA records bound to your ACME account).

#Prove it

The test in secure-tests/letsencrypt-dns-01-rfc2136-tsig/ runs BIND 9.20 as the primary and a second container as the client. First, the update cert-manager sends:

nsupdate -k acme-cert-manager.key <<'EOF'
server ns1
zone acme.example.com
update add app.acme.example.com. 60 TXT "example-challenge-token"
send
EOF
echo "nsupdate exit $?"
dig +noall +answer TXT _acme-challenge.app.example.com @ns1
dig +noall +answer TXT app.acme.example.com @ns1

nsupdate exit 0
_acme-challenge.app.example.com. 300 IN	CNAME	app.acme.example.com.
app.acme.example.com.	60	IN	TXT	"example-challenge-token"

Now use the same key for anything else. An A record in the ACME zone, a TXT record in the main zone, and replacing the main zone's A record for app:

update failed: REFUSED
nsupdate exit 2
update failed: REFUSED
nsupdate exit 2
update failed: REFUSED
nsupdate exit 2

An unsigned update to the ACME zone:

update failed: REFUSED
nsupdate exit 2

The server logs every refusal with the key name:

docker logs ns1 | grep -E "denied|rejected"

172.21.0.3#40970/key acme-cert-manager: updating zone 'acme.example.com/IN': update failed: rejected by secure update (REFUSED)
172.21.0.3#41447/key acme-cert-manager: update 'example.com/IN' denied
172.21.0.3#35305/key acme-cert-manager: update 'example.com/IN' denied
172.21.0.3#32855: update 'acme.example.com/IN' denied

The cleanup cert-manager does after the challenge works too:

nsupdate exit 0
(no answer)

The ClusterIssuer validates against cert-manager v1.21.2's CRDs:

Summary: 1 resource found in 1 file - Valid: 1, Invalid: 0, Errors: 0, Skipped: 0

#A full issuance on a cluster

The same BIND setup ran in a lab cluster, with the ClusterIssuer above pointed at Pebble, Let's Encrypt's ACME test server, and caBundle set to trust it. Two lab-only changes: cert-manager resolved names through the lab BIND (--dns01-recursive-nameservers and --dns01-recursive-nameservers-only), because public DNS does not know the lab zones; and Pebble queried an Unbound resolver in front of BIND. Pebble asking BIND directly got only the CNAME and failed with No TXT records found for DNS challenge; Let's Encrypt uses recursive resolvers that follow the CNAME, as Unbound did.

A Certificate for app.example.com, polled every 5 seconds:

t+5s  challenge=pending ready=False TXT@bind="EO4eVl7xdTl1kDwNBwm18AgjflJIw7XAcUETw8aPacM"
...
t+50s challenge=pending ready=False TXT@bind="EO4eVl7xdTl1kDwNBwm18AgjflJIw7XAcUETw8aPacM"
t+55s challenge=        ready=True  TXT@bind=

BIND's log shows the two updates, both signed with the key:

10.244.2.216#39733/key acme-cert-manager: updating zone 'acme.example.com/IN': adding an RR at 'app.acme.example.com' TXT "EO4eVl7xdTl1kDwNBwm18AgjflJIw...
10.244.2.216#51834/key acme-cert-manager: updating zone 'acme.example.com/IN': deleting an RR at app.acme.example.com TXT

The result:

$ kubectl -n dns get certificate app
NAME   READY   SECRET                AGE
app    True    app-example-com-tls   81s
$ kubectl -n dns get secret app-example-com-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -issuer -ext subjectAltName
issuer=CN = Pebble Intermediate CA 5f8300
X509v3 Subject Alternative Name: critical
    DNS:app.example.com
$ kubectl -n dns describe order | grep Complete
  Normal  Complete  20s   cert-manager-orders  Order completed successfully

The TXT record lived at app.acme.example.com only while the challenge ran, and the main zone was never written to.

On your cluster, with the real issuer, create a Certificate for your name and run kubectl get challenges -A -w. What you should see: the Challenge goes from pending to valid, then Order completed successfully, and the TXT record is gone.

#Mistakes people make

#allow-update { key ... } on the main zone

That grants every record type at every name. Use update-policy and limit the key to TXT in a zone that holds nothing else.

#A wildcard CNAME in the main zone

With cnameStrategy: Follow, cert-manager follows any CNAME it finds at _acme-challenge.<name>, including one made up by a * record. Create an explicit CNAME for each challenge name.

#Updates allowed by IP address

Addresses can be shared or spoofed over UDP. Require a TSIG key for every update.

#nsupdate -y in scripts

It puts the key on the command line, in ps and shell history. Use nsupdate -k with a key file.

#One key for everything

A key shared with DHCP, zone transfers or a second cluster cannot be scoped or rotated without breaking the others. One key per client, named after it.

#No CAA record

With the key limited, the next weak point is any other CA. A CAA record with accounturi limits issuance to your ACME account.

#Checklist

• Create a zone used only for ACME challenges.
• Set allow-update { none; } on the main zone.
• Grant the cert-manager key zonesub TXT in the ACME zone only.
• Create one explicit _acme-challenge CNAME per name that needs a certificate.
• Set cnameStrategy: Follow on the solver.
• Generate a dedicated TSIG key for cert-manager with HMAC-SHA256 or stronger.
• Store the key as a Secret in the cert-manager namespace only.
• Test with nsupdate -k that the key is refused outside the ACME zone and for non-TXT types.
• Publish a CAA record with accounturi for your ACME account.
• Alert on denied and rejected update lines in the DNS server log.