Building your own GeoIP CDN, the secure way

#What goes wrong

A CDN, at its simplest, is two things: several points of presence (PoPs) that serve your site, and DNS that sends each user to the nearest one. The second part is GeoDNS. The nameserver looks at where the query comes from and answers with a different address per region.

DNSSEC makes this harder. Every answer needs a signature, and a GeoDNS answer is not in your zone file: the edge makes it up per region. So the edge has to sign it, which means the edge needs a signing key. Put the KSK there and a stolen edge can take over your whole domain, including the key the parent zone trusts.

The usual fix is to keep the KSK on a hidden signer and give the edges only the ZSK. That part is well understood. The part that bites later is time. Knot's geoip module signs its answers once, when it loads, and those signatures have a lifetime. Nothing re-signs them on its own. When they expire, validating resolvers in the regions that get tailored answers return SERVFAIL. Users who get the default answer, often including you, see nothing wrong.

#What the docs say

If automatic DNSSEC signing is disabled, it's possible to combine externally pre-signed zone with module pre-signing of the alternative RRsets when the module is loaded. In this mode, only ZSK has to be present in the KASP database.

Source: Knot DNS docs, geoip module, Module signing

DNSSEC keys for computing record signatures MUST exist in the KASP database or be generated before the module is launched, otherwise the module fails to compute the signatures and does not load.

Source: Knot DNS docs, geoip module, DNSSEC support

Clients from the specified subnets will receive the responses defined in the module config. Others will receive the default records defined in the zone (if any).

Source: Knot DNS docs, geoip module, Using subnets

"When the module is loaded" is the sentence to remember. The docs do not say what happens when those signatures reach the end of their lifetime. The lab below does.

#The secure configuration

The shape:

hidden signer (KSK + ZSK)  --AXFR over TSIG-->  edge 1 (ZSK only, geoip)  --> users in region A
      not reachable from                         edge 2 (ZSK only, geoip)  --> users in region B
      the internet

1. The signer. It holds both keys and signs the zone. Only the edges may transfer it, and only with the TSIG key:

# signer: knot.conf (excerpt)
key:
  - id: edge-1
    algorithm: hmac-sha256
    secret: <from keymgr -t edge-1 hmac-sha256>
remote:
  - id: edge-1
    address: 10.60.0.53@53
    key: edge-1
acl:
  - id: xfr-edge
    address: 10.60.0.53
    key: edge-1
    action: transfer
policy:
  - id: manual
    manual: on
    algorithm: ecdsap384sha384
zone:
  - domain: example.com
    file: /config/example.com.zone
    dnssec-signing: on
    dnssec-policy: manual
    notify: edge-1
    acl: xfr-edge

The zone keeps a default record for every name the edges tailor. Users from unknown places get it, and Knot needs it for the NSEC chain:

www  A  203.0.113.10

2. The keys. Generate both on the signer, then copy the ZSK's private key, and only that key, to each edge:

keymgr -c knot.conf example.com. generate algorithm=ecdsap384sha384 ksk=yes zsk=no
keymgr -c knot.conf example.com. generate algorithm=ecdsap384sha384 ksk=no zsk=yes
keymgr -c knot.conf example.com. list        # note the ZSK's key id
# on the edge: import it with absolute timestamps in the past
keymgr -c kasp.conf example.com. import-pem zsk.pem \
  algorithm=ecdsap384sha384 ksk=no zsk=yes publish=20250101000000 active=20250101000000

Move it over an authenticated channel (a secrets store the edge reads, not a file copy by hand), and treat the edge's key directory as a secret.

3. The edge. A plain secondary: it does not sign the zone. Only the geoip module signs, with a manual policy that expects no KSK:

# edge: knot.conf (excerpt)
server:
    edns-client-subnet: on          # use the client's subnet when a resolver sends it
remote:
  - id: signer
    address: 10.60.0.10@53
    key: edge-1
acl:
  - id: notify-from-signer
    address: 10.60.0.10
    key: edge-1
    action: notify
policy:
  - id: presigned
    manual: on
    unsafe-operation: no-check-keyset   # the zone's KSK is not here, by design
    algorithm: ecdsap384sha384
    rrsig-lifetime: 14d
mod-geoip:
  - id: geo
    config-file: /config/geo.conf
    mode: subnet                    # or geodb with a MaxMind-format database
    ttl: 60
    dnssec: on
    policy: presigned
zone:
  - domain: example.com
    master: signer
    acl: notify-from-signer
    module: mod-geoip/geo

offline-ksk: on in the same policy also satisfies the key check; both loaded in the lab. The module's answers, per region:

# geo.conf
www.example.com:
  - net: 10.61.0.0/24
    A: 198.51.100.10               # PoP for region A
  - net: 10.62.0.0/24
    A: 192.0.2.10                  # PoP for region B

4. Re-sign on a schedule. Run knotc reload on every edge at least daily, from a timer or a sidecar. With a 14-day rrsig-lifetime, that leaves nearly two weeks of margin for a missed run. Also reload after every ZSK change. Then watch it: alert when the RRSIG on a geo answer, queried from each region, expires in less than a few days (see monitoring DNSSEC signature expiry).

5. Taking a PoP out. Change its lines in geo.conf to point at another PoP and run knotc reload. The module TTL (60 seconds here) bounds how long resolvers keep sending users to the dead PoP.

The PoPs themselves are ordinary edge sites; see multi-region edge PoPs on Kubernetes.

#Prove it

A lab in containers: a Knot 3.5.8 signer, a Knot 3.5.8 edge configured as above (with rrsig-lifetime: 10m so expiry happens during the test), and clients on two networks, 10.61.0.0/24 for region A and 10.62.0.0/24 for region B.

The keys, on each side:

signer:
9fcc27ba93ad7cd5ab540d6da482e54d7011280f 60256 KSK ECDSAP384SHA384 created=1790317702 publish=1790317702 ready=1790317702 active=1790317702
cf78eb451d8b59e79691f81b3b312771374e7884 29862 ZSK ECDSAP384SHA384 created=1790317703 publish=1790317703 active=1790317703
edge:
cf78eb451d8b59e79691f81b3b312771374e7884 29862 ZSK ECDSAP384SHA384 publish=1735689600 active=1735689600

The edge pulled the signed zone with the TSIG key:

[example.com.] AXFR, incoming, remote 10.60.0.10@53 TCP, key edge-1., finished, remote serial 2, 0.00 seconds, 1 messages, 1731 bytes

1. Every region validates, and gets its own PoP. delv with the KSK as trust anchor, from a client in each network:

region A  ; fully validated www.example.com.	60	IN	A	198.51.100.10
   RRSIG key tag 29862, expires 20260925063935
region B  ; fully validated www.example.com.	60	IN	A	192.0.2.10
   RRSIG key tag 29862, expires 20260925063935
other     ; fully validated www.example.com.	300	IN	A	203.0.113.10
   RRSIG key tag 29862, expires 20261009062824

Look at the expiry dates. The default answer carries the signer's signature, valid for two weeks. The geo answers carry the edge's, made when the module loaded at 06:29:35 and valid for the 10 minutes of this lab policy.

2. Past that time, with no reload:

region A  ;; validating www.example.com/A: verify failed due to bad signature (keyid=29862): RRSIG has expired
          ;; validating www.example.com/A: no valid signature found
region B  ;; validating www.example.com/A: verify failed due to bad signature (keyid=29862): RRSIG has expired
          ;; validating www.example.com/A: no valid signature found
other     ; fully validated www.example.com.	300	IN	A	203.0.113.10

The zone never changed and the key never changed. Only time passed. A validating resolver in region A or B now returns SERVFAIL; everyone else is fine.

3. knotc reload, ten seconds later:

Reloaded
region A  ; fully validated www.example.com.	60	IN	A	198.51.100.10
   RRSIG key tag 29862, expires 20260925065023
region B  ; fully validated www.example.com.	60	IN	A	192.0.2.10
   RRSIG key tag 29862, expires 20260925065023

New signatures, no restart, no gap in answers.

4. Resolvers that forward the client's subnet. Asked from a third network, with EDNS Client Subnet:

subnet 10.61.0.0/24 -> 198.51.100.10
subnet 10.62.0.0/24 -> 192.0.2.10
no ECS              -> 203.0.113.10

5. PoP failover. Region A's line pointed at region B's PoP, then knotc reload:

region A  ; fully validated www.example.com.	60	IN	A	192.0.2.10

6. An edge that insists on a KSK. The same edge with neither unsafe-operation: no-check-keyset nor offline-ksk: on:

$ knotc -c knot.conf conf-check
Configuration is valid
error: [example.com.] DNSSEC, keys validation failed (missing active KSK or ZSK)
error: [example.com.] module 'mod-geoip/geo', failed to load DNSSEC keys
error: [example.com.] module 'mod-geoip/geo', failed to load (missing active KSK or ZSK)
error: config, failed to activate modules (invalid module)
error: [example.com.] zone cannot be activated (invalid module)

The config check passes and the zone does not come up at all. Test a changed edge on a spare instance before it serves users.

#Mistakes people make

#No periodic reload

The geo signatures age from the moment the module loads. A long uptime is a countdown. Reload on a timer, and alert on the expiry of a geo answer's RRSIG from each region.

#Trusting checks from one place

Your monitor and your laptop probably get the default answer, signed by the signer and valid for weeks. Query and validate from a client in every region, or with ECS for every region's subnet.

#Thinking the ZSK on the edge is harmless

The ZSK signs zone data. Whoever steals an edge can sign any record in the zone until you replace that key. What the split buys you is recovery: roll a new ZSK on the signer and push it to the edges, with no change at the parent (see emergency DNSSEC key revocation). Put the KSK on an edge and the recovery goes through your registrar.

#A tailored name with no default record

Clients outside every listed subnet get the zone's record. Without one they get an empty answer, and Knot cannot build a correct NSEC chain for it.

#Believing conf-check

It validates syntax, not whether the geo module can find its keys. The zone only fails when knotd loads it.

#Checklist

• The KSK and the signer are not reachable from the internet.
• Zone transfers to the edges use TSIG, and the signer allows only the edges.
• Each edge holds only the ZSK, delivered through an authenticated channel.
• The edge policy is manual, with no-check-keyset (or offline-ksk), and a known rrsig-lifetime.
• Every tailored name has a default record in the zone.
• knotc reload runs on every edge at least daily, and after every ZSK change.
• Geo answers are validated from every region, and their RRSIG expiry is alerted on.
• Taking a PoP out is one geo.conf edit and a reload, and the module TTL is short.