Zone files in git, reconciled to PowerDNS, the secure way

#What goes wrong

Zone files in git give you review, history and rollback. PowerDNS stores zones in a database and offers an HTTP API. The two drift apart in three ways.

First, people and tools edit through the API. An ACME client adds TXT records, an engineer fixes a record during an incident. Git no longer matches production, and the next deploy either fails or silently reverts the fix.

Second, a bad file reaches production. A CNAME next to an A record, a typo in a name: PowerDNS accepts some of these into the database and serves broken data.

Third, the serial goes backwards. The API bumps the serial on each change. A deploy from git then loads an older serial, and secondaries that compare serials ignore the new zone.

Around all of this sits the API itself. It can change every zone. The official container image enables it on all interfaces if you set one environment variable.

#What the docs say

To achieve defense-in-depth, expose the webserver only to client addresses that have a real need for access, and configure a webserver password.

Source: PowerDNS docs, Built-in Webserver and HTTP API

Static pre-shared authentication key for access to the REST API. Since 4.6.0 the key can be hashed and salted using pdnsutil hash-password instead of being stored in the configuration in plaintext

Source: PowerDNS docs, api-key

If such a variable is found, /etc/powerdns/recursor.d/_api.conf / /etc/powerdns/pdns.d/_api.conf / /etc/dnsdist/conf.d/_api.conf is written, enabling the webserver in all products, and the dnsdist console.

Source: PowerDNS Docker README

The Docker README does not say what that file contains. The test shows it: webserver-address=0.0.0.0 and webserver-allow-from=0.0.0.0/0, with the key in plain text. The PowerDNS docs also have no guidance for keeping a database-backed zone in line with files in git; the reconciler below fills that gap with pdnsutil alone.

#The secure configuration

API settings, /etc/powerdns/pdns.d/api.conf. Do not use the image's PDNS_AUTH_API_KEY variable:

api=yes
api-key=$scrypt$ln=10,p=1,r=8$...     # from: pdnsutil hash-password
webserver=yes
webserver-address=127.0.0.1            # only local tools; put a proxy with auth in front if needed
webserver-port=8081
webserver-allow-from=127.0.0.1

The reconciler, run on the primary by a service account that can use pdnsutil. It is given a file from a reviewed, merged commit:

#!/bin/sh
# reconcile.sh check|apply ZONE FILE
# Runs ON the PowerDNS primary, fed with a reviewed zone file from git.
#   check: validate FILE in a scratch database, then diff it against the live zone. Exit 1 on drift.
#   apply: validate, replace the live zone atomically, keep the SOA serial moving forward,
#          rectify, notify. DNSSEC keys and zone metadata are not touched.
# Git owns the records; the reconciler owns the SOA serial (it is left out of the diff).
set -eu
mode=$1; zone=$2; file=$3
scratch=$(mktemp -d)
trap 'rm -rf "$scratch"' EXIT
cat > "$scratch/pdns.conf" <<C
launch=gsqlite3
gsqlite3-database=$scratch/db.sqlite3
gsqlite3-dnssec=yes
C
sqlite3 "$scratch/db.sqlite3" < /usr/local/share/doc/pdns/schema.sqlite3.sql
if ! pdnsutil --config-dir="$scratch" zone load "$zone" "$file" > "$scratch/load" 2>&1; then
  echo "REFUSED: $file does not load"; tail -2 "$scratch/load"; exit 2
fi
if ! pdnsutil --config-dir="$scratch" zone check "$zone" > "$scratch/check" 2>&1; then
  echo "REFUSED: $file fails zone check"; grep -v '^Checked' "$scratch/check" | head -3; tail -1 "$scratch/check"; exit 2
fi
norm() { awk '$4=="SOA"{$7="<serial>"} $4!~/^(RRSIG|NSEC|NSEC3|NSEC3PARAM|DNSKEY)$/ {print}' | sort; }
pdnsutil --config-dir="$scratch" zone list "$zone" | norm > "$scratch/want"
pdnsutil zone list "$zone" | norm > "$scratch/have"
if diff -u "$scratch/have" "$scratch/want" > "$scratch/diff"; then
  echo "in sync: $zone"; exit 0
fi
echo "DRIFT in $zone (- live, + git):"; sed -n '3,$p' "$scratch/diff" | grep '^[-+]'
[ "$mode" = check ] && exit 1
old=$(pdnsutil zone list "$zone" | awk '$4=="SOA"{print $7}')
pdnsutil zone load "$zone" "$file" >/dev/null 2>&1
new=$(pdnsutil zone list "$zone" | awk '$4=="SOA"{print $7}')
while [ "$new" -le "$old" ]; do          # never let the serial go backwards: secondaries would ignore it
  pdnsutil zone increase-serial "$zone" >/dev/null 2>&1
  new=$(pdnsutil zone list "$zone" | awk '$4=="SOA"{print $7}')
done
pdnsutil zone rectify "$zone" >/dev/null 2>&1
pdns_control notify "$zone" >/dev/null 2>&1 || true
echo "applied: $zone matches $file, serial $old -> $new"

The flow around it:

pull request  ->  CI: reconcile check (scratch validation + diff shown in the review)
merge         ->  primary pulls the merged commit, verifies its signature, runs reconcile apply
every 15 min  ->  reconcile check; exit 1 pages the DNS owner

The primary pulls; CI never holds database or API credentials for production. Records that must change outside git, such as ACME DNS-01 challenges, go in a separate delegated zone (for example _acme-challenge.example.com delegated to a zone the reconciler does not manage).

#Prove it

The test (secure-tests/zone-files-git-reconciled-powerdns/run.sh) runs PowerDNS 5.1.4 in the official image.

The trap: what PDNS_AUTH_API_KEY writes:

docker run -e PDNS_AUTH_API_KEY=changeme ... ; cat /etc/powerdns/pdns.d/_api.conf

webserver
api
api-key=changeme
webserver-address=0.0.0.0
webserver-allow-from=0.0.0.0/0
webserver-password=changeme

The secure setup stores the key hashed and answers only on localhost. A wrong key is refused:

api-key is stored hashed: $scrypt$ln=10,p=1,r=8$1o...
$ api wrong-key GET /zones
401 Unauthorized
$ api <key> GET /zones
200 OK

The signed zone at the start:

pdnsutil zone list-keys example.com

example.com                   CSK  Act Pub 256     ECDSAP256SHA256 1    cryptokeys  38029

A file with a CNAME next to an A record is refused before live is touched:

reconcile check example.com broken.zone; echo "exit $?"

REFUSED: /tmp/repo/broken.zone fails zone check
[Error] CNAME www.example.com found, but other records with same label exist.
Checked 6 records of 'example.com', 1 errors, 0 warnings.
exit 2

The scheduled check on a clean zone:

$ reconcile check example.com example.com.zone
in sync: example.com
exit 0

Someone adds a record through the API. The API bumps the serial to 2026092402; the file in git still says 2026092401:

$ api <key> PATCH /zones/example.com. {add TXT "temporary" at debug.example.com.}
204 No Content

$ reconcile check example.com example.com.zone
DRIFT in example.com (- live, + git):
-debug.example.com.	300	IN	TXT	"temporary"
exit 1

Apply makes live match git, and moves the serial forward instead of back to the file's value. The DNSSEC key is unchanged:

$ reconcile apply example.com example.com.zone
DRIFT in example.com (- live, + git):
-debug.example.com.	300	IN	TXT	"temporary"
applied: example.com matches /tmp/repo/example.com.zone, serial 2026092402 -> 2026092403
exit 0
$ reconcile check example.com example.com.zone
in sync: example.com
exit 0

$ pdnsutil zone list-keys example.com   # DNSSEC keys survived the reload
example.com                   CSK  Act Pub 256     ECDSAP256SHA256 1    cryptokeys  38029

#Mistakes people make

#Using the image's API variable in production

PDNS_AUTH_API_KEY is a quick-start convenience. It binds the API to all interfaces, allows every source address, and stores the key in clear text. Write your own api.conf instead.

#Letting CI push to the API

A CI system with a production API key can change every zone, and CI systems run code from many people. Let the primary pull reviewed commits, or give CI a key that reaches only a staging server.

#Comparing serials in the diff

If the serial is part of the diff, every API bump and every signature change looks like drift, and people learn to ignore the alert. Leave the serial to the reconciler and keep it moving forward.

#Reverting an incident fix without a word

The drift alert is the moment to ask why someone changed live. Either the fix goes into git through a pull request, or the reconciler removes it. Do not auto-apply on drift without a human reading the diff.

#Mixing machine-written records into a git-managed zone

ACME challenge records and similar automated entries will always drift. Delegate them to their own zone, so the zone in git stays fully owned by git.

#Checklist

• Keep every zone as a zone file in git, changed only by pull request.
• Run reconcile check in CI and show the diff in the review.
• Refuse files that fail pdnsutil zone check before touching live.
• Apply only merged, signed commits, pulled by the primary.
• Keep the SOA serial out of the diff and never let it decrease.
• Run reconcile check every 15 minutes and page on exit code 1.
• Bind the API to 127.0.0.1 with webserver-allow-from=127.0.0.1.
• Store the API key hashed with pdnsutil hash-password.
• Never use PDNS_AUTH_API_KEY outside a lab.
• Delegate ACME challenge names to a separate zone.