Package repository keys and version locks, the secure way

#What goes wrong

APT and DNF check a signature on every repository. The question is which keys they accept, and for which repositories.

The old apt pattern, curl ... | apt-key add - or a key file dropped in /etc/apt/trusted.gpg.d/, makes the key global. apt then accepts that vendor's signature on any repository that has no key of its own, including Debian's. If the vendor's key leaks, or the vendor ships a package called openssh-server with a higher version, apt installs it.

The second problem is scope. Even with a scoped key, a repository can offer packages with the same names as the distribution's packages. Without a pin, apt picks the highest version, wherever it comes from.

The third is timing. An upgrade you did not plan can break a service in the middle of the day. Version locks let you choose when a package moves.

#What the docs say

The certificate MUST NOT be placed in /etc/apt/trusted.gpg.d or loaded by apt-key add.

Source: Debian wiki, DebianRepository/UseThirdParty

Signed-By (signed-by) is an option to require a repository to pass apt-secure(8) verification with a certain set of keys rather than all trusted keys apt has configured.

Source: sources.list(5), Debian 12

versionlock is a plugin that takes a set of names and versions for packages and excludes all other versions of those packages.

Source: DNF plugins, versionlock

The Debian wiki also says a malicious repository "can easily circumvent these protections", because package scripts run as root. Scoped keys and pins limit mistakes and stolen keys; they do not make an untrusted vendor safe. Add only repositories you would trust with root.

#The secure configuration

Debian and Ubuntu, using the nginx.org repository as the example:

# 1. Fetch the key into a directory only root can write, as a binary keyring.
install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://nginx.org/keys/nginx_signing.key | gpg --dearmor -o /etc/apt/keyrings/nginx.gpg
# 2. Compare the fingerprints with the ones the vendor publishes on its website.
gpg --show-keys --with-fingerprint /etc/apt/keyrings/nginx.gpg

# /etc/apt/sources.list.d/nginx.sources
Types: deb
URIs: https://nginx.org/packages/debian
Suites: bookworm
Components: nginx
# This key is trusted for this repository only.
Signed-By: /etc/apt/keyrings/nginx.gpg

# /etc/apt/preferences.d/nginx
# Nothing from nginx.org may be installed...
Package: *
Pin: origin nginx.org
Pin-Priority: -1

# ...except the nginx package itself, which may replace Debian's nginx.
Package: nginx
Pin: origin nginx.org
Pin-Priority: 900

apt-get update
apt-cache policy nginx          # candidate and where it comes from
apt-mark hold nginx             # stop upgrades of this package until you unhold it
apt-mark showhold

The Debian wiki recommends a pin of 100 or lower for third-party repositories. The 900 above is deliberate and limited to one package name, because the goal here is to replace Debian's nginx with the vendor's build.

Fedora, RHEL and Rocky: keep gpgcheck=1 on every repository, import vendor keys with rpm --import only after checking the fingerprint, and lock versions with the versionlock plugin:

dnf install -y python3-dnf-plugin-versionlock
dnf versionlock add openssl-libs      # lock at the installed version
dnf versionlock list
dnf versionlock delete openssl-libs   # when you are ready to move

#Prove it

Debian 12's own sources already use Signed-By:

grep -E "^(URIs|Suites|Signed-By)" /etc/apt/sources.list.d/debian.sources

URIs: http://deb.debian.org/debian
Suites: bookworm bookworm-updates
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
URIs: http://deb.debian.org/debian-security
Suites: bookworm-security
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg

The fingerprints in the nginx.org key file. The nginx.org install page asks you to find 573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62, and it is the second one; the file also carries two other nginx signing keys:

gpg --show-keys --with-fingerprint --with-colons /etc/apt/keyrings/nginx.gpg | awk -F: '/^fpr/ {print $10}'

8540A6F18833A80E9C1653A42FD21310B49F6B46
573BFD6B3D8FBC641079A6ABABF5BD827BD9BF62
9E9BE90EACBCDE69FE9B204CBCDCD8A38D88A2B3

With the source and the pin in place, nginx comes from nginx.org, and another package from the same repository cannot be installed at all:

apt-cache policy nginx | sed -n 1,4p
apt-cache policy nginx-module-njs | sed -n 1,3p
apt-mark hold nginx
apt-mark showhold

nginx:
  Installed: (none)
  Candidate: 1.30.5-1~bookworm
  Version table:
nginx-module-njs:
  Installed: (none)
  Candidate: (none)
nginx set on hold.
nginx

Point the same source at the wrong key, here Debian's own keyring, and apt refuses the repository:

apt-get update 2>&1 | grep -E "NO_PUBKEY|W:" | head -3

  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 2FD21310B49F6B46
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nginx.org/packages/debian bookworm InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 2FD21310B49F6B46
W: Failed to fetch https://nginx.org/packages/debian/dists/bookworm/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 2FD21310B49F6B46

Rocky Linux 9: signature checks are on for every stock repository, and a version lock hides a waiting update:

grep -E "^(gpgcheck|repo_gpgcheck|gpgkey)" /etc/yum.repos.d/rocky.repo | sort | uniq -c
rpm -q alternatives
dnf -q check-update alternatives; echo "check-update exit: $?"
dnf -q versionlock add alternatives
dnf -q versionlock list
dnf -q check-update alternatives; echo "check-update exit: $?"
dnf -q upgrade -y alternatives; rpm -q alternatives

      9 gpgcheck=1
      9 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Rocky-9
alternatives-1.24-1.el9.x86_64

alternatives.x86_64                      1.24-2.el9                       baseos
check-update exit: 100
Adding versionlock on: alternatives-0:1.24-1.el9.*
alternatives-0:1.24-1.el9.*
check-update exit: 0
alternatives-1.24-1.el9.x86_64

The test script is secure-tests/package-repository-keys-version-locks/run.sh.

#Mistakes people make

#Copying the vendor key into trusted.gpg.d

It works, and it trusts that key for every repository without Signed-By. Move third-party keys to /etc/apt/keyrings/ and add Signed-By to their sources. Check what is there with ls /etc/apt/trusted.gpg.d/.

#Never checking the fingerprint

Downloading a key over HTTPS proves it came from that website. Comparing the fingerprint with the one in the vendor's documentation proves it is the key the vendor meant. Do it once, when you add the repository.

#Setting gpgcheck=0 to "fix" an error

A signature error means the repository or the key is wrong. Turning the check off installs whatever the mirror serves. Fix the key instead.

#Holding packages forever

A hold or a versionlock also blocks security fixes for that package. Keep a list of what is locked and why, and review it every month.

#Checklist

• No third-party key is in /etc/apt/trusted.gpg.d/ or the legacy trusted.gpg.
• Every third-party apt source has Signed-By pointing to a file in /etc/apt/keyrings/.
• Every third-party key fingerprint was compared with the vendor's published value.
• Every third-party repository has a pin that limits which packages it can supply.
• Every repository on RPM systems has gpgcheck=1.
• apt-mark showhold or dnf versionlock list shows only packages with a written reason.
• Locked packages are reviewed for security fixes at least monthly.