Git forge disaster recovery, the secure way

#What goes wrong

A forge backup fails in the same few ways every time:

• The backup lives on the forge's own disk or in a bucket the forge's credentials can delete. Ransomware or a bad cleanup script takes both.
• The database and repositories are copied at different moments, so the restore has pull requests that point at commits that do not exist.
• The backup includes the data but not SECRET_KEY, so every 2FA secret and encrypted value is unreadable after restore.
• The backup includes SECRET_KEY in plain text next to the data, so anyone who reads the backup bucket can decrypt everything.
• Nobody has restored it, so nobody knows it does not work.

The last one is the one that hurts.

#What the docs say

The reliable way to perform a backup is with a synchronized point-in-time snapshot of all the storage used by Forgejo.

Source: Forgejo docs, Upgrade guide

Although the zip file created by forgejo dump contains a copy of the database it has serious long standing open bugs that may introduce problems when re-injecting the SQL dump in a new database.

Source: Forgejo docs, Upgrade guide

Bundles are used for the "offline" transfer of Git objects without an active "server" sitting on the other side of the network connection.

Source: Git docs, git-bundle

But keep in mind that these tools will not help you backup state other than refs and commits.

Source: Git docs, git-bundle

The Forgejo docs say what a reliable backup is, but not that forgejo dump also packs app.ini (with SECRET_KEY and every other secret) into the zip. The test below shows it does, which makes the dump a secret in its own right.

#The secure configuration

#Copy 1: git bundles, from outside

Run from a backup host that the forge cannot log in to, with a read-only token:

#!/usr/bin/env bash
# forge-bundles.sh: one verified bundle per repository.
set -euo pipefail
DATE=$(date +%F)
OUT=/srv/forge-backup/$DATE
mkdir -p "$OUT"
while read -r repo; do                       # e.g. "team-a/app", from the API
  name=${repo//\//__}
  git clone --quiet --mirror "https://git.example.com/${repo}.git" "$OUT/$name.git"
  git -C "$OUT/$name.git" bundle create "../$name.bundle" --all
  git -C "$OUT/$name.git" bundle verify "../$name.bundle" >/dev/null
  rm -rf "$OUT/$name.git"
done < repos.txt
( cd "$OUT" && sha256sum ./*.bundle > SHA256SUMS )
# Upload $OUT to object storage with object lock / versioning, using a
# credential that can write but not delete.

Bundles restore anywhere Git runs, with no forge at all. They carry refs and commits only: no issues, pull requests, wiki settings, users or Actions secrets.

#Copy 2: a consistent snapshot of everything

• PostgreSQL: a base backup with WAL archiving (or your database operator's backup), so you can restore to a point in time.
• The Forgejo data directory (repositories, LFS, attachments, packages): a filesystem or volume snapshot taken right after the database checkpoint, or with Forgejo briefly stopped.
• app.ini and the secrets in it: stored in your secret manager, not only in the snapshot. SECRET_KEY also kept offline, split between two people.

Encrypt snapshots before they leave the host (for example with age to a recipient key whose private half is offline), and store them in a different account from the forge.

forgejo dump is useful for a quick copy before an upgrade. Treat its zip as containing every secret the forge has.

#Restore drills

Every quarter, on a throwaway machine:

1. Restore the snapshot and start Forgejo. Log in with 2FA; open an issue and a pull request; confirm Actions secrets still decrypt.
2. Restore five random repositories from bundles and compare their refs with the live forge.
3. Write down how long it took and what was missing.

#Prove it

A throwaway Forgejo with one private repository (two branches and an annotated tag). Backup 1, from outside the forge:

git clone --mirror https://git.example.com/site-admin/app.git app.git
git -C app.git bundle create ../app.bundle --all && git -C app.git bundle verify ../app.bundle

The bundle contains these 4 refs:
<sha> refs/heads/feature
<sha> refs/heads/main
<sha> refs/tags/v1.0.0
<sha> HEAD
The bundle records a complete history.
The bundle uses this hash algorithm: sha1
../app.bundle is okay

Backup 2, inside the container:

forgejo dump --file /backup/forgejo-dump.zip --tempdir /tmp/gitea
unzip -l forgejo-dump.zip    # top-level entries

app.ini
data
forgejo-db.sql
repos

app.ini is in the zip, so the dump holds SECRET_KEY, INTERNAL_TOKEN and the JWT secrets. Encrypt it or do not keep it.

Restore the repository from the bundle alone and compare every ref:

git clone --mirror app.bundle restored.git
diff <(git -C app.git for-each-ref) <(git -C restored.git for-each-ref)
git -C restored.git cat-file -p v1.0.0 | tail -1

identical: 3 refs
<sha> refs/heads/feature
<sha> refs/heads/main
<sha> refs/tags/v1.0.0
release 1.0.0

(Commit hashes replaced with <sha> because they change on every run.) The annotated tag and its message came back too. A full database restore from the dump was not part of this test; the Forgejo docs warn about exactly that path.

#Mistakes people make

#Push mirrors as the backup

A push mirror copies whatever happens, including a force-push that wipes a branch. Forgejo's own docs note that LFS objects are not mirrored over SSH. Mirrors are for availability; bundles in locked storage are for recovery.

#Backups the forge can delete

If the forge host holds a credential that can delete backups, an attacker on the forge can delete them too. Write-only credentials and object lock.

#Snapshotting the database and the disk at different times

Pull requests, releases and LFS pointers in the database must match the repositories on disk. Take the snapshot of the data directory right after the database checkpoint, or stop Forgejo for the few seconds it takes.

#SECRET_KEY only inside the backup

If the only copy of SECRET_KEY is in app.ini inside an encrypted backup whose key is on the forge, you have a locked box with the key inside. Keep the secrets in two places, one of them offline.

#Never restoring

A backup that has never been restored is a hope, not a backup. Schedule the drill and keep the notes.

#Checklist

• A host outside the forge creates, verifies and checksums a bundle of every repository daily.
• Bundles go to storage with versioning or object lock, written with a credential that cannot delete.
• The database has point-in-time backups; the data directory has matching snapshots.
• app.ini secrets and SECRET_KEY are in a secret manager and in an offline copy.
• Snapshots and dumps are encrypted before leaving the host, since they contain every secret.
• Backups live in a different account from the forge.
• A restore drill runs every quarter and covers login with 2FA, issues, pull requests and Actions secrets.
• Bundle restores are compared ref by ref with the live forge.