Operator secret stores with pass and an offline copy, the secure way

#What goes wrong

OpenBao holds the application secrets. The secrets that start OpenBao, and the ones you need when it is down, cannot live inside it. They end up in a shared password manager account, a text file on a laptop, or one person's memory.

Teams that do use pass encrypt every file to one shared GPG key. Everyone has the private key, so nobody can be removed without a new key for everyone.

The recipient list, .gpg-id, is a plain text file in git. Anyone who can push can add their own key. The next time someone saves a secret, it is encrypted to the intruder as well.

When a person leaves, the store is re-encrypted and everyone feels safe. Their old clone still has every old version, encrypted to their key.

#What the docs say

Multiple GPG keys can be specified, for using pass in a team setting, and different folders can have different GPG keys, by using -p.

Source: pass, the standard unix password manager

If this environment variable is set, then all .gpg-id files and non-system extension files must be signed using a detached signature using the GPG key specified by the full 40 character upper-case fingerprint in this variable.

Source: pass man page, PASSWORD_STORE_SIGNING_KEY

The docs explain how re-encryption works. They do not say that git history keeps every earlier version, so re-encrypting protects new values only. The test shows a removed member reading an old commit.

#The secure configuration

Each operator has their own GPG key, ideally on a hardware token. One more key, the backup key, is generated offline and never touches a networked machine. Before adding anyone's key, check the full fingerprint with them in person or over a call, then sign it locally:

gpg --import bob.asc
gpg --fingerprint [email protected]          # compare every character with bob
gpg --quick-lsign-key <bob's full fingerprint>

Set the signing key in every operator's shell profile. The value is the full 40-character upper-case fingerprint of the person allowed to change who can read the store. For more than one person, list several fingerprints separated by spaces:

# ~/.profile
export PASSWORD_STORE_SIGNING_KEY="<alice's 40-character upper-case fingerprint>"

Initialize with all operators and the backup key. Use folders for scope:

pass init <alice> <bob> <backup>              # root: all operators
pass init -p registrar <alice> <backup>       # a folder only two keys can read
pass git init
pass git remote add origin [email protected]:ops/pass-store.git

Store a secret from stdin, never as an argument:

pass insert prod/db             # no echo, asked twice
pass insert -m prod/tls-key     # multi-line: echoed on screen, end with Ctrl-D

When someone leaves, remove their key, re-encrypt, and rotate what they could read:

pass init <alice> <backup>      # re-encrypts every file under this .gpg-id
pass git push
# Then change each secret at its source (database, registrar, CA passphrase)
# and store the new value with: pass insert -f <name>

Make the offline copy after every change that matters, and at least monthly:

pass git bundle create /media/usb/pass-$(date +%Y%m%d).bundle --all
pass git bundle verify /media/usb/pass-$(date +%Y%m%d).bundle

Store the media in a safe. Store the backup key's hardware token or paper copy in a different safe.

#Prove it

From secure-tests/operator-secret-stores-pass-offline-copy/. Three people, each with a separate GnuPG home. The store is signed and each file has three recipients:

pass init <alice> <bob> <backup>
ls -a ~/.password-store
gpg --list-packets prod/db.gpg

Password store initialized for <alice>, <bob>, <backup>
.git .gitattributes .gpg-id .gpg-id.sig prod
gpg: encrypted with cv25519 key, ID 2EF6B23CEE6148A4
      "backup <[email protected]>"
gpg: encrypted with cv25519 key, ID CE1119FDCA284540
      "bob <[email protected]>"
gpg: encrypted with cv25519 key, ID EC4A12D9A86867B3
      "alice <[email protected]>"

Someone with push access appends a key to .gpg-id. The next write is refused:

echo "0000000000000000000000000000000000000000" >> ~/.password-store/.gpg-id
pass insert -m prod/api

Signature for /srv/alice/.password-store/.gpg-id is invalid.
exit 1

Bob leaves. The store is re-encrypted to alice and the backup key:

pass init <alice> <backup>

Password store initialized for <alice>, <backup>
[main 2d964c6] Set GPG id to <alice>, <backup>.
 1 file changed, 1 deletion(-)
[main e48ba91] Signing new GPG id with <alice>.
 1 file changed, 0 insertions(+), 0 deletions(-)
prod/db: reencrypting to 2EF6B23CEE6148A4 EC4A12D9A86867B3
[main 4b37db2] Reencrypt password store using new GPG id <alice>, <backup>.
 1 file changed, 0 insertions(+), 0 deletions(-)

Bob cannot read the current file:

GNUPGHOME=~bob/.gnupg gpg -d prod/db.gpg

gpg: encrypted with ECDH key, ID 2EF6B23CEE6148A4
gpg: encrypted with cv25519 key, ID EC4A12D9A86867B3
      "alice <[email protected]>"
gpg: public key decryption failed: No secret key
gpg: decryption failed: No secret key

He can still read the version in his old clone:

git log --format='%h %s'
git show ec21e13:prod/db.gpg | GNUPGHOME=~bob/.gnupg gpg -d

4b37db2 Reencrypt password store using new GPG id <alice>, <backup>.
e48ba91 Signing new GPG id with <alice>.
2d964c6 Set GPG id to <alice>, <backup>.
ec21e13 Add given password for prod/db to store.
166c81f Configure git repository for gpg file diff.
b9d2f06 Add current contents of password store.
example-db-password-1

That is why the next step is rotation. After it, the offline bundle is made and restored with nothing but the bundle and the backup key:

pass git bundle verify /media/usb/pass-*.bundle
git clone /media/usb/pass-*.bundle /restore/.password-store
GNUPGHOME=<backup key home> PASSWORD_STORE_DIR=/restore/.password-store pass show prod/db

The bundle contains these 2 refs:
pass-20260924.bundle is okay
e8d2b1e4ebb53dce1cc699248a5525df2a93f0bc refs/heads/main
example-db-password-2

#Mistakes people make

#One shared key for the team

Everyone holds the same private key, so nobody can be removed on their own. Encrypt to each person's key.

#An unsigned .gpg-id

Without PASSWORD_STORE_SIGNING_KEY, a single push can add a reader to every future secret, and nobody notices. Set it in every operator's profile.

#Re-encrypting and calling it done

Git history and old clones keep the old ciphertext, readable with the old key. Rotate every value the person could read.

#Adding a key without checking the fingerprint

A key from a mail or a key server may not belong to the person it names. Compare the full fingerprint over a trusted channel, then lsign it.

#The backup key on a laptop

The backup key exists for the day every laptop is lost. If it sits on one, it is just another operator key. Generate it offline and keep it offline.

#Secrets as command arguments

pass insert name prompts for the value without echo and asks twice. -m reads until Ctrl-D and shows what you type, so use it only for multi-line values, with nobody watching the screen. Never pipe a secret from echo in an interactive shell, where it lands in history (see never put a secret on the command line).

#An offline copy nobody has restored

A bundle that has never been cloned and decrypted is a hope. Restore it twice a year on an offline machine with the backup key.

#Checklist

• Give each operator their own GPG key, on a hardware token where possible.
• Generate the backup key offline and store it apart from the offline copy.
• Verify every fingerprint over a trusted channel before lsign.
• Set PASSWORD_STORE_SIGNING_KEY in every operator's profile.
• Use pass init -p to limit sensitive folders to fewer keys.
• On departure, run pass init without the person's key and push.
• On departure, rotate every secret the person could read.
• Create a git bundle on offline media after important changes, at least monthly.
• Restore the bundle with only the backup key twice a year.