Break-glass accounts, the secure way

#What goes wrong

The emergency account is a normal admin account with a scary name. One person knows the password, or everyone does. Either way, one person can use it alone, at any time, and nobody is told.

It is also used on normal days, "just this once", because it is faster. Now it is a daily account with no MFA and no expiry.

The OpenBao version is the initial root token, kept "for emergencies". It never expires, and anyone who finds it has everything.

When the emergency finally comes, the account does not work: the password was rotated by a policy nobody remembered, the MFA device left with a former employee, or the procedure depends on the system that is down.

#What the docs say

In fact, the OpenBao team recommends that root tokens are only used for just enough initial setup (usually, setting up auth methods and policies necessary to allow administrators to acquire more limited tokens) or in emergencies, and are revoked immediately after they are no longer needed.

Source: OpenBao docs, Tokens: root tokens

It is also good security practice for there to be multiple eyes on a terminal whenever a root token is live.

Source: OpenBao docs, Tokens: root tokens

Store the PAT offline according to your incident-access procedures.

Source: Zitadel docs, Hardening instance administrators: break-glass

Since OpenBao 2.5.3, the legacy unauthenticated /sys/generate-root endpoints are disabled by default, and since 2.6.0 bao operator generate-root uses the authenticated /sys/generate-root-token endpoints. The docs present that as closing an unauthenticated hole. It also gives you a clean two-part design: a break-glass login that can only start the process, and recovery share holders who finish it.

#The secure configuration

The rules, for any system:

• One break-glass identity per system, never used for daily work.
• Its power needs two parts held by different people: a login plus a quorum of shares, or a secret split across two sealed envelopes.
• Parts are stored offline (see operator secret stores with pass and an offline copy).
• Any use raises an alert to people who are not the user.
• Whatever it creates is revoked when the incident ends, and the parts are rotated.
• The procedure is rehearsed every quarter, on the real system.

For OpenBao with auto-unseal, the break-glass login may only drive root generation:

# breakglass-initiator.hcl: start, read, advance and cancel a root token
# generation. Nothing else. Without a quorum of recovery shares it cannot
# produce a token.
path "sys/generate-root-token/attempt" {
  capabilities = ["create", "update", "read", "delete"]
}
path "sys/generate-root-token/update" {
  capabilities = ["update"]
}

bao policy write breakglass-initiator breakglass-initiator.hcl
# The password goes in the offline store; use userpass or cert auth that does
# not depend on your IdP, which may be the thing that is down.
# breakglass.pass: mode 0600, written with printf '%s' (no trailing newline).
bao write auth/userpass/users/breakglass [email protected] \
  token_policies=breakglass-initiator token_ttl=30m token_max_ttl=30m

Keep the listener default disable_unauthed_generate_root_endpoints = true, and give out recovery shares to five named people (threshold three), each encrypted to their own PGP key at init time with -recovery-pgp-keys.

The procedure, run with two people watching:

bao login -method=userpass username=breakglass           # prompts for the password
bao operator generate-root -generate-otp > otp            # keep it in a 0600 file
bao operator generate-root -init -otp="$(cat otp)"
# Each share holder, on their own machine, enters their share on stdin:
bao operator generate-root -nonce=<nonce> -
# After the third share:
bao operator generate-root -decode=<encoded token> -otp="$(cat otp)"
# ... the emergency work ...
bao token revoke -self

Alert on any audit log entry for sys/generate-root-token/* and any login by breakglass. In LogQL, for audit logs shipped to Loki:

{job="openbao-audit"} | json | request_path=~"sys/generate-root-token/.*|auth/userpass/login/breakglass"

#Prove it

From secure-tests/break-glass-accounts/. After setup, the initial root token is revoked:

bao token revoke -self
bao token lookup   # with that root token

Success! Revoked token (if it existed)
Code: 403. Errors:
* permission denied

An ordinary admin cannot even see a root generation:

bao operator generate-root -status   # as alice

Code: 403. Errors:
* 1 error occurred:
* permission denied

The break-glass login has a 30-minute token, and on its own reads nothing:

bao login -method=userpass username=breakglass
bao kv get secret/app

{"display_name":"userpass-breakglass","policies":["breakglass-initiator","default"],"ttl":1799}
Code: 403. Errors:
* preflight capability check returned 403, please ensure client's policies grant access to path "secret/"

It starts the attempt; three recovery share holders complete it:

bao operator generate-root -init -otp=<otp>
bao operator generate-root -nonce=<nonce> -   # three times, one share each

{"started":true,"progress":0,"required":3}
{"progress":1,"required":3,"complete":false}
{"progress":2,"required":3,"complete":false}
{"progress":3,"required":3,"complete":true}

The decoded token is a root token that never expires. That is why the last step is not optional:

bao token lookup
bao token revoke -self
bao token lookup

{"policies":["root"],"ttl":0,"expire_time":null}
Success! Revoked token (if it existed)
Code: 403. Errors:
* permission denied

The audit log has every step, including alice's failed attempt:

auth/token/revoke-self  update  root  ok
auth/userpass/login/alice  update  userpass-alice  ok
sys/generate-root-token/attempt  read  userpass-alice  error
auth/userpass/login/breakglass  update  userpass-breakglass  ok
sys/generate-root-token/attempt  read  userpass-breakglass  ok
sys/generate-root-token/attempt  update  userpass-breakglass  ok
sys/generate-root-token/attempt  read  userpass-breakglass  ok
sys/generate-root-token/update  update  userpass-breakglass  ok
sys/generate-root-token/attempt  read  userpass-breakglass  ok
sys/generate-root-token/update  update  userpass-breakglass  ok
sys/generate-root-token/attempt  read  userpass-breakglass  ok
sys/generate-root-token/update  update  userpass-breakglass  ok
sys/generate-root-token/attempt  read  userpass-breakglass  ok
auth/token/revoke-self  update  root  ok

The log shows three share submissions, not who held them. Record the names in the incident ticket.

#Mistakes people make

#One person, one password

If one person can use it alone, it is not break-glass, it is a backdoor with a label. Split the power.

#Using it on normal days

Every use should be an incident. If it is used for convenience, find out what daily access is missing and grant that instead.

#Depending on the system that is broken

A break-glass login through your SSO fails when the SSO is down. Use a local login method, and keep the parts offline.

#Forgetting to revoke

The generated root token never expires. Revoke it before closing the incident, and rotate the break-glass password and, if they were exposed, the shares.

#No alert

A break-glass use that nobody hears about is indistinguishable from an attack. Alert on the login and on every sys/generate-root-token request.

#Never rehearsed

Shares get lost, people leave, OTP lengths change. Run the procedure each quarter, on the real system, and time it.

#Checklist

• Create one break-glass identity per critical system.
• Limit the OpenBao break-glass policy to sys/generate-root-token/*.
• Keep disable_unauthed_generate_root_endpoints at true.
• Encrypt recovery shares to named holders' PGP keys at init time.
• Store the break-glass password and shares offline, in separate places.
• Use a login method that does not depend on your IdP.
• Alert on break-glass logins and on sys/generate-root-token requests.
• Revoke the generated token at the end of every use.
• Rotate the break-glass password after every use.
• Rehearse the procedure every quarter and record the time it took.