OpenBao transit auto-unseal, the secure way

#What goes wrong

Shamir unsealing is safe and slow. Every restart needs three people with key shares. So teams switch to auto-unseal, and the shortcut shows.

The seal token is created from the root token, so it is a child of it. When someone revokes the root token (as they should), the seal token dies with it. The next restart fails and nobody knows why.

The seal token gets a broad policy such as transit/*. Anyone who reads it can now rotate, trim or delete the key that protects every secret in the main cluster.

The token is pasted into the seal stanza of the config file. That file goes into git, a ConfigMap or a backup.

The transit server is run in the same cluster as the main one, or on the same host. One outage then takes both down, and the main one cannot start until the transit one is back.

#What the docs say

it should probably be an orphan token, otherwise when the parent token expires or gets revoked the seal will break.

Source: OpenBao docs, Transit seal

Although the configuration file allows you to pass in BAO_TOKEN as part of the seal's parameters, it is strongly recommended to set these values via environment variables.

Source: OpenBao docs, Transit seal

Recovery keys cannot decrypt the root key, and thus are not sufficient to unseal OpenBao if the Auto Unseal mechanism isn't working.

Source: OpenBao docs, Seal/Unseal

The transit seal page does not say what the main server does when the transit server is sealed or unreachable. The Seal/Unseal page says only that access cannot be recovered until the seal mechanism is available again. The test below shows what that means in practice: the main server refuses to start and exits. Plan the transit server's availability as if the main cluster depends on it, because it does.

#The secure configuration

The transit server. It is small, it runs on its own host or cluster, and it is unsealed by hand with Shamir shares.

# transit.hcl: the OpenBao that holds the wrapping key
ui = false

storage "raft" {
  path    = "/openbao/file"
  node_id = "transit-1"
}

listener "tcp" {
  address         = "0.0.0.0:8200"
  tls_cert_file   = "/openbao/tls/transit.crt"
  tls_key_file    = "/openbao/tls/transit.key"
  tls_min_version = "tls13"          # no plaintext, no old TLS
}

api_addr     = "https://transit.example.com:8200"
cluster_addr = "https://transit.example.com:8201"

On the transit server, create the key, a two-path policy and the seal token:

bao operator init -key-shares=5 -key-threshold=3   # shares go to five people
bao secrets enable transit
# Not exportable and not deletable (both are the defaults; do not change them).
bao write -f transit/keys/autounseal type=aes256-gcm96

cat > autounseal-policy.hcl <<'EOF'
# The only two things the seal token may do.
path "transit/encrypt/autounseal" {
  capabilities = ["update"]
}
path "transit/decrypt/autounseal" {
  capabilities = ["update"]
}
EOF
bao policy write autounseal autounseal-policy.hcl

# Orphan: it survives revoking the root token that created it.
# Periodic with no max TTL: the seal renews it for as long as it is used.
bao token create -orphan -period=24h -policy=autounseal -display-name=seal-app

The main server. The token is not in the file.

# app.hcl: the main OpenBao, unsealed by the transit server
ui = false

storage "raft" {
  path    = "/openbao/file"
  node_id = "app-1"
}

listener "tcp" {
  address         = "0.0.0.0:8200"
  tls_cert_file   = "/openbao/tls/app.crt"
  tls_key_file    = "/openbao/tls/app.key"
  tls_min_version = "tls13"
}

seal "transit" {
  address     = "https://transit.example.com:8200"
  mount_path  = "transit/"
  key_name    = "autounseal"
  tls_ca_cert = "/openbao/tls/ca.crt"   # verify the transit server; never tls_skip_verify
  # token: from the BAO_TOKEN environment variable only
}

api_addr     = "https://bao.example.com:8200"
cluster_addr = "https://bao.example.com:8201"

# /etc/openbao/seal.env, mode 0600, owner openbao, read by the systemd unit
# with EnvironmentFile=/etc/openbao/seal.env. On Kubernetes, use a Secret
# mounted as an environment variable, never a ConfigMap.
BAO_TOKEN=s.EXAMPLE-not-a-real-token

Initialize the main server with recovery keys. It returns no unseal keys, because the transit server holds the wrapping key:

bao operator init -recovery-shares=5 -recovery-threshold=3

#Prove it

The test in secure-tests/openbao-transit-auto-unseal/ runs both servers on a throwaway Docker network. First, the transit server after a manual unseal:

bao status   # on transit

Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3

The seal token is orphan and periodic:

bao token create -orphan -period=24h -policy=autounseal -display-name=seal-app

policies: ["autounseal","default"]  orphan: true  renewable: true  lease_duration: 86400

With the seal token, reading or rotating the key is refused:

bao read transit/keys/autounseal
bao write -f transit/keys/autounseal/rotate

Code: 403. Errors:
	* permission denied
Code: 403. Errors:
	* permission denied

The main server initializes with recovery keys only, and is unsealed:

bao operator init -recovery-shares=5 -recovery-threshold=3
bao status

recovery keys returned: 5   unseal keys returned: 0
Seal Type                transit
Recovery Seal Type       shamir
Initialized              true
Sealed                   false

After a restart it unseals itself:

docker restart app && bao status

Seal Type                transit
Sealed                   false

Now the failure modes. Seal the transit server, then restart the main one. It does not start sealed and wait. It exits:

bao operator seal   # on transit
docker restart app; docker ps -a; docker logs app

Success! Vault is sealed.
Exited (1)
Error configuring seal "transit": Error making API request.
URL: PUT https://transit:8200/v1/transit/encrypt/autounseal
Code: 503. Errors:
* Vault is sealed

Unseal the transit server by hand and the main server starts again:

docker start app && bao status

Seal Type                transit
Sealed                   false

Revoke the seal token on the transit server. This is also your emergency stop for the main cluster: the next restart cannot unseal.

bao token revoke <seal token>
docker restart app; docker logs app

Success! Revoked token (if it existed)
Exited (1)
Error configuring seal "transit": Error making API request.
URL: PUT https://transit:8200/v1/transit/encrypt/autounseal
Code: 403. Errors:
* permission denied

#Mistakes people make

#The seal token is a child of root

A token created with the root token and without -orphan is revoked when the root token is revoked. Revoking root is correct practice, so the seal breaks at the next restart, often weeks later. Always use -orphan.

#A max TTL on the seal token

A periodic token with no explicit max TTL renews for as long as it is used. Add -explicit-max-ttl and the token dies on that date, together with every restart after it.

#transit/* in the policy

The seal needs update on two paths. Anything more lets a token holder rotate, trim or delete the key. Trimming old key versions makes old data impossible to decrypt.

#The token in the config file

The token parameter works, which is the problem. Config files end up in git and backups. Put it in the environment from a 0600 file or a Kubernetes Secret.

#Both servers in one failure domain

If the transit server runs in the same cluster, one outage stops both, and the main one cannot start until the transit one is unsealed by hand. Run it on separate hardware and give it its own Shamir key holders.

#Treating recovery keys as unseal keys

Recovery keys authorize operations such as generate-root. They cannot unseal the main server. If the transit server is lost for good, so is the main cluster's data. Back up the transit server's storage and its unseal shares.

#Copying disable_mlock from a Vault guide

OpenBao does not use mlock since 2.0.0. Remove the setting and disable swap for the process instead (see OpenBao hardening).

#Checklist

• Run the transit server on hardware separate from the main cluster.
• Initialize the transit server with Shamir shares held by at least three people.
• Create the transit key with exportable and deletion_allowed left false.
• Write a policy with update on transit/encrypt/<key> and transit/decrypt/<key> only.
• Create the seal token with -orphan and -period, and no explicit max TTL.
• Keep the seal token out of the config file; load it from a 0600 file or a Secret.
• Set tls_ca_cert in the seal stanza; never set tls_skip_verify.
• Store the main server's recovery shares with separate people.
• Back up the transit server's storage and test a restore.
• Alert when the main server fails to start with a seal error.