Secrets and PKI

OpenBao hardening, the secure way

The secrets manager is the one server where "it works" is not the finish line. A root token in a shell history and one audit device on a full disk will undo every policy you wrote.

The short answer

Run OpenBao as a non-root user with swap disabled for the process, TLS 1.3 on every listener, the web UI left off, and two audit devices declared in the config file. Use self-initialization so the root token is used once and revoked, and give people named logins with narrow, time-limited policies.

Updated Houssam Hammoudi, CTOTested with OpenBao 2.7.0, Docker 29 on cgroup v2

On this page
  1. What goes wrong
  2. What the docs say
  3. The secure configuration
  4. Prove it
  5. Mistakes people make
  6. Checklist

What goes wrong

OpenBao encrypts everything it stores. It cannot stop the kernel from writing its memory to swap, where decrypted secrets sit on disk in the clear.

The initial root token is printed once at bao operator init. It is then copied into a password manager, a chat message and a shell history, and it never expires.

One audit device writes to a disk that fills up. OpenBao refuses requests it cannot audit, so the outage looks like an OpenBao bug. The fix people reach for is to disable auditing.

When API audit creation is turned on, anyone with sudo on sys/audit/* can add an audit device that writes to a path or address of their choice. That is a quiet way to copy every request.

Settings copied from old guides are ignored. OpenBao prints a warning in the startup log and starts anyway.

What the docs say

To provide extra security, you will want to make sure that your OS has swap disabled or that its swap space is encrypted.

Source: OpenBao docs, Install: post-installation hardening

When running the Docker image, include the flag --memory-swappiness=0.

Source: OpenBao docs, Install: post-installation hardening

It is highly recommended that you configure OpenBao to use multiple audit devices.

Source: OpenBao docs, Audit devices

The root token is not returned to the caller and is revoked after use.

Source: OpenBao docs, Self-initialization

On a cgroup v2 host, Docker discards --memory-swappiness=0 with a warning and the container can still swap. The test below shows it. Set --memory-swap to the same value as --memory instead, which gives the container no swap at all.

The secure configuration

hcl
# /openbao/config/bao.hcl
ui        = false            # no web UI (default; when on, it is served on every listener)
log_level = "info"           # never "trace": it logs self-init requests with secrets

storage "raft" {
  path    = "/openbao/file"
  node_id = "bao-1"
}

listener "tcp" {
  address         = "0.0.0.0:8200"
  tls_cert_file   = "/openbao/tls/bao.crt"
  tls_key_file    = "/openbao/tls/bao.key"     # mode 0600, owner openbao
  tls_min_version = "tls13"
  # Safe defaults, written out so a reviewer can see them:
  disable_unauthed_generate_root_endpoints = true
  disable_unauthed_rekey_endpoints         = true
  telemetry {
    unauthenticated_metrics_access = false     # only valid inside this block
  }
  profiling {
    unauthenticated_pprof_access = false
  }
}

api_addr     = "https://bao.example.com:8200"
cluster_addr = "https://bao.example.com:8201"

# Auto-unseal is required for self-init. Use transit or a KMS.
seal "transit" {
  address     = "https://transit.example.com:8200"
  mount_path  = "transit/"
  key_name    = "autounseal"
  tls_ca_cert = "/openbao/tls/ca.crt"
}

# Audit devices live in the config file. API creation stays disabled
# (unsafe_allow_api_audit_creation defaults to false).
# Two devices: a request succeeds if at least one of them can log it.
audit "file" "file" {
  options {
    file_path = "/openbao/logs/audit.log"
    mode      = "0600"
  }
}
audit "file" "stdout" {
  options {
    file_path = "stdout"   # collected by the log pipeline, stored elsewhere
  }
}

# Self-initialization: runs once on first start with a root token
# that is never shown and is revoked at the end.
initialize "auth" {
  request "enable-userpass" {
    operation = "update"
    path      = "sys/auth/userpass"
    data      = { type = "userpass" }
  }
}
initialize "policies" {
  request "ops-admin" {
    operation = "update"
    path      = "sys/policies/acl/ops-admin"
    data = {
      policy = {
        eval_type   = "string"
        eval_source = "file"
        path        = "/openbao/init/ops-admin.hcl"
      }
    }
  }
}
initialize "mounts" {
  request "kv" {
    operation = "update"
    path      = "sys/mounts/secret"
    data      = { type = "kv", options = { version = "2" } }
  }
}
initialize "users" {
  request "alice" {
    operation = "update"
    path      = "auth/userpass/users/alice"
    data = {
      password = {
        eval_type       = "string"
        eval_source     = "env"
        env_var         = "INITIAL_ADMIN_PASSWORD"   # rotate after first login
        require_present = true
      }
      token_policies = ["ops-admin"]
      token_ttl      = "1h"
      token_max_ttl  = "8h"
    }
  }
}

The day-to-day admin policy. It has no path to change audit devices, generate a root token or touch raft:

hcl
# /openbao/init/ops-admin.hcl
path "sys/policies/acl/*" { capabilities = ["create", "read", "update", "delete", "list"] }
path "sys/auth"           { capabilities = ["read"] }
path "sys/auth/*"         { capabilities = ["create", "read", "update", "delete", "sudo"] }
path "sys/mounts"         { capabilities = ["read"] }
path "sys/mounts/*"       { capabilities = ["create", "read", "update", "delete"] }
path "secret/*"           { capabilities = ["create", "read", "update", "delete", "list"] }
path "sys/audit"          { capabilities = ["read", "sudo"] }

Run it with no swap. The image already runs as the openbao user and sets ulimit -c 0, so no core dumps.

bash
docker run -d --name bao --memory 512m --memory-swap 512m \
  --env-file /etc/openbao/init.env \
  -v /etc/openbao/bao.hcl:/openbao/config/bao.hcl:ro \
  -v /etc/openbao/init:/openbao/init:ro \
  -v /etc/openbao/tls:/openbao/tls:ro \
  -v bao-data:/openbao/file -v bao-logs:/openbao/logs \
  openbao/openbao:2.7.0 server

With systemd, the unit shipped with OpenBao sets MemorySwapMax=0. Check it with systemctl cat openbao.

Prove it

The test in secure-tests/openbao-hardening/ starts one server with this configuration (a static test seal stands in for transit). Inside the container:

bash
id; cat /sys/fs/cgroup/memory.swap.max
text
uid=100(openbao) gid=1000(openbao) groups=1000(openbao),1000(openbao)
0

The flag from the docs, on the same cgroup v2 host:

bash
docker run --rm --memory 256m --memory-swappiness=0 alpine:3.22 cat /sys/fs/cgroup/memory.swap.max
text
WARNING: Your kernel does not support memory swappiness capabilities or the cgroup is not mounted. Memory swappiness discarded.
268435456

The server initialized itself. Nobody received a root token:

bash
bao status
bao operator init
text
Seal Type                static
Initialized              true
Sealed                   false
Error initializing: Error making API request.
URL: PUT https://bao:8200/v1/sys/init
Code: 400. Errors:
* Vault is already initialized

The audit log records each self-init request, then the root token revoking itself:

bash
jq -r 'select(.type=="response") | [.request.id, .request.path, .auth.display_name] | @tsv' audit.log
text
initialize[0].auth.request[0].enable-userpass	sys/auth/userpass	root
initialize[1].policies.request[0].ops-admin	sys/policies/acl/ops-admin	root
initialize[2].mounts.request[0].kv	sys/mounts/secret	root
initialize[3].users.request[0].alice	auth/userpass/users/alice	root
self-init-revoke-root	auth/token/revoke-self	root

A named admin logs in and gets a one-hour token:

bash
bao login -method=userpass username=alice
bao token lookup -format=json | jq -c '{policies: .data.policies, ttl: .data.ttl, orphan: .data.orphan}'
bao audit list
text
{"policies":["default","ops-admin"],"ttl":3600,"orphan":true}
Path       Type    Description
----       ----    -----------
file/      file    n/a
stdout/    file    n/a

Adding an audit device over the API fails twice over. The admin policy has no path for it. A test user whose policy does allow sys/audit/* is refused by the server itself:

bash
bao audit enable -path=elsewhere file file_path=/tmp/copy.log   # as alice
bao audit enable -path=elsewhere file file_path=/tmp/copy.log   # as a user allowed sys/audit/*
text
Error enabling audit device: Error making API request.
URL: PUT https://bao:8200/v1/sys/audit/elsewhere
Code: 403. Errors:
* 1 error occurred:
	* permission denied
Error enabling audit device: Error making API request.
URL: PUT https://bao:8200/v1/sys/audit/elsewhere
Code: 400. Errors:
* cannot enable audit device via API; use declarative, config-based audit device management instead

Root generation needs a token, so nobody on the network can start or cancel an attempt:

bash
bao operator generate-root -status   # no token
text
Error getting root generation status: Error making API request.
URL: GET https://bao:8200/v1/sys/generate-root-token/attempt
Code: 403. Errors:
* permission denied

Secrets in the audit log are HMACs, and the file is readable by its owner only:

bash
bao kv put secret/app db_password=example-plaintext-4711
grep -c example-plaintext-4711 /openbao/logs/audit.log
jq -c 'select(.type=="request" and .request.path=="secret/data/app") | .request.data' audit.log
ls -l /openbao/logs/audit.log
text
0
{"data":{"db_password":"hmac-sha256:f38ac311c04f7f1b96bb4a1bcff51f97b546adf9350f0a3a70cdb8b8b2181dc1"},"options":{}}
-rw------- openbao openbao /openbao/logs/audit.log

Mistakes people make

Trusting --memory-swappiness=0

On cgroup v2 Docker drops the flag with a warning, and the container keeps its swap limit. Use --memory-swap equal to --memory, or MemorySwapMax=0 in systemd, and read memory.swap.max to check.

Keeping the root token "for emergencies"

A root token that never expires is the most valuable string you own. Use self-init, or revoke the initial root token as soon as named admins exist. For emergencies, generate a new one with a quorum of key holders (see break-glass accounts).

One audit device

A request succeeds if at least one audit device can log it. With one device, a full disk stops OpenBao. With two, one can fail and requests still get logged. The exception is a blocking failure, such as a network device that never answers: then requests hang until it can write again.

Top-level unauthenticated_metrics_access

It belongs inside the listener's telemetry block. At the top level of the listener, OpenBao logs unknown or unsupported field and ignores it. Read the startup log for warnings after every config change.

Copying disable_mlock

OpenBao has not used mlock since 2.0.0. The setting does nothing. Swap control is your job now.

log_level = "trace" left on

Trace logging during self-init prints request and response data, including secrets. Keep it off in production.

Turning on unsafe_allow_api_audit_creation and forgetting it

If you need it, turn it on, make the change, and turn it off again with a config reload. Better, declare the device in the config file.

Checklist

  • Confirm the OpenBao process runs as a non-root user.
  • Disable swap for the process and read memory.swap.max or MemorySwapMax to confirm.
  • Set tls_min_version = "tls13" on every listener.
  • Leave ui = false (the default); when on, the UI is served on every listener.
  • Declare at least two audit devices in the config file.
  • Leave unsafe_allow_api_audit_creation at false.
  • Use self-init, or revoke the initial root token once named admins exist.
  • Give admins named logins with token_ttl and token_max_ttl set.
  • Keep sys/audit/*, sys/generate-root* and sys/storage/raft/* out of day-to-day policies.
  • Check the startup log for unknown or unsupported field after each config change.
  • Ship the stdout audit stream to storage the OpenBao admins cannot delete.

A hardened OpenBao is mostly boring: no root token, two audit logs, no swap. Boring is the goal for the server that holds everyone else's keys.

H2-CIAE

Learn it on a live range

Secrets and custody, in Identity and Access Engineering: a real host in your browser, and every objective checked on the machine.

Start free

The Secure Way

More on secrets and pki

OpenBao, External Secrets, internal certificate authorities and keeping secrets off the command line.

All secrets and pki guides