Book a Security Review

Secrets and PKI

OpenBao policies as code with drift detection, the secure way

Every OpenBao has one policy that was "just for the weekend." It is still there, it has `path "*"`, and git has never heard of it.

The short answer

Store every ACL policy, including `default`, as one file per policy in git. A deploy job with a write token applies the directory and deletes policies that are not in it. A scheduled job with a list-and-read token compares live policies to git and fails on any missing, extra or changed policy.

Updated Houssam Hammoudi, CTOTested with OpenBao 2.7.0

On this page
  1. What goes wrong
  2. What the docs say
  3. The secure configuration
  4. Prove it
  5. Mistakes people make
  6. Checklist

What goes wrong

Policies are written once from a laptop, then edited in the UI during an incident. Nobody writes the change back. Six months later, the policy in git is fiction.

A hand edit is live the moment it is saved. Every existing token that carries the policy gets the new rights at once, with no login and no restart. The test below shows an app token reading a payroll secret seconds after someone widened its policy.

People keep the default policy out of git because it is "built in". It is attached to almost every token, and it can be edited.

What the docs say

Policies are deny by default, so an empty policy grants no permission in the system.

Source: OpenBao docs, Policies

The default policy is a built-in OpenBao policy that cannot be removed.

Source: OpenBao docs, Policies: default policy

It can be modified to suit your needs; OpenBao will never overwrite your modifications.

Source: OpenBao docs, Policies: default policy

The docs do not describe a way to detect changes against a source of truth. Terraform or OpenTofu with a vault_policy resource finds edits to policies it manages, but not a new policy created by hand. The check below finds both.

The secure configuration

One file per policy. The file name is the policy name.

text
policies/
  app-read.hcl
  ci-deploy.hcl
  default.hcl               # copied from a dev server of the same version, then reviewed
  policy-drift-reader.hcl
hcl
# policies/app-read.hcl: the app reads its own secrets, nothing else.
path "secret/data/app/*" {
  capabilities = ["read"]
}
hcl
# policies/policy-drift-reader.hcl: the drift check can list and read policies.
# It cannot write, delete or read any secret.
path "sys/policies/acl" {
  capabilities = ["list"]
}
path "sys/policies/acl/*" {
  capabilities = ["read"]
}

The sync script. apply runs in the deploy job after review. check runs on a schedule with the reader token.

bash
#!/bin/sh
# bao-policy-sync.sh check|apply DIR
# check: compare every ACL policy in OpenBao with DIR/<name>.hcl; exit 1 on drift.
# apply: write every DIR/<name>.hcl, delete policies that are not in DIR.
# Needs BAO_ADDR, BAO_CACERT and BAO_TOKEN. "root" cannot change and is skipped.
# "default" CAN be edited, so it is kept in DIR like any other policy.
set -eu
mode=$1 dir=$2
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
drift=0
ls "$dir"/*.hcl | sed 's#.*/##; s#\.hcl$##' | sort > "$tmp/want"
bao policy list | grep -v -x root | sort > "$tmp/have"

if [ "$mode" = apply ]; then
  while read -r p; do bao policy write "$p" "$dir/$p.hcl" >/dev/null; echo "wrote   $p"; done < "$tmp/want"
  comm -13 "$tmp/want" "$tmp/have" | grep -v -x default | while read -r p; do bao policy delete "$p" >/dev/null; echo "deleted $p"; done
  exit 0
fi

for p in $(comm -23 "$tmp/want" "$tmp/have"); do echo "MISSING  $p (in git, not in OpenBao)"; drift=1; done
for p in $(comm -13 "$tmp/want" "$tmp/have"); do echo "EXTRA    $p (in OpenBao, not in git)"; drift=1; done
for p in $(comm -12 "$tmp/want" "$tmp/have"); do
  bao policy read "$p" > "$tmp/live"
  if ! diff -u "$dir/$p.hcl" "$tmp/live" > "$tmp/diff"; then
    echo "CHANGED  $p"; sed '1,2d' "$tmp/diff"; drift=1
  fi
done
[ "$drift" = 0 ] && echo "no drift: $(wc -l < "$tmp/want") policies match git"
exit "$drift"

The two tokens:

bash
# Deploy job: a token from a CI login with a policy that may write
# sys/policies/acl/*. Never the root token.
# Drift job: read-only, renewable, not tied to a person.
bao token create -orphan -period=24h -policy=policy-drift-reader

Prove it

From secure-tests/openbao-policies-as-code-drift-detection/. Apply, then check with the reader token:

bash
./bao-policy-sync.sh apply policies
./bao-policy-sync.sh check policies; echo "exit $?"
text
wrote   app-read
wrote   ci-deploy
wrote   default
wrote   policy-drift-reader
no drift: 4 policies match git
exit 0

An app token with app-read cannot read another team's secret:

bash
bao kv get -mount=secret payroll/db   # with the app token
text
Code: 403. Errors:
* permission denied

Someone widens app-read by hand and adds a temp-debug policy with path "*". The same app token, with no new login, now reads the secret:

bash
bao kv get -field=password -mount=secret payroll/db
text
example-payroll-only

The drift check fails and shows what changed:

bash
./bao-policy-sync.sh check policies; echo "exit $?"
text
EXTRA    temp-debug (in OpenBao, not in git)
CHANGED  app-read
@@ -1,4 +1,6 @@
-# app-read: the app reads its own secrets, nothing else.
 path "secret/data/app/*" {
   capabilities = ["read"]
+}
+path "secret/data/*" {
+  capabilities = ["read", "list"]
 }
exit 1

The drift job's token cannot "fix" anything:

bash
bao policy delete temp-debug   # with the reader token
text
Code: 403. Errors:
* permission denied

The deploy job restores git as the truth:

bash
./bao-policy-sync.sh apply policies
./bao-policy-sync.sh check policies; echo "exit $?"
text
wrote   app-read
wrote   ci-deploy
wrote   default
wrote   policy-drift-reader
deleted temp-debug
no drift: 4 policies match git
exit 0

Mistakes people make

Leaving default out of git

It is attached to most tokens and it can be edited. A line added to default reaches almost everyone. Keep it in git and in the check.

Running the check with the admin token

A drift job with write rights is one bug away from rewriting policies. Give it list and read on sys/policies/acl and nothing else.

Believing a hand edit waits for the next login

Policies are evaluated on each request. The edit is live for every token that carries the policy, immediately.

Checking only the policies you manage

A tool that tracks known policies misses the one created by hand. Compare the full list, and fail on extras.

Reverting without asking who

Before apply removes a hand edit, find it in the audit log (sys/policies/acl/<name> with its display_name). A hand edit during an incident may need to become a reviewed change, or it may be the first sign of an intruder.

Checklist

  • Store every ACL policy, including default, as <name>.hcl in one directory.
  • Apply the directory only from a reviewed CI job, never from a laptop.
  • Delete live policies that are not in git during apply.
  • Run the drift check on a schedule with a list-and-read token.
  • Fail the job and alert on any MISSING, EXTRA or CHANGED line.
  • Look up each drift in the audit log before reverting it.
  • Keep sys/policies/acl/* write access out of human day-to-day policies.

Git is only the source of truth if something checks it every day; otherwise it is a nicely formatted rumor.

H2-CIAE

Learn it on a live range

Authorization as code, in Identity and Access Engineering: a real host in your browser, and every objective checked on the machine.

Start free

The Secure Way

More on secrets and pki

OpenBao, External Secrets, internal certificate authorities and keeping secrets off the command line.

All secrets and pki guides