Self-managed Pulumi state, the secure way

#What goes wrong

Pulumi records every resource's inputs and outputs in a state file, called a checkpoint. With a self-managed backend (Pulumi calls it DIY), that file lives in your bucket or on disk, and nobody else encrypts it for you.

Pulumi encrypts values it knows are secret: config set with --secret, and anything computed from them. Everything else is written as plain JSON. That includes values that are secret in practice but were never marked: a file read with readFile, a user data blob, a connection string built from a non-secret variable, an API token returned by a provider that does not mark it.

The state is also copied. Each update writes the current checkpoint, a backup, and a history entry. A leaked value lives in all of them, so fixing the program does not clean the bucket.

The secrets that are encrypted are only as safe as their key. The passphrase provider derives the key from PULUMI_CONFIG_PASSPHRASE, which usually sits in a CI variable next to the bucket credentials. Whoever has both reads every secret.

#What the docs say

Most of these values are plain-text strings, such as configuration settings, computed URLs, or resource identifiers.

Source: Pulumi docs, Secrets

For DIY backends, state management—including backup, sharing, and team access synchronization—is custom and implemented manually.

Source: Pulumi docs, Using a DIY backend

DIY backends also maintain checkpoint history (in the .pulumi/history/ directory), but because they are fundamentally limited by the non-transactional protocols of blob storage, they cannot transparently recover from certain kinds of partial failures

Source: Pulumi docs, State and backends

The docs explain which values are encrypted. They do not warn that a value you forgot to mark is copied into history and backups on every update, or that a passphrase stored next to the bucket credentials protects nothing against whoever holds the CI variables.

#The secure configuration

1. Mark everything sensitive as secret in the program. In Pulumi YAML, fn::secret wraps a value; secret config stays secret through interpolation:

# Pulumi.yaml
name: state-demo
runtime: yaml
config:
  dbPassword:
    type: string
    secret: true                        # refuses a plain `pulumi config set`
variables:
  machineConfig:
    fn::readFile: ./machine-config.yaml # a Talos config: holds CA keys
outputs:
  dsn: postgres://app:${dbPassword}@db.internal.example.net/app   # secret: built from a secret
  userDataPlain: ${machineConfig}       # WRONG: written to state in plain text
  userDataSecret:
    fn::secret: ${machineConfig}        # right: encrypted in state

In TypeScript, Python and Go, use pulumi.secret(...) for values and additionalSecretOutputs for provider outputs that should be secret.

2. Use a key service, not a passphrase, for the secrets provider. With an OpenBao or Vault transit key (the hashivault provider speaks the transit API), the pipeline authenticates with a short-lived identity and never holds the key:

export VAULT_SERVER_URL=https://bao.example.net:8200
export VAULT_SERVER_TOKEN="$(cat /run/secrets/bao-token)"   # short-lived, from the CI identity
pulumi stack init prod --secrets-provider="hashivault://pulumi-prod"

# Existing stacks: re-encrypt config and state under the new provider.
pulumi stack change-secrets-provider "hashivault://pulumi-prod"

3. Keep state in a private, versioned bucket with scoped keys. Pulumi's S3 backend works with any S3-compatible store, DigitalOcean Spaces included. It needs exactly four actions: s3:ListBucket, s3:GetObject, s3:PutObject and s3:DeleteObject.

# Credentials limited to this one bucket, from the CI secret store:
export AWS_ACCESS_KEY_ID="$(cat /run/secrets/state-key-id)"
export AWS_SECRET_ACCESS_KEY="$(cat /run/secrets/state-key)"
pulumi login 's3://iac-state?endpoint=s3.example.net&s3ForcePathStyle=true&region=us-east-1'

On the bucket itself:

• Private: no public ACL, no public listing, no CDN in front of it.
• Versioning on, so a corrupted or deleted checkpoint can be restored; DIY backends cannot repair a partial write themselves.
• Encryption at rest on the provider side, as a second layer.
• One key pair per pipeline, scoped to this bucket, and a separate read-only key for anyone who only needs pulumi stack output.
• Access logs sent somewhere the pipeline's key cannot delete.

4. Check the state for canaries. Put a canary string in a test input that should be secret, and fail the pipeline if it ever appears in the bucket.

#Prove it

Real output from secure-tests/self-managed-pulumi-state/, run in the pulumi/pulumi-base:3.264.0 container with a local file:// backend and a machine-config.yaml that contains the canary CANARY-KEY-0001.

1. Setup and one update:

pulumi login file:///tmp/state
pulumi stack init dev --secrets-provider passphrase
pulumi config set --secret dbPassword "Hunter2-Canary"
pulumi up --yes --skip-preview   # shown from "Outputs:" on

Outputs:
    dsn           : [secret]
    userDataPlain : (yaml) {
        machine: {
            ca: {
                key: "CANARY-KEY-0001"
            }
        }
    }

    userDataSecret: [secret]

Resources:
    + 1 created

Duration: 1s

The unmarked output is printed in full, so it is also in your CI log. The marked one shows as [secret].

2. The stack config holds only ciphertext:

cat Pulumi.dev.yaml

encryptionsalt: v1:fi8fdG9Fmqs=:v1:RCSwfY24eL0Y7S1E:vecShc3AlPVhrktChWxBoXGhUccHKA==
config:
  state-demo:dbPassword:
    secure: v1:bRgNaYdNE3akQOLC:el2GbmpddQ8ud8fzb52UKfkRYLjAWekfy3hxIoIt

3. Where the canaries ended up:

grep -rl 'Hunter2-Canary' /tmp/state | wc -l
grep -rl 'CANARY-KEY-0001' /tmp/state

0
/tmp/state/.pulumi/history/state-demo/dev/dev-<timestamp>.checkpoint.json
/tmp/state/.pulumi/backups/state-demo/dev/dev.<timestamp>.json
/tmp/state/.pulumi/stacks/state-demo/dev.json

Timestamps in the file names are replaced with <timestamp>. The secret config value is in no file. The unmarked readFile value is in the current checkpoint, its backup and the history entry: fixing the program later leaves these copies in the bucket.

#Mistakes people make

#Assuming "self-managed" means "encrypted"

A DIY backend stores what Pulumi hands it. Only values marked secret are encrypted; the rest is readable by anyone who can read the bucket.

#Reading config files straight into resources

readFile on a Talos machine config, a kubeconfig or a private key produces a plain string. Wrap it in fn::secret or pulumi.secret() before it reaches a resource or an output.

#Passphrase next to the credentials

PULUMI_CONFIG_PASSPHRASE in the same CI secret store as the bucket keys gives one compromised job both halves. Use a transit or KMS key that the job can use but not export.

#Deleting the checkpoint to "clean up"

The value is also in .pulumi/backups/ and .pulumi/history/, and in the bucket's old versions. Rotate the leaked secret instead; then remove old objects if your retention rules allow it.

#One bucket key for everything

A key that can write state can also delete it and rewrite resource IDs. Give each pipeline its own scoped key, and give readers a read-only one.

#Checklist

• Every sensitive input and output is marked secret (--secret, fn::secret, pulumi.secret, additionalSecretOutputs).
• No readFile of keys or machine configs reaches a resource or output unwrapped.
• The secrets provider is a KMS or transit key, not a passphrase stored with the bucket keys.
• The state bucket is private, versioned and encrypted at rest.
• Pipeline keys are scoped to the state bucket and to the four S3 actions Pulumi needs.
• Readers use a separate read-only key.
• A canary check fails the pipeline if a secret value appears in any state object.
• Leaked values were rotated, not only deleted.