Promotion workflows where CI can only write digests, the secure way

#What goes wrong

In most GitOps setups, CI builds an image and then pushes a commit to the deployment repository to roll it out. To do that it gets a deploy key or token with write access to the whole repository, often to the main branch directly.

That token can then change anything Argo CD or Flux will apply: scale a service to zero, remove a NetworkPolicy, add a ClusterRoleBinding, or deploy an image nobody tested. A compromised pipeline, or a malicious pull request that runs in it, inherits all of that.

Promotion needs one fact to move: which digest runs in the next environment.

#What the docs say

The git write-back method uses Git to permanently store its parameter overrides along with the Application's resource manifests.

Source: Argo CD Image Updater docs, Update methods

Please note that all update strategies except digest assume tags to be immutable and that new images will be pushed with a new, unique tag.

Source: Argo CD Image Updater docs, Update strategies

Kargo is an unopinionated continuous promotion platform that helps developers orchestrate the movement of new code and configuration through the various stages of their applications' lifecycles using GitOps principles.

Source: Kargo docs, Core concepts

Protected branches enforce restrictions such as force pushing or merging unless a given number of approvals are obtained on a pull request.

Source: Forgejo docs, Protected branches

Tools like Image Updater and Kargo automate the write. None of them limits what else the writing identity could change in the same repository; that is the job of branch protection and a check you own. Forgejo's branch protection also has push allowlists and protected file patterns in its settings page, though its user docs do not describe them.

#The secure configuration

#1. Repository layout

deploy/
  base/                         # people only, by reviewed pull request
  overlays/
    staging/kustomization.yaml  # CI may change digest and newTag lines
    prod/kustomization.yaml     # CI may change digest and newTag lines
  .forgejo/workflows/           # people only; protected by code owners

#2. The promote step (CI, bot identity)

# DIGEST comes from the build (kaniko --digest-file) for staging, and from
# the staging overlay for prod. Nothing else is edited.
cd overlays/prod
kustomize edit set image "app=registry.example.com/team-a/app:${VERSION}@${DIGEST}"
git commit -am "promote app ${VERSION} to prod"
git push origin HEAD:refs/heads/promote/prod-${VERSION}   # a branch, not main

The bot pushes a branch and opens a pull request; it cannot push to main.

#3. The required check

#!/bin/sh
# check-promotion.sh BASE HEAD
# Accept a promotion commit only if:
#   1. it changes nothing but overlays/prod/kustomization.yaml,
#   2. every changed line is a "digest:" or "newTag:" line,
#   3. the new prod digest is a digest that staging already runs.
set -eu
BASE=$1; HEAD=$2
files=$(git diff --name-only "$BASE" "$HEAD")
if [ "$files" != "overlays/prod/kustomization.yaml" ]; then
  echo "REJECT: promotion may only change overlays/prod/kustomization.yaml, changed:"; echo "$files"; exit 1
fi
other=$(git diff -U0 "$BASE" "$HEAD" -- overlays/prod/kustomization.yaml \
  | grep -E '^[+-]' | grep -v -E '^(\+\+\+|---) ' \
  | grep -v -E '^[+-][[:space:]]*(- )?(digest: sha256:[0-9a-f]{64}|newTag: [0-9A-Za-z._-]+)$' || true)
if [ -n "$other" ]; then echo "REJECT: lines other than digest/newTag changed:"; echo "$other"; exit 1; fi
new=$(git show "$HEAD:overlays/prod/kustomization.yaml" | sed -n -E 's/^[[:space:]]*(- )?digest: (sha256:[0-9a-f]{64})$/\2/p')
staged=$(git show "$HEAD:overlays/staging/kustomization.yaml" | sed -n -E 's/^[[:space:]]*(- )?digest: (sha256:[0-9a-f]{64})$/\2/p')
if [ "$new" != "$staged" ]; then echo "REJECT: prod digest $new is not the digest staging runs ($staged)"; exit 1; fi
echo "OK: digest-only promotion of $new"

This version handles one image per overlay; extend the last step to compare each image by name if an overlay has several.

#4. Branch protection on main

• Pushes to main only through pull requests.
• The promotion check is a required status check.
• Required approvals apply to the bot's pull requests too. A person approves promotions (the check makes that a one-minute review), or a separate approver identity approves only pull requests where the check passed. Every other pull request needs a code owner's approval.
• base/, .forgejo/ and the check script itself are protected files that only people can change.
• Argo CD syncs main only (see the Argo CD hardening page for project limits).

#Prove it

A promotion that only moves the digest (and the tag next to it) passes:

kustomize edit set image app=registry.example.com/team-a/app:1.4.2@<staging digest>   # in overlays/prod
git diff -U0 HEAD~1 -- overlays/prod
check-promotion.sh BASE HEAD

-- digest: sha256:a7ba68946b509f4f2c214955090896c7ee77f3cbe22cf720e3fa21cb6e2c2613
+- digest: sha256:a73e0339b4d8713fd41910c280275ba340c19fc13be7f9af84bc017aa70bbc6b
-  newTag: 1.4.1
+  newTag: 1.4.2
OK: digest-only promotion of sha256:a73e0339b4d8713fd41910c280275ba340c19fc13be7f9af84bc017aa70bbc6b

The same promotion, plus a replica change in the same file:

REJECT: lines other than digest/newTag changed:
+replicas:
+- count: 0
+  name: app
exit code 1

A digest that staging never ran:

REJECT: prod digest sha256:43100289640d4b79bc070c4c60d160fb0e0f0385257523d9017ae2ff0af82433 is not the digest staging runs (sha256:a73e0339b4d8713fd41910c280275ba340c19fc13be7f9af84bc017aa70bbc6b)
exit code 1

A promotion that also edits the base:

REJECT: promotion may only change overlays/prod/kustomization.yaml, changed:
base/deployment.yaml
overlays/prod/kustomization.yaml
exit code 1

#Mistakes people make

#A deploy key with write access to main

Deploy keys usually cannot be limited to paths. If CI pushes straight to main, no check runs before Argo CD applies the change. Push to a branch and let the required check decide.

#Checking the diff in the same pipeline that made it

If the job that writes the promotion also runs the check, a compromised job skips it. The check runs as a required status on the pull request, from the protected workflow on main.

#Promoting tags

A tag-only promotion (newTag: 1.4.2) moves whatever the tag points to at sync time. Promote digests; keep the tag for people.

#Skipping the previous stage

Without rule 3, CI can send an untested digest straight to production. Production takes only what staging already runs.

#Letting the bot edit its own rules

If the bot can change the workflow or the check script, rules 1 to 3 are suggestions. Protect them as files only people can change.

#Checklist

• CI uses a bot identity that can push branches but not main.
• Promotion commits change only one overlay's kustomization.yaml.
• Only digest: and newTag: lines change in a promotion.
• A production digest must already run in staging.
• The promotion check is a required status check, run from the protected branch.
• The workflow directory, the check and base/ are protected and owned by people.
• Promotion pull requests are approved by a person or by an approver identity that requires the check; all others need a code owner.
• Argo CD syncs only the protected branch.