Digest pinning in GitOps, the secure way

#What goes wrong

GitOps promises that Git describes what runs. With image tags, it does not. A tag is resolved by each node when it pulls, so:

• A re-pushed tag reaches production with no commit, no review and no rollback point.
• Two nodes can run different code under the same tag, depending on when they pulled.
• A scan or signature approved one digest; the tag may now point to another.

Pinning the main image is not enough. The init container, the sidecar that a chart adds, and the job image in a CronJob are the ones people forget.

#What the docs say

Digests are a unique identifier for a specific version of an image. Digests are hashes of the image's content, and are immutable. Tags can be moved to point to different images, but digests are fixed.

Source: Kubernetes docs, Images

Specifying an image by digest pins the code that you run so that a change at the registry cannot lead to that mix of versions.

Source: Kubernetes docs, Images

Image name with tag and digest. Only the digest will be used for pulling.

Source: Kubernetes docs, Images

The digest for an image may be set by specifying digest and the name of the container image.

Source: kustomize reference, images

The docs explain how to pin one image. They do not tell you how to prove that every image in every overlay is pinned, which is what stops the forgotten sidecar.

#The secure configuration

Manifests use a placeholder name; the kustomization sets the full reference.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  replicas: 2
  selector:
    matchLabels: { app: app }
  template:
    metadata:
      labels: { app: app }
    spec:
      containers:
        - name: app
          image: app            # a placeholder name; kustomization.yaml sets the real reference

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
images:
  - name: app
    newName: registry.example.com/team-a/app
    newTag: 1.4.2                 # for people: which release this is
    digest: sha256:<64 hex>       # for the kubelet: exactly what runs

Set it with a command, not by hand, from the digest your build recorded:

DIGEST=$(cat out/app.digest)     # kaniko --digest-file, or: crane digest <image>:<tag>
kustomize edit set image "app=registry.example.com/team-a/app:1.4.2@${DIGEST}"

Then gate every change in CI on the rendered output, not on the source files:

#!/bin/sh
# check-digests.sh RENDERED.yaml
# Fail if any rendered container image is not pinned by sha256 digest.
# Matches "image:" and "- image:" (kustomize renders the list form).
bad=$(grep -E '^[[:space:]]*(-[[:space:]]+)?image:' "$1" | grep -v -E '@sha256:[0-9a-f]{64}' || true)
[ "$(grep -c -E '^[[:space:]]*(-[[:space:]]+)?image:' "$1")" -gt 0 ] || { echo "no images found: check the pattern"; exit 1; }
if [ -n "$bad" ]; then echo "NOT PINNED:"; echo "$bad"; exit 1; fi
echo "all images pinned by digest"

for overlay in overlays/*/; do
  kustomize build "$overlay" > rendered.yaml
  sh check-digests.sh rendered.yaml
done

The same check works for Helm (helm template ... > rendered.yaml). The second grep -c line is there on purpose: a gate whose pattern matches nothing passes everything.

#Prove it

A digest keeps meaning the same image after the tag moves:

crane digest registry:5000/team-a/app:1.4.2          # what CI resolved and pinned
# someone re-pushes 1.4.2 with different content
crane digest registry:5000/team-a/app:1.4.2
crane export registry:5000/team-a/app@sha256:12be14a1... - | tar -xO app/version

sha256:12be14a1a6cd5e3de4b33768cc0b54d937e1ab073ea15743434454f4ba086c21
sha256:e67fd6fc9112bce07a2931b75ee9f068627e80b36552bf13fd3c90111f3a7a79
build 1

Setting and rendering the pinned reference:

kustomize edit set image app=registry.example.com/team-a/app:1.4.2@sha256:12be14a1...
cat kustomization.yaml
kustomize build . | grep image:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
images:
- digest: sha256:12be14a1a6cd5e3de4b33768cc0b54d937e1ab073ea15743434454f4ba086c21
  name: app
  newName: registry.example.com/team-a/app
  newTag: 1.4.2
      - image: registry.example.com/team-a/app:1.4.2@sha256:12be14a1a6cd5e3de4b33768cc0b54d937e1ab073ea15743434454f4ba086c21

The gate passes, then a pull request adds a sidecar by tag and the gate fails:

sh check-digests.sh rendered.yaml
# a pull request adds a sidecar by tag
kustomize build . > rendered.yaml && sh check-digests.sh rendered.yaml

all images pinned by digest
NOT PINNED:
      - image: docker.io/example/log-shipper:2.1
exit code 1

A first version of this gate matched only lines starting with image:. It printed "all images pinned by digest" for the sidecar too, because kustomize writes - image:. That is why the gate now fails when it finds no images at all.

#Mistakes people make

#Pinning in the source, not the render

Charts and remote bases add containers you never see in your own files. Check the rendered output of every overlay.

#Digest without a tag

app@sha256:... works, but nobody can tell which release it is during an incident. Keep name:tag@sha256:digest; the kubelet uses only the digest.

#Copying digests by hand

A pasted digest from the wrong architecture or the wrong build passes review because nobody reads 64 hex characters. Write digests from the build's --digest-file or crane digest, in automation.

#Multi-arch confusion

For a multi-arch image, pin the index digest (what crane digest returns for the tag) so each node pulls its own platform. Pinning one platform's manifest digest breaks nodes of the other architecture.

#Pinning and then never updating

A pinned digest does not get security fixes. Pair pinning with an automated pull request that proposes new digests after a scan.

#Checklist

• Every image in Git is written as name:tag@sha256:digest.
• Digests are written by automation from the build or crane digest, never pasted.
• CI renders every overlay (kustomize or Helm) and fails on any image without @sha256:.
• The gate fails if it finds no images, so a broken pattern cannot pass.
• Init containers, sidecars, Jobs and CronJobs are covered by the same check.
• Multi-arch images are pinned by index digest.
• New digests arrive by pull request after a scan.