Rolling out the Sigstore policy-controller, the secure way

#What goes wrong

A cluster without an admission check runs any image that someone can name. A leaked deploy token, a typo in a registry host, or a tag that moved at the registry is enough to run code you never built.

The Sigstore policy-controller closes that gap. It is an admission webhook that checks each image against a signature policy before the pod is created. It fails in two ways:

• It is installed in enforce mode on day one. The first unsigned image it meets is a system add-on, and the cluster stops healing itself.
• It is installed and never enforced. It opts in no namespaces, or every policy stays in warn mode forever, so it checks nothing.

The fix is an order of operations, not a clever policy.

#What the docs say

The policy-controller admission controller will by default only validate resources in namespaces that have chosen to opt-in.

Source: Sigstore docs, Policy Controller overview

By default, any image that does not match a policy is rejected whenever no-match-policy is not configured in the ConfigMap.

Source: Sigstore docs, Policy Controller overview

When set to warn, it will not block the admission, but instead will allow it through and emit a warning.

Source: Sigstore docs, Policy Controller overview

The webhook validates that container images have been signed, and resolves image tags to digests to ensure the image being run is not different from when it was admitted.

Source: Sigstore docs, Policy Controller installation

Two things the docs leave to you. First, the default for unmatched images is reject, so an opted-in namespace with one forgotten sidecar image breaks as soon as you label it. Second, the overview says "When ctlog key is not specified, the public Rekor instance will be used", but in the v0.15.1 source (pkg/webhook/validator.go) a key authority with no ctlog block skips the transparency-log check. If you sign offline without a log, test one real image in warn mode before you trust either reading.

#The secure configuration

Before anything else: every image that will run in an opted-in namespace is signed, including mirrored third-party images. See the pages on offline signing and on mirroring base images.

Install policy-controller into its own namespace, cosign-system as in the upstream examples, and leave that namespace and kube-system unlabeled. Pin the version by digest. At the time of writing the Sigstore Helm chart (0.10.8) still ships v0.13.1, while the current release is v0.15.1, so set the image yourself:

# v0.15.1: crane digest ghcr.io/sigstore/policy-controller/policy-controller:v0.15.1
helm install policy-controller sigstore/policy-controller -n cosign-system --create-namespace \
  --version 0.10.8 \
  --set webhook.image.version=sha256:0a5806a61e0482e56153ae2ce6f6707846ffe0f9e3cd4be58655fbf88b874a3c \
  --set webhook.registryCaBundle.name=registry-ca   # a ConfigMap with your private registry's CA, if any

Then apply the policy in stages. Stage 2 below is the file as you first apply it.

# Stage 1: opt one namespace in. Namespaces without this label are not checked.
apiVersion: v1
kind: Namespace
metadata:
  name: team-a
  labels:
    policy.sigstore.dev/include: "true"
---
# Images from your own registry must carry a signature from your release key.
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: release-key
spec:
  mode: warn                    # stage 2: warn; stage 3: change to enforce
  images:
    - glob: "registry.example.com/apps/**"   # your images only; not the mirror path
  authorities:
    - name: release-key
      key:
        hashAlgorithm: sha256
        data: |
          -----BEGIN PUBLIC KEY-----
          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEexampleexampleexampleexampleexam
          pleexampleexampleexampleexampleexampleexampleexampleexampleexampleA==
          -----END PUBLIC KEY-----
---
# Mirrored third-party base images: signed by the mirror job's key after the scan.
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: mirror-key
spec:
  mode: warn
  images:
    - glob: "registry.example.com/mirror/**"
  authorities:
    - name: mirror-key
      key:
        secretRef:
          name: mirror-cosign-pub   # Secret in the policy-controller namespace, key "cosign.pub"
---
# What happens to images that match no policy at all (for example docker.io/*).
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-policy-controller
  namespace: cosign-system
data:
  no-match-policy: warn         # stage 2: warn; stage 3: deny

Keep the two globs apart. Every policy whose glob matches an image must pass, so registry.example.com/** would also cover registry.example.com/mirror/ and demand the release key on mirrored images too.

Sign in the format the policy can read. With a public key in the policy, policy-controller v0.15.1 checks only the classic signature format (the sha256-<digest>.sig tag). cosign v3 writes the new bundle format by default, including through a signing config, and a policy set to signatureFormat: bundle accepts that format only from keyless authorities. Sign images for this policy like this:

cosign sign --yes --key cosign.key \
  --use-signing-config=false --tlog-upload=false --new-bundle-format=false \
  registry.example.com/apps/app@sha256:<digest>

Only public keys go into the cluster. The docs say a secretRef names a secret "in the same namespace where policy-controller is installed", so the mirror key secret goes there, not in team-a.

The rollout, one namespace at a time:

1. Label the namespace. Policies in warn, no-match-policy: warn.
2. Redeploy everything in the namespace and collect every warning.
3. Sign, mirror or remove each image that warned. Repeat until a full redeploy prints no warning.
4. Change mode: warn to mode: enforce in each policy.
5. Change no-match-policy to deny. This affects every opted-in namespace, so do it once all of them are clean.

#Prove it

The manifests pass kubeconform against the Kubernetes 1.34 schemas and the ClusterImagePolicy v1beta1 schema (4 resources, all valid; a misspelled imagez is reported as "additional properties 'imagez' not allowed").

Then on a lab cluster: policy-controller v0.15.1 installed as above, a private registry kind-registry:5000 with its own CA, the release key policy on kind-registry:5000/team-a/** and the mirror key policy on kind-registry:5000/mirror/**, and team-a labelled.

1. Which signatures pass. The same key, three ways of signing:

cosign v3 default (new bundle format):
  Warning: ... signature key validation failed for authority release-key for kind-registry:5000/team-a/app@sha256:66a6306d...
policy switched to signatureFormat: bundle:
  Warning: ... constructing checkOpts for release-key: when using the new bundle format, the authority must be keyless
cosign sign --use-signing-config=false --tlog-upload=false --new-bundle-format=false:
  pod/signed created   (no warning)
$ curl -s https://kind-registry:5000/v2/team-a/app/tags/list
{"name":"team-a/app","tags":["sha256-66a6306d....sig","1.0","sha256-66a6306d..."]}

The .sig tag is the format the key policy reads. Signing through a signing config (as on the offline signing page) gave the new format and the same failure, and that mode refuses --new-bundle-format=false.

2. Stage 2, warn. With the page's first glob, ** over the whole registry, a mirrored image was checked against both policies:

$ kubectl -n team-a run mirror --image=kind-registry:5000/mirror/alpine:3.22 ...
Warning: failed policy: mirror-key: spec.containers[0].image
Warning: failed policy: release-key: spec.containers[0].image
pod/mirror created
$ kubectl -n team-a run nomatch --image=docker.io/library/busybox:1.37 ...
Warning: no matching policies: spec.containers[0].image
pod/nomatch created
$ kubectl -n team-a get pod signed -o jsonpath='{.spec.containers[0].image}'
kind-registry:5000/team-a/app:1.0@sha256:66a6306db78bf2dbf3487f293aa8d6990d8e506fdffab9cc43fe422becf886e4

The webhook pinned the tag to a digest in the stored spec.

3. Stage 3, enforce and deny, with separate globs:

signed    kind-registry:5000/team-a/app:1.0          pod/signed created
unsigned  kind-registry:5000/team-a/unsigned:1.0     Error from server (BadRequest): admission webhook "policy.sigstore.dev" denied the request: validation failed: failed policy: release-key: ...
mirror    kind-registry:5000/mirror/alpine:3.22      pod/mirror created
nomatch   docker.io/library/busybox:1.37             Error from server (BadRequest): admission webhook "policy.sigstore.dev" denied the request: validation failed: no matching policies: ...

A namespace without the label was not checked (pod/nolabel created for the same unmatched image).

Before you label a namespace, check a signed image with the same key the policy uses:

cosign verify --key cosign.pub --insecure-ignore-tlog=true \
  registry.example.com/apps/app@sha256:<digest>

What you should see: a Verification for ... line. Note that cosign verify accepts both formats, so it can pass while the admission check fails: look for the .sig tag as well.

#Mistakes people make

#Enforce on the first day

The default for unmatched images is reject. Labeling a namespace with enforce policies and no no-match-policy turns every image you forgot into a failed rollout. Warn first; enforce when warnings stop.

#Labeling kube-system

System add-ons come from many registries and are rarely signed with your key. If they must be covered, write a separate policy for their upstream signatures, and test node replacement and upgrades while it is in warn mode.

#Writing the private key into a Secret

The admission check needs only the public key. A private signing key in the cluster means anyone who can read that Secret can sign anything. Keep the private key where no cluster can read it.

#Glob patterns that match nothing

registry.example.com/* does not match registry.example.com/team-a/app; ** does. A policy that matches nothing falls through to no-match-policy, which you may still have on warn.

#Warn mode forever

Warn mode checks and reports but stops nothing. Put the switch to enforce on a date, with an owner.

#Checklist

• Every image running in the target namespace is signed or mirrored and signed.
• policy-controller is installed at a pinned version in its own namespace.
• Only public keys are in the cluster; secretRef secrets are in the policy-controller namespace.
• One namespace is labeled policy.sigstore.dev/include: "true" at a time.
• Policies start in mode: warn and no-match-policy starts at warn.
• A full redeploy of the namespace prints no warnings.
• Policies are switched to mode: enforce.
• no-match-policy is set to deny once every opted-in namespace is clean.
• An unsigned probe image is rejected in each enforced namespace.
• Transparency-log behavior is tested with a real signed image before enforcing.