Registry tags that never move, the secure way

#What goes wrong

An image tag is a pointer from a name to a manifest digest. On most registries, anyone with push rights can point it somewhere else at any time. Nothing in the pull tells you it moved.

That breaks three things:

• Rollbacks. "Roll back to 1.0.0" may deploy code that never ran before.
• Audits. The scan, the signature and the approval were for one digest; the tag now names another.
• Trust in CI. A compromised pipeline, or a careless one, can replace a released version without creating a new one.

Digest pinning fixes deployments. Immutable tags fix the registry itself, so a release name keeps meaning one thing for everyone.

#What the docs say

Tag: a custom, human-readable pointer to a manifest. A manifest digest may have zero, one, or many tags referencing it.

Source: OCI Distribution Specification

This is due to Distribution/Distribution upstream which does not enforce the mapping between an image tag and the image digest.

Source: Harbor docs, Create tag immutability rules

Tag immutability guarantees that an immutable tagged artifact cannot be deleted, and also cannot be altered in any way such as through re-pushing, re-tagging, or replication from another target registry.

Source: Harbor docs, Create tag immutability rules

While zot does not have an explicit configuration flag to make image tags immutable, the same effect can be achieved with authorization as follows.

Source: zot docs, Immutable tags

The OCI specification defines what a tag is but not whether it may move; that is left to each registry. CNCF Distribution (registry:2) has no immutability setting at all. On AWS ECR the setting is per repository (imageTagMutability: IMMUTABLE), and its page says "Amazon ECR returns an ImageTagAlreadyExistsException if you attempt to push an image with an existing tag."

#The secure configuration

#zot: CI can create, only a release admin can update

{
  "distSpecVersion": "1.1.1",
  "storage": { "rootDirectory": "/var/lib/registry" },
  "http": {
    "address": "0.0.0.0",
    "port": "5000",
    "auth": { "htpasswd": { "path": "/etc/zot/htpasswd" } },
    "accessControl": {
      "repositories": {
        "**": {
          "policies": [
            { "users": ["release-admin"], "actions": ["read", "create", "update", "delete"] }
          ],
          "defaultPolicy": ["read", "create"]
        }
      }
    }
  },
  "log": { "level": "warn" }
}

defaultPolicy applies to every authenticated user without a matching policy, including the CI identity: it can push new tags and pull, but not overwrite (update) or delete. The release-admin account is for emergencies, used by a person, and its use is logged and reviewed. In production, zot also needs TLS; the test below used plain HTTP on a private Docker network.

#Harbor

In the project, add a tag immutability rule that matches every repository (**) and every tag (**). Harbor then refuses re-pushes, re-tags and deletes of matching tags.

#AWS ECR

aws ecr put-image-tag-mutability \
  --repository-name team-a/app \
  --image-tag-mutability IMMUTABLE

#Everywhere: tags that are never reused

Immutability only helps if your tags are unique per build. Tag with the version and the commit, never with latest for anything deployed:

TAG="1.4.2-$(git rev-parse --short=12 HEAD)"

If a workflow truly needs a moving tag (for example staging), keep it in a separate repository path with its own mutable policy, and never deploy from it by tag.

#Prove it

On registry:2 with its default configuration, the same tag is pushed twice with different content:

crane append -f build1.tar -t registry:5000/team-a/app:1.0.0 ; crane digest registry:5000/team-a/app:1.0.0
crane append -f build2.tar -t registry:5000/team-a/app:1.0.0 ; crane digest registry:5000/team-a/app:1.0.0

sha256:9ae6ec75e4976ab1fa6bccbcde092ee9a3f5d23b8573522ac5b2d2c8dd0644bf
sha256:f7af38aa9da7f395fb81e6a77a0bce74aa1b98fedd3c21fa8bbb4b56228f5a99

The tag silently moved. The same steps on zot with the policy above, logged in as the CI user:

crane append -f build1.tar -t registry:5000/team-a/app:1.0.0
crane append -f build2.tar -t registry:5000/team-a/app:1.0.0      # move the tag
crane digest registry:5000/team-a/app:1.0.0

registry:5000/team-a/app@sha256:9ae6ec75e4976ab1fa6bccbcde092ee9a3f5d23b8573522ac5b2d2c8dd0644bf
Error: pushing image registry:5000/team-a/app:1.0.0: PUT http://registry:5000/v2/team-a/app/manifests/1.0.0: DENIED: requested access to the resource is denied
sha256:9ae6ec75e4976ab1fa6bccbcde092ee9a3f5d23b8573522ac5b2d2c8dd0644bf

Pushing the same bytes again, pushing a new tag, and deleting:

crane append -f build1.tar -t registry:5000/team-a/app:1.0.0      # same bytes again
crane append -f build2.tar -t registry:5000/team-a/app:1.0.1      # a new tag
crane delete registry:5000/team-a/app@sha256:...                  # CI tries to delete

registry:5000/team-a/app@sha256:9ae6ec75e4976ab1fa6bccbcde092ee9a3f5d23b8573522ac5b2d2c8dd0644bf
registry:5000/team-a/app@sha256:f7af38aa9da7f395fb81e6a77a0bce74aa1b98fedd3c21fa8bbb4b56228f5a99
Error: DELETE http://registry:5000/v2/team-a/app/manifests/sha256:9ae6ec75e4976ab1fa6bccbcde092ee9a3f5d23b8573522ac5b2d2c8dd0644bf: DENIED: requested access to the resource is denied

A retry of an identical push succeeds, so flaky pipelines do not break. A new version gets a new tag. Deletion needs the release admin.

Signatures still work. Signing the same digest twice with cosign v3.0.6 succeeded, and no tag was added, because cosign stored the signatures as OCI referrers:

cosign sign --key cosign.key ... registry:5000/team-a/app@sha256:...   # twice
crane ls registry:5000/team-a/app

Signing artifact...
Signing artifact...
1.0.0
1.0.1

Older cosign versions stored signatures under a sha256-<digest>.sig tag. If you use one, test that a second signature is not blocked by the policy.

#Mistakes people make

#Immutable tags, but deploying :latest

latest either cannot move (and is useless) or is excluded from the rule (and is the tag everyone deploys). Deploy by digest and keep latest out of production manifests.

#CI with delete rights

Immutability that allows delete-and-re-push is not immutability. The CI identity gets read and create only.

#Reusing version numbers after a failed release

With immutable tags, a broken 1.0.0 stays 1.0.0. Publish 1.0.1. That is the point: the history is honest.

#Believing the tag instead of the digest

Even with immutable tags, a registry admin or a registry bug can change things. Pin digests in deployments, and let admission check signatures on the digest.

#Checklist

• The registry refuses to overwrite or delete existing tags for CI identities.
• An admin identity that can update tags exists, is used by people only, and is audited.
• Every build pushes a unique tag (version plus commit).
• No production manifest deploys latest or any moving tag.
• Deployments reference images by digest.
• Retries of identical pushes are tested and succeed.
• Signing and attestation still work under the policy, tested with your cosign version.