Offline cosign signing, the secure way

#What goes wrong

Sigstore's default is keyless signing: a short-lived certificate from Fulcio and an entry in the public Rekor log. That needs the internet, and it publishes your signing identity and artifact digests to a public log. Some build networks cannot reach the internet, and some organizations cannot publish that data.

The usual workaround is cosign sign --key ... --tlog-upload=false. In cosign v3 that flag is deprecated and fails in the default mode, so people reach for the next thing that makes the error go away. Common results:

• The private key sits in a CI variable that every pipeline can read.
• Images are signed by tag, so the signature covers whatever the tag pointed to at that moment.
• Verification is switched off "for now" because it kept failing offline.

#What the docs say

While keyless signing is recommended, you may specify your own keys for signing.

Source: Sigstore docs, Signing blobs

Make sure to sign the image by its digest (@sha256:...) rather than by tag (:latest) so that you actually sign what you think you're signing! This prevents race conditions or (worse) malicious tampering.

Source: cosign reference, cosign sign

ignore transparency log verification, to be used when an artifact signature has not been uploaded to the transparency log. Artifacts cannot be publicly verified when not included in a log

Source: cosign reference, cosign verify-blob

The docs still show --tlog-upload=false in examples. The cosign source marks it deprecated with the note "prefer using a --signing-config file with no transparency log services", and cosign v3.0.6 refuses the flag unless you also turn off signing configs. No docs page walks through the signing-config route end to end; the commands below do.

#The secure configuration

Generate the key pair once, on a machine with no network, and store the private key and its password apart.

# On an offline machine. The password comes from a prompt, not the command line.
cosign generate-key-pair
# cosign.key  (encrypted private key)  -> offline storage or an HSM/KMS
# cosign.pub  (public key)             -> Git, policy-controller, verifiers

Create a signing config with no services. It is the same for every signer, so commit it next to the public key.

cosign signing-config create --out signing-config.json
# {"mediaType":"application/vnd.dev.sigstore.signingconfig.v0.2+json", "rekorTlogConfig":{}, "tsaConfig":{}}

Sign release files. The bundle holds the signature; ship it with the file.

# COSIGN_PASSWORD is read from the environment; set it from your secret store,
# never type it on the command line.
cosign sign-blob --yes \
  --key cosign.key \
  --signing-config signing-config.json \
  --bundle app-1.0.0.bundle \
  app-1.0.0.tar.gz

Sign images by digest only. The signature is stored next to the image in the same registry.

DIGEST=$(crane digest registry.example.com/team-a/app:1.0.0)
cosign sign --yes \
  --key cosign.key \
  --signing-config signing-config.json \
  "registry.example.com/team-a/app@${DIGEST}"

If the images are checked at admission by the Sigstore policy-controller with a public key, sign them in the classic format instead: the controller (v0.15.1) rejects the new bundle format that a signing config produces. See rolling out the Sigstore policy-controller:

cosign sign --yes --key cosign.key \
  --use-signing-config=false --tlog-upload=false --new-bundle-format=false \
  "registry.example.com/team-a/app@${DIGEST}"

Verify with the public key. Without a log, verification must say so explicitly:

cosign verify-blob --key cosign.pub --bundle app-1.0.0.bundle \
  --insecure-ignore-tlog=true app-1.0.0.tar.gz

cosign verify --key cosign.pub --insecure-ignore-tlog=true \
  "registry.example.com/team-a/app@${DIGEST}"

Replace what the log would have given you: append every signature (date, digest, signer, pipeline run) to a signing record that CI cannot rewrite, such as a protected Git branch or write-once storage.

#Prove it

All commands below ran with --network none. Key pair and signing config:

cosign generate-key-pair
cosign signing-config create --out signing-config.json

Private key written to cosign.key
Public key written to cosign.pub
{"mediaType":"application/vnd.dev.sigstore.signingconfig.v0.2+json", "rekorTlogConfig":{}, "tsaConfig":{}}

Signing a file. cosign tries to fetch its trust root, fails because there is no network, and continues:

cosign sign-blob --key cosign.key --signing-config signing-config.json --bundle app-1.0.0.bundle app-1.0.0.tar.gz

WARNING: Could not fetch trusted_root.json from the TUF repository. Continuing with individual targets. Error from TUF: error getting live trusted root: failed to create TUF client failed to load metadata: tuf refresh failed: Get "https://tuf-repo-cdn.sigstore.dev/14.root.json": dial tcp: lookup tuf-repo-cdn.sigstore.dev on <resolver>:53: dial udp <resolver>:53: connect: network is unreachable
Using payload from: app-1.0.0.tar.gz
Signing artifact...
Wrote bundle to file app-1.0.0.bundle

Verifying without --insecure-ignore-tlog fails offline; with it, the check passes; after one byte changes, it fails:

cosign verify-blob --key cosign.pub --bundle app-1.0.0.bundle app-1.0.0.tar.gz
cosign verify-blob --key cosign.pub --bundle app-1.0.0.bundle --insecure-ignore-tlog=true app-1.0.0.tar.gz
echo tampered >> app-1.0.0.tar.gz
cosign verify-blob --key cosign.pub --bundle app-1.0.0.bundle --insecure-ignore-tlog=true app-1.0.0.tar.gz

Error: trusted root is required when using new bundle format

WARNING: Skipping tlog verification is an insecure practice that lacks transparency and auditability verification for the blob.
Verified OK

Error: failed to verify signature: could not verify message: invalid signature when validating ASN.1 encoded signature

An image in a registry on an internal network that cannot reach the internet:

cosign sign --key cosign.key --signing-config signing-config.json registry:5000/team-a/app@sha256:...
cosign verify --key cosign.pub --insecure-ignore-tlog=true registry:5000/team-a/app@sha256:...

Signing artifact...

WARNING: Skipping tlog verification is an insecure practice that lacks transparency and auditability verification for the signature.
Verification for registry:5000/team-a/app@sha256:ee88369d765afd93855cdef1d524ab409d6df363fe33f912febe9abf8994b0c6 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The signatures were verified against the specified public key
1 signature(s) for sha256:ee88369d765afd93855cdef1d524ab409d6df363fe33f912febe9abf8994b0c6

Note the second check in that list. cosign v3.0.6 prints it even though the transparency log was skipped and there is no log entry. Trust the warning line above it, not that bullet.

#Mistakes people make

#Signing the tag

cosign sign registry.example.com/app:1.0.0 signs whatever the tag points to when cosign resolves it. Resolve the digest yourself, sign the digest, deploy the digest.

#The private key in a CI variable for every pipeline

Anyone who can edit any pipeline can print the variable. Sign in one dedicated, protected job, or use a KMS key that the job can use but not export.

#The password on the command line

COSIGN_PASSWORD=... cosign sign in a script ends up in shell history and process lists. Load it from a secret store into the environment of the signing job only.

#Turning verification off because it fails offline

The fix for "trusted root is required" in a key-based, no-log setup is --insecure-ignore-tlog=true with --key, not skipping verification. Keep the verify step, and make it fail the pipeline.

#No record of what was signed

A public log lets anyone see every signature made with a key. Without one, a stolen key can sign quietly. Keep your own append-only record, and compare signatures found in the registry against it.

#Checklist

• The key pair was generated offline; the private key is not in any cluster or shared CI variable.
• The signing password is read from a secret store, never typed on a command line.
• signing-config.json with no services is committed next to cosign.pub.
• Images are signed by @sha256: digest only.
• Release files ship with a .bundle next to them.
• Every verify step uses --key cosign.pub --insecure-ignore-tlog=true and fails the pipeline on error.
• Each signature is appended to a signing record CI cannot rewrite.
• A key rotation and revocation procedure is written down.