GitOps without circular dependencies, the secure way

#What goes wrong

GitOps makes Git the source of truth, so the cluster rebuilds itself from a repository. The trap is that the repository, and everything needed to read it, often runs inside the cluster it is supposed to rebuild.

Common loops:

• Argo CD reads from a forge that Argo CD deploys on the same cluster.
• The registry runs in the cluster and serves its own image and the image of the forge.
• Argo CD's repository credentials come from a secrets store that Argo CD deploys, and the store's unseal key is inside the store.
• Argo CD login goes through an SSO service in the same cluster, and the local admin is disabled.
• CI runners in the cluster build the images the cluster needs to start.

Each loop is fine while the cluster is up. When the cluster is lost, you need the forge to restore the forge. That is also a security problem: under pressure, people restore from whatever copy they can find, with whatever credentials still work, and skip every check.

#What the docs say

Argo CD is able to manage itself since all settings are represented by Kubernetes manifests.

Source: Argo CD docs, Declarative Setup

When managing Argo CD with Argo CD, you must enable the ServerSideApply=true sync option.

Source: Argo CD docs, Declarative Setup

The ability to create Applications in arbitrary Projects is an admin-level capability. Only admins should have push access to the parent Application's source repository.

Source: Argo CD docs, Cluster Bootstrapping

Besides installing the controllers, the bootstrap command pushes the Flux manifests to the Git repository and configures Flux to update itself from Git.

Source: Flux docs, Installation

Both tools explain how to manage themselves from Git. Neither page discusses what happens when that Git server is itself deployed by the tool, which is the case that breaks a rebuild.

#The secure configuration

#1. Write the tiers down

A component may depend only on components in a lower tier.

If the forge runs on the same cluster it deploys, it is tier 3, and Argo CD must be able to read the tier 0 mirror when the forge is gone. Better still, run the forge on a different cluster from the ones it deploys.

#2. Manifests that do not assume they are already running

# Only platform admins can push to the repository that holds these files.
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: platform
  namespace: argocd
spec:
  sourceRepos:
    - https://git.example.com/platform/gitops.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: "*"
  clusterResourceWhitelist:
    - { group: "*", kind: "*" }
---
# Argo CD managing itself. The same manifests are applied by hand in a
# rebuild, so the cluster can come back without Argo CD already running.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://git.example.com/platform/gitops.git
    targetRevision: main
    path: bootstrap/argocd                 # kustomization pinning the Argo CD release
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    syncOptions:
      - ServerSideApply=true               # required when Argo CD manages Argo CD
    # no "automated": a bad commit must not be able to break the tool that reverts it
---
# App of apps: everything else, in sync waves. Tier 0 never depends on this cluster.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://git.example.com/platform/gitops.git
    targetRevision: main
    path: clusters/prod/apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: false                         # deleting a line in Git must not delete a database
      selfHeal: true

The platform project is the only one allowed to deploy into argocd and to create cluster-scoped objects. Team projects never get that.

The kustomization that Argo CD manages itself from must work on an empty cluster, and must carry everything Argo CD needs to reach Git:

# bootstrap/argocd/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd
resources:
  - namespace.yaml          # install.yaml has no Namespace; a rebuild starts from an empty cluster
  - install.yaml            # the Argo CD release, vendored: no download during a rebuild
patches:
  - path: argocd-tls-certs-cm.yaml   # the forge's CA, so a self-sync keeps it

argocd-tls-certs-cm.yaml is the argocd-tls-certs-cm ConfigMap with the CA of the forge's certificate under the forge's host name. Anything you add to Argo CD's ConfigMaps by hand is replaced by the version in Git the first time Argo CD syncs itself.

#3. The tier 0 kit

# Nightly, from a machine outside the cluster: a verifiable copy of the GitOps repo.
git clone --mirror https://git.example.com/platform/gitops.git gitops.git
git -C gitops.git bundle create ../gitops-$(date +%F).bundle --all
git -C gitops.git bundle verify ../gitops-$(date +%F).bundle   # verify needs a repository
sha256sum gitops-$(date +%F).bundle > gitops-$(date +%F).bundle.sha256
# Copy the bundle, the bootstrap images (by digest) and a checksum file to
# storage the cluster's own credentials cannot delete.

Break-glass credentials (a Kubernetes admin kubeconfig, the secrets-store unseal or recovery keys, a registry push credential for the mirror) are stored offline, split between people where the tool supports it, and never only inside the cluster.

The kit also holds what it takes to serve the bundle under the forge's name when the forge is gone: a small read-only Git server definition and a way to get a certificate for git.example.com from the forge's CA (the offline CA itself, or a certificate kept for this purpose). Every Application in Git names the forge's URL, so changing repoURL on the root app is not enough; serving the bundle under the same name is. The server used in the test, nginx in front of git http-backend:

server {
  listen 8443 ssl;
  ssl_certificate /tls/tls.crt;          # git.example.com, from the forge's CA
  ssl_certificate_key /tls/tls.key;
  ssl_protocols TLSv1.3;
  location ~ ^/.*/git-receive-pack$ { return 403; }        # read-only: no pushes
  location / {
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /usr/libexec/git-core/git-http-backend;
    fastcgi_param GIT_PROJECT_ROOT /srv;   # the bundle, cloned with --mirror to /srv/platform/gitops.git
    fastcgi_param GIT_HTTP_EXPORT_ALL "";
    fastcgi_param PATH_INFO $uri;
    fastcgi_param QUERY_STRING $args;
    fastcgi_pass unix:/run/fcgiwrap.sock;
  }
}

It has to speak Git's smart HTTP protocol: Argo CD lists refs with a client that fails on a plain static ("dumb HTTP") copy with failed to list refs: unexpected EOF.

#4. The rebuild, in order

# Tier 1 is up (nodes, CNI, DNS). The forge is gone.
(cd tier0 && sha256sum -c gitops-2026-09-24.bundle.sha256)
git clone gitops-2026-09-24.bundle gitops                      # from tier 0
kubectl apply --server-side -k gitops/bootstrap/argocd         # tier 2, by hand
kubectl apply -f gitops/bootstrap/appproject.yaml              # the project the apps below belong to
# Start the temporary Git server from the kit, then point the forge's name at it:
#   CoreDNS: rewrite name git.example.com git.tier0-git.svc.cluster.local
kubectl apply -f gitops/bootstrap/repo-mirror-secret.yaml      # repo credential from the offline kit
kubectl apply -f gitops/bootstrap/argocd-app.yaml -f gitops/bootstrap/root-app.yaml
# Review, then sync the argocd Application once by hand (it has no automated sync).

Once the forge is restored, remove the DNS rewrite and the temporary server; nothing in Git changes.

#Prove it

The drill, run on a lab machine. A GitOps repo held the manifests on this page, Argo CD v3.5.3 vendored in bootstrap/argocd, and a demo app. The tier 0 commands made the bundle; then the source repo was deleted, and a new empty kind cluster was rebuilt from the bundle. git.example.com has no public address, so nothing could reach a "live" forge.

1. The bundle commands, as first written, fail at verify:

$ git bundle verify gitops-2026-09-25.bundle
error: need a repository to verify a bundle
$ git -C gitops.git bundle verify ../gitops-2026-09-25.bundle
The bundle contains these 2 refs:
2e03b923b58bfb6534271d111b064fcc06fe83cc refs/heads/main
2e03b923b58bfb6534271d111b064fcc06fe83cc HEAD
The bundle records a complete history.
../gitops-2026-09-25.bundle is okay

Same result with git 2.43 and 2.52.

2. What each missing piece did to the rebuild:

no Namespace in the kustomization:   Error from server (NotFound): namespaces "argocd" not found
AppProject not applied:              root  Unknown  Application referencing project platform which does not exist
static (dumb HTTP) Git server:       failed to list refs: unexpected EOF
forge CA added by hand, not in Git:  argocd synced itself, then:
  failed to list refs: Get "https://git.example.com/platform/gitops.git/info/refs?service=git-upload-pack": tls: failed to verify certificate: x509: certificate signed by unknown authority

The last one is the dangerous one: Argo CD syncs its own ConfigMaps from Git, drops the CA you added during the rebuild, and cuts itself off from the only Git server it has.

3. The order in section 4, from an empty cluster:

gitops-2026-09-25.bundle: OK
$ kubectl apply --server-side -k gitops/bootstrap/argocd
$ kubectl apply -f gitops/bootstrap/appproject.yaml
appproject.argoproj.io/platform created
temporary git server up as git.example.com
$ kubectl apply -f gitops/bootstrap/repo-mirror-secret.yaml
secret/gitops-repo created
$ kubectl apply -f gitops/bootstrap/argocd-app.yaml -f gitops/bootstrap/root-app.yaml
application.argoproj.io/argocd created
application.argoproj.io/root created
== after 139s from an empty cluster
NAME     SYNC     HEALTH    MESSAGE
argocd   Synced   Healthy   <none>
demo     Synced   Healthy   <none>
root     Synced   Healthy   <none>
web   1/1   1     1     73s

Before its first manual sync, the argocd Application showed OutOfSync with 57 resources: objects applied by hand carry no Argo CD tracking annotation until Argo CD syncs them once.

4. Run it yourself, twice a year, on a throwaway cluster:

1. Block the forge, the registry and the SSO service from the drill cluster with a firewall rule, so nothing can quietly use the live ones.
2. Run the rebuild in section 4 using only the tier 0 kit.
3. Time it and write down every step that needed something not in the kit.

What you should see: every Application Synced and Healthy, and Argo CD still able to fetch after it has synced itself. Every manual workaround you needed is a loop to break before the next drill.

The bootstrap manifests also validate offline:

kubeconform -strict -summary -kubernetes-version 1.34.0 \
  -schema-location default \
  -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
  bootstrap.yaml

What you should see: a summary with 3 resources, 3 valid, 0 invalid.

#Mistakes people make

#Automated sync on the Argo CD Application

With automated on Argo CD's own Application, one bad commit can break Argo CD, and there is then no Argo CD to sync the revert. Sync Argo CD by hand after review.

#Trust added by hand during the rebuild

Repository CAs, known hosts and settings you add to Argo CD's ConfigMaps by hand are replaced by Git's version at the first self-sync. Put the forge's CA in the kustomization Argo CD manages itself from.

#Repointing only the root app

Every child Application in Git names the forge's URL. Serve the bundle under the forge's name instead of editing repoURL.

#The only copy of the repo is on the forge

A forge backup inside the cluster is lost with the cluster. Keep bundles outside it, in storage the cluster's credentials cannot delete.

#Admin disabled and SSO in the same cluster

If SSO is down, nobody can log in to Argo CD to fix SSO. Keep a documented break-glass path, such as kubectl access with an offline kubeconfig, and test it in the drill.

#Unseal keys inside the thing they unseal

A secrets store that auto-unseals from a service in the same cluster, or whose recovery keys are stored in it, cannot start after a full loss. Keep the recovery material offline.

#Pruning on the root app

prune: true on the app of apps means a deleted or moved directory deletes workloads, volumes included. Prune per application, where you understand what it removes.

#Checklist

• Each platform component has a tier, and depends only on lower tiers.
• The GitOps repo is bundled and verified nightly to storage outside the cluster.
• Bootstrap images are mirrored by digest outside the cluster.
• Break-glass credentials and unseal or recovery keys are stored offline.
• Argo CD's own Application uses ServerSideApply=true and no automated sync.
• Only the platform project can deploy into argocd or create cluster-scoped objects.
• The root app does not prune automatically.
• The Argo CD kustomization includes its Namespace and the forge's CA.
• The tier 0 kit can serve the bundle read-only, over smart HTTP, under the forge's name.
• A written rebuild order exists and uses only the tier 0 kit.
• A cold-start drill with the forge, registry and SSO blocked runs twice a year.