Argo CD at 34 Clusters: GitOps Patterns for Safer Kubernetes Fleet Management
Running GitOps across dozens of Kubernetes clusters turns small configuration choices into fleet-wide operational decisions. At Lucid Motors, one Argo CD instance managed 34 clusters across 22 AWS accounts and an Oracle Cloud tenancy, spanning five regions on three continents. The system drew on 174 Helm charts to render 1,214 Argo CD Application objects. At that scale, the hard work was defining safe boundaries, making changes reviewable and ensuring a controller’s reach didn’t exceed the change’s intended impact.
The timing offered a reminder about a dependency that’s easy to overlook. On Oct. 6, Red Hat published a walkthrough of using Argo CD ApplicationSets to provision hosted control planes from Git. On Oct. 7, GitHub experienced an outage affecting Actions, Git operations and pull requests, following smaller incidents on Oct. 5 and 6. GitOps now reaches beyond application deployment into cluster provisioning. That makes Git and the central control plane important dependencies, even when workloads are running elsewhere.
Put the control plane’s authority on a short leash
The hub hosted the only Argo CD instance for platform components. It held cluster registrations, the Cluster API controller identity and three ApplicationSets. One rolled out a per-cluster baseline through an app-of-apps chart. Another deployed shared platform services to the clusters hosting them. The third managed the Cluster API definitions used to create clusters.
Each spoke retained control over access within its own boundary. Its authentication configuration determined whether to admit the hub’s role, and the spoke owned the role Cluster API used there. Workloads received cloud roles through each spoke’s identity provider. That division meant the hub could coordinate fleet operations without owning every cluster’s credentials and permissions.
On AWS, the hub used short-lived credentials rather than storing long-lived cross-account secrets. Each grant was declared on the spoke side and reviewed in Git. Oracle Cloud did not support the same arrangement in this setup: registration still read a stored credential from a vault. That exception matters because a multi-cloud architecture can look consistent in its diagrams while relying on different security mechanics in practice. Teams should record where the credential model changes; here, the boundary appeared at the second provider.
A small cluster contract drives the fleet
The design hinged on treating every cluster as a compact configuration file containing about 20 keys. Those values described identity, account boundary, region and environment; how to reach the cluster API; delegated DNS and certificates; and any opt-outs.
Everything downstream was derived from that contract. The same source determined which Applications a cluster received, its IAM role names, its Cluster API identity, DNS delegation and cost tags. There was no separate set of hand-maintained downstream configuration to reconcile with the cluster definition.
This made routine fleet changes simple to express. Adding a cluster to a fleet meant changing a label on its registration. Adding a component to a fleet took one line in a list, with matrix generators handling the fan-out. An Oracle Cloud tenancy joined the system during a Middle East expansion through the same contract. That was a useful test: the model could accommodate a new account boundary without turning the deployment into a one-off.
Render the chart graph before changes reach clusters
With 174 charts, Argo CD itself wasn’t the primary risk. A values change could render successfully for one cluster and still break deployments across many others. The team addressed that with CI checks that rendered the entire configuration graph on every merge request.
Each chart had to render independently. Each app-of-apps chart had to render against every cluster contract, and every cluster definition had to render. CI also rendered each chart referenced by an Application with the exact values that Application would receive. The goal was to catch failures where they could actually occur: in the combination of chart, cluster and supplied values.
Reviewers also needed to see the effect of a change, not just the template code behind it. CI rendered the target branch and attached the resulting manifest diff to the merge request. That put the expected cluster changes in front of reviewers before merge, making it easier to assess the operational result of a values edit.
Measure drift without overstating the result
Drift incidents fell from 24.3 a month before cutover to 0.9 afterward, measured over two 90-day windows. Those figures point in the same direction, but the methods differed. The pre-cutover count came from incident tickets tagged as drift, and tagging was incomplete. The later count came from controller diff events with a confirmed out-of-band cause. The numbers therefore don’t support a clean percentage comparison.
They do show the value of instrumentation: once drift could be identified through controller events, confirmed cases were rare, and the controller corrected each one within its reconciliation interval without human intervention. For teams evaluating a rollout, consistent measurement before and after would make the change easier to quantify.
Make destructive changes hard to trigger
A single bad commit was the failure mode the system needed to contain. ApplicationSets were configured to preserve deployed resources when an entry was removed, preventing removal from cascading into deletion. Cluster objects carried a no-prune setting, so reverting a commit couldn’t delete a running cluster. These controls matched the potential impact of reconciliation to the scale and reversibility of the change.
The same principle applies when GitHub or the hub is unavailable. Workloads run in the spokes, so existing services can continue even if Git or the central control plane stops responding. Deployments and configuration changes that depend on GitOps will pause, but the outage needn’t become a workload outage. The architecture should make that safe failure mode the default.
The hub itself deserves treatment as a highly privileged system. A compromised repository or CI pipeline could give an attacker the hub’s reach once a change merges. The control plane’s access and the path by which changes enter it are part of the same operational risk.
Onboarding gets shorter when the path is repeatable
Bringing a cluster under management followed four steps: create the matching IAM roles once in Terraform, write the cluster’s values file, open a merge request for CI to render the full graph and post its diff, then merge and label the registration. Older clusters that had been built by hand could be adopted in place through an explicit import path.
Mean onboarding time was 4.0 days for eight earlier clusters, reconstructed from change tickets. After cutover, it was 1.7 hours across eleven clusters, measured from merge to first successful sync. The figures come from small samples and different measurement windows, and the cutover also removed manual approval steps. They shouldn’t be read as a direct measurement of GitOps alone. Still, the operational direction is clear: a defined contract and a repeatable review path can sharply reduce the hands-on work of adding clusters.
GitOps still relies on declaring state, reviewing changes and allowing a controller to converge on the desired configuration. Fleet scale raises the cost of shortcuts: credentials in the hub, charts that CI hasn’t rendered with their actual values, and default pruning behavior can become incidents when repeated across dozens of clusters. A clear cluster contract, full-graph rendering and safeguards matched to the blast radius give platform teams a practical way to expand control without making every change a fleet-wide gamble.
Nitish Mane is a software engineer at Philo, where he builds infrastructure for AI-assisted production operations. He previously worked on platform infrastructure at Lucid Motors, including its large-scale GitOps control plane.
