GitOps: Managing Infrastructure Through Git

If you manage an engineering team running workloads on Kubernetes, you already know the failure mode: someone runs kubectl apply against production at 11 PM to fix an incident, forgets to update the manifests in the repo, and three weeks later a routine redeploy silently reverts the fix. Multiply this across a dozen engineers and several clusters, and your "infrastructure as code" quietly becomes infrastructure as folklore. GitOps exists to close that gap. It makes Git the single source of truth for both your applications and your infrastructure, and it uses automated controllers to guarantee that what's running matches what's declared. This article breaks down how GitOps infrastructure management actually works in practice — the reconciliation model, the tooling choices between Argo CD and Flux, and the operational realities you'll hit as you scale.

• Background / Why This Matters
• Core Concepts and Architecture
• Implementation Strategy
• Scaling and Operational Considerations
• Common Mistakes / What to Avoid
• Frequently Asked Questions
• Conclusion

Background / Why This Matters

Before GitOps, most teams practiced a push-based deployment model: a CI pipeline (Jenkins, GitLab CI, GitHub Actions) held cluster credentials and pushed changes into the environment. This works until it doesn't. The problems compound quietly:

• Configuration drift. Manual hotfixes, dashboard clicks, and out-of-band kubectl commands mean your cluster no longer matches any file in any repo. Nobody can answer "what is actually running right now?" with confidence.
• Credential sprawl. Every CI runner that can deploy to production holds cluster-admin-level credentials. That's a large, mobile attack surface.
• Weak audit trails. When a regulator or a post-incident review asks "who changed this and why," a mix of pipeline logs and Slack messages is not a satisfying answer.
• Slow, error-prone rollbacks. Reverting means re-running pipelines and hoping the previous state was captured somewhere.

GitOps reframes deployment around a simple principle: the desired state of your system lives in Git, and an agent running inside the cluster continuously pulls and applies that state. Research and industry surveys consistently suggest that teams adopting GitOps report faster recovery times and fewer configuration incidents, though the exact numbers vary by organization. The mechanism behind those gains is straightforward — if Git is the source of truth and drift is automatically corrected, your worst 3 AM question ("what's running?") always has a version-controlled answer.

Takeaway: If your team can't reconstruct production purely from a Git repository, you have drift risk today. GitOps is the discipline that eliminates it.

Core Concepts and Architecture

GitOps rests on four principles, popularized by the OpenGitOps project: the system is declarative, versioned and immutable (Git), pulled automatically, and continuously reconciled.

The reconciliation loop

The heart of GitOps is the reconciliation loop, a concept borrowed directly from Kubernetes controllers. A GitOps agent inside the cluster runs a continuous cycle:

1. Observe. Read the desired state from the Git repository (manifests, Helm charts, Kustomize overlays).
2. Compare. Diff the desired state against the actual live state in the cluster.
3. Act. If they differ, apply changes to bring the cluster in line with Git.

This loop runs every few minutes indefinitely. That's the key difference from a CI pipeline: a pipeline runs once when triggered, but a reconciler runs forever. If someone manually edits a deployment, the reconciler detects the drift on its next pass and reverts it — self-healing infrastructure without human intervention.

Pull vs. push

Traditional CI/CD is push-based; GitOps is pull-based. The agent runs inside the cluster and reaches out to Git, rather than an external system reaching in. This means your CI system never needs cluster credentials — a meaningful security improvement. CI's job shrinks to building and testing artifacts and updating the Git repo; deployment becomes Git's job.

The two tools: Argo CD and Flux

The two dominant CNCF-graduated GitOps controllers are Argo CD and Flux. Both are excellent; the right choice depends on how your team works.

Dimension	Argo CD	Flux
UI	Rich built-in web UI with visual app topology	CLI-first; UI via third-party (e.g., Weave GitOps, Capacitor)
Model	Application-centric (Application CRD)	Toolkit of composable controllers (Source, Kustomize, Helm)
Multi-tenancy	Projects, RBAC, SSO integration out of the box	Strong via namespaces and per-tenant reconcilers
Best fit	Teams wanting visibility and self-service dashboards	Teams preferring a lean, GitOps-native, automation-heavy setup
Image automation	Via Argo CD Image Updater (separate component)	Built-in image reflector and automation controllers

A useful rule of thumb: if your engineering managers and on-call engineers value a visual dashboard to see sync status and diffs at a glance, Argo CD tends to win. If your team is comfortable operating from the CLI and wants a composable, minimal toolkit, Flux fits naturally.

Where Terraform fits

GitOps and Terraform are complementary, not competing. Terraform excels at provisioning the cloud primitives underneath Kubernetes — VPCs, managed clusters (EKS, GKE, AKS), IAM roles, databases. GitOps controllers then manage everything inside the cluster. A common pattern is: Terraform builds the platform, then bootstraps Argo CD or Flux, which takes over from there. There are also GitOps-style Terraform controllers (like the Flux Terraform Controller or Atlantis) that apply the reconciliation model to Terraform itself.

Takeaway: Use Terraform for the cloud foundation, and a GitOps controller for in-cluster state. Keep the boundary clean and documented.

Implementation Strategy

Rolling out GitOps successfully is less about the tool and more about repository structure, promotion flow, and secrets handling. Here's a pragmatic sequence.

1. Decide your repository topology

Two common patterns:

• Monorepo: All environments and apps in one repository with directory-based separation. Simpler to start, easier cross-cutting changes, but access control is coarser.
• Multi-repo: Separate repos per team, app, or environment. Better isolation and RBAC, more overhead to coordinate.

Most teams start with a single "config repo" separate from application source code. Keeping application code and deployment manifests in different repos avoids the awkward loop where a deploy commit triggers a rebuild.

2. Structure environments with overlays

Use Kustomize overlays or Helm value files to represent dev, staging, and production as variations of a shared base. This prevents copy-paste divergence and makes promotion a matter of updating an image tag or version in one place.

3. Design promotion as a pull request

Promoting a release from staging to production should be a PR that bumps a version or image digest. This gives you the audit trail, review gate, and rollback path for free — reverting is just reverting the commit. This is where GitOps genuinely changes team behavior: production changes become reviewable, discussable artifacts rather than ephemeral commands.

4. Solve secrets before you go live

Plain secrets can't live in Git. The mature options:

• Sealed Secrets (Bitnami) — encrypts secrets so only the in-cluster controller can decrypt them; the encrypted form is safe to commit.
• External Secrets Operator — syncs from a real secret manager (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) into the cluster.
• SOPS — encrypts values with age or KMS keys; Flux and Argo CD both integrate with it.

5. Bootstrap and hand off

Install your controller (via Terraform or the tool's bootstrap command), point it at your config repo, and let it take over. From this point, no human should run kubectl apply against production.

This is precisely the kind of platform foundation Halkwinds' engineering teams build for clients — a clean Terraform base, a chosen GitOps controller wired to well-structured repositories, and a secrets strategy that survives an audit. Getting the topology right at the start saves painful migrations later.

Takeaway: Decide repo structure and secrets handling before installing anything. Those two decisions are the hardest to change later.

Scaling and Operational Considerations

GitOps that works for one cluster and five services can strain at fifty clusters and hundreds of services. Plan for these dimensions:

Multi-cluster fleet management

As you add clusters, managing each independently doesn't scale. Argo CD supports an "app of apps" pattern and ApplicationSets to template deployments across many clusters. Flux uses the same manifests targeted at multiple reconcilers. Consider a hub-and-spoke model where a management cluster orchestrates deployments to workload clusters.

Reconciliation performance

Each reconciliation pass costs CPU and API calls. With thousands of resources, tune reconciliation intervals, use webhook-triggered syncs instead of pure polling, and shard reconcilers across multiple controller replicas. Watch the controller's own resource consumption — it's a real workload, not free.

Drift detection vs. auto-heal

You can configure the controller to report drift but not automatically correct it, or to aggressively self-heal. Auto-heal is powerful but dangerous during incidents — if an engineer legitimately patches something live to stop the bleeding, an aggressive reconciler will revert it mid-incident. Establish a clear "break glass" procedure: how to pause reconciliation, and how to fold emergency changes back into Git