Week 4: GitOps & Declarative Operations (Days 22-28)

What We’re Building Today

Today we’re building a complete GitOps operations platform that handles everything from multi-environment Argo CD sync to policy gates, infrastructure claims, disaster recovery, and real-time SLO monitoring. Think of it as building the control surface that platform teams at Stripe, Spotify, and Shopify use during incidents—one place to answer: Is Git in sync with the cluster? Are we compliant? Can we fail over? Are we burning error budget?

Build Agenda:

• Argo CD sync hub that tracks applications across dev, staging, and prod with one-click reconciliation

• Advanced GitOps engine with ApplicationSet matrices, app-of-apps, and canary rollout weight progression

• Secrets & drift controller combining Sealed Secrets in Git with External Secrets sync and live drift scans

• Policy-as-code gate (Gatekeeper-style) with audit, remediation, and compliance scoring

• Crossplane infrastructure broker that provisions database and storage claims across AWS and GCP

• Velero disaster-recovery plane with backup schedules, RTO/RPO modeling, and failover simulation

• Observability & SLO layer with error budgets, sync latency P99, WebSocket live ticks, and incident resolution

Core Concepts: Why GitOps Control Planes Matter

The Reconciliation Economics Problem

A mid-size SaaS fleet might run 200+ Argo CD Applications across three regions. Each out-of-sync app is latent debt: the cluster is running something Git never approved. Incidents often start with a manual kubectl patch that never made it back to the repo—then drift scanners, policy audits, and SOC2 evidence all disagree on “source of truth.”

GitOps fixes the contract: Git is desired state; the cluster is the receipt. Argo CD reconciles continuously. The expensive failure mode isn’t a failed deploy—it’s silent divergence between what engineers think is live and what Git declares.

Multi-Environment Promotion: The Game Changer

ApplicationSets generate the same chart across dev → staging → prod with parameter overlays. One promotion mistake can fan out to every environment. Production teams gate staging → prod on analysis runs (Argo Rollouts) and policy compliance—exactly the handoff Week 4 simulates in Day 23 canary metrics and Day 25 violation gates.

When sync success drops from 99% to 92%, you’re not looking at a broken pod—you’re looking at reconciliation debt that will compound until someone runs a game-day sync across the fleet.

GitOps Sharing Strategies (Operational Patterns)

Per-app sync (Day 22): Operators trigger sync for one application/environment pair. Netflix-style microservices: many small apps, independent blast radius.

Matrix + canary (Day 23): ApplicationSet generates N environments; Rollouts advance canary weight. Spotify-style: same artifact, progressive exposure.

Secrets dual-path (Day 24): Sealed Secrets encrypted in Git; External Secrets pull from vault at runtime. The pattern enterprises use for “audit in Git, rotate in vault.”

Policy before pod (Day 25): Gatekeeper denies non-compliant manifests at admission—cheaper than debugging a running deployment that should never have been admitted.

Infra as claims (Day 26): Crossplane XRDs turn “I need a Postgres” into a reconciled cloud resource—GitOps for infrastructure, not just apps.

DR as scheduled truth (Day 27): Velero backups with explicit RPO; failover API models RTO trade-offs—the same questions execs ask in regional outages.

System Architecture