Hands-On DevOps Engineering

Hands-On DevOps Engineering

Day 24: Configuration Management & Secrets in GitOps

sdr's avatar
sdr
Dec 09, 2025
∙ Paid

Today’s Mission

You’re about to solve one of the hardest problems in Kubernetes: managing secrets without storing them in Git. By the end of today, you’ll have built a complete secrets management system that automatically syncs secrets from external vaults, encrypts them for Git storage, and detects configuration drift within seconds.

What You’ll Build:

  • External Secrets Operator connected to cloud secret stores

  • Sealed Secrets controller for Git-safe secret encryption

  • Configuration drift detection system with auto-remediation

  • Real-time dashboard showing secret sync status and configuration health


The Core Problem: GitOps Meets Secrets

GitOps promises everything in Git, but secrets create a paradox. You can’t commit plaintext API keys to Git, yet GitOps requires declarative manifests. The industry solved this with two patterns: external secret stores (AWS Secrets Manager, HashiCorp Vault) and encrypted secrets (Sealed Secrets). External Secrets Operator pulls secrets from vaults at runtime, while Sealed Secrets encrypt them so only your cluster can decrypt.

Configuration drift happens when someone runs kubectl apply directly, bypassing Git. In production systems at companies like Netflix, drift detection runs continuously, comparing cluster state against Git truth and auto-remediating within seconds.

The Secret Lifecycle

Developer → Commits SealedSecret → Git Repository → ArgoCD Syncs → 
Sealed Secrets Controller Decrypts → Kubernetes Secret Created →
External Secrets Operator Refreshes from Vault → Application Consumes

Meanwhile, a drift detector compares every resource against Git every 30 seconds, flagging unauthorized changes.


Architecture: Three-Layer Secrets Defense

User's avatar

Continue reading this post for free, courtesy of devops.

Or purchase a paid subscription.
© 2026 ctoi · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture