The GitOps Philosophy

Why this comes after the pipeline#

The pipeline you built in Continuous Integration (Jenkins) stops one step short of the cluster. It compiles, tests, scans, pushes an image to a registry — and then something still has to put that image on Kubernetes.

So far that something has been you, running kubectl with credentials that can change anything in the cluster. Which means your pipeline needs those credentials too, and that whatever is running is whatever the last person to run a command left behind. There is no record of why the cluster looks the way it does.

This phase moves that last step, and moves the authority with it.

Introduction to GitOps#

The next chapter covers Argo CD, the tool. This one is about GitOps, the philosophy it implements.

GitOps inverts who is in charge of deployment. Instead of a pipeline pushing changes into a cluster, an agent inside the cluster pulls them from git — which makes the repository the single source of truth for what is running. If a server, a firewall rule, or a Kubernetes pod is not defined in Git, it is considered rogue and must be deleted.

Level 1 — Beginner#

What is GitOps?#

Imagine a strict law inside a kingdom. The Law says: "If a rule is not written in the Great Book, it is not a law."

• The Great Book: The Git repository (GitHub).
• The Kingdom: The Kubernetes Cluster.
• The King's Guard: ArgoCD.

Suppose someone builds a house (a Pod) that is not in the Great Book. The Guard does not see it straight away. The Guard checks on a schedule, and notices at the next inspection.

Even then, the Guard only removes it if you gave the Guard permission to remove things. Both of those are settings you choose, and the next section names them. If you want to build a house, you must write a request in the Great Book (a Pull Request), get it approved by the King (Merged), and then the Guard will build it for you.

ASCII Diagram: The Single Source of Truth#

[ Developer ] ---> Types `kubectl edit deployment` ---> ❌ BLOCKED
 
[ Developer ] ---> Opens Pull Request in GitHub
                         |
                   (Manager Approves)
                         |
[ GitHub Repo ] <==========================> [ Kubernetes Cluster ]
   (The Book)          (ArgoCD Sync)             (The Kingdom)

Level 2 — Intermediate#

The 4 Principles of GitOps#

According to OpenGitOps (a CNCF working group), a system must follow 4 rules to be considered GitOps:

1. Declarative: The entire system must be described declaratively (e.g., YAML, Terraform). No imperative bash scripts.
2. Versioned and Immutable: The desired state is stored in a way that enforces immutability and versioning (e.g., Git). You can always rollback.
3. Pulled Automatically: Software agents automatically pull the desired state from Git. No humans push.
4. Continuously Reconciled: The agent observes the live system on a loop and compares it with Git. If they differ it reports the difference, and — where it has been configured to act — corrects it. "Continuously" means repeatedly, not instantly: correction arrives within the reconciliation interval, and the agent only reverts out-of-band changes if self-healing is switched on.

Why is this better?#

• Auditing: In traditional Ops, if the database goes down, you ask, "Who ran a command to delete the database?" Nobody knows. In GitOps, you look at the git log. Every change has an author, a timestamp, and a Pull Request approval.
• Disaster Recovery: If the entire AWS datacenter burns to the ground, you just create a new empty cluster, install ArgoCD, and point it at GitHub. The entire architecture rebuilds itself in minutes.

Level 3 — Advanced#

Analyzing the Actual Code (GitOps in Practice)#

How does GitOps handle Image Updates?

When Jenkins finishes compiling ivolve-api:v2.0 and pushes the image to ECR, how does the cluster know to update? Jenkins does NOT run kubectl set image.

Instead, the last stage of our Jenkinsfile looks like this (pseudocode):

stage('GitOps Update') {
    steps {
        sh '''
        git clone https://github.com/Waleeddarwesh/ivolve-cloud-devops-internship.git
        cd kubernetes/base/
        sed -i 's/ivolve-api:v1.0/ivolve-api:v2.0/g' api-deployment.yaml
        git commit -am "chore: update API image to v2.0"
        git push origin main
        '''
    }
}

The Workflow:

1. Jenkins acts as a normal Git user. It edits the text file and pushes a commit.
2. The Jenkins pipeline is now finished. It goes to sleep.
3. ArgoCD sees the new commit (ivolve-api:v2.0).
4. ArgoCD updates the cluster.

Practise: GitOps Delivery with Argo CD: Sync, Drift & Self-Heal closes the loop: change Git, watch the cluster follow, then break it by hand.

The Two-Repo Pattern#

In an advanced architecture, you do not put your application source code (Java/Python) in the same repository as your GitOps manifests (YAML).

• Repo 1 (App Repo): Developers work here. A commit here triggers Jenkins to build a Docker image.
• Repo 2 (Manifest Repo): Only ArgoCD watches this repo. Jenkins pushes YAML updates here.

This prevents an infinite loop where Jenkins updates a YAML file, which creates a Git commit, which triggers Jenkins to run again, which updates a YAML file...

Level 4 — Enterprise#

Enterprise Patterns: Secrets Management in GitOps#

GitOps demands that everything must be in Git. But what about the Database Password? If you put password: SuperSecret123 in a YAML file in GitHub, you will be fired.

The Solution: Sealed Secrets or External Secrets Operator (ESO) In this platform, we use ESO.

1. We store the real password in AWS Secrets Manager.
2. In Git, we define a Custom Resource called an ExternalSecret.

   yamlCopy

   apiVersion: external-secrets.io/v1beta1
   kind: ExternalSecret
   metadata:
     name: db-credentials
   spec:
     secretStoreRef:
       name: aws-secrets-manager
     target:
       name: k8s-db-secret
     data:
     - secretKey: password
       remoteRef:
         key: prod/db/password

3. ArgoCD applies this file to the cluster.
4. The ESO controller inside the cluster reads this file, authenticates with AWS via an IAM Role, fetches the real password from AWS, and creates a native Kubernetes Secret purely in memory. Result: The entire cluster is defined in Git, but zero passwords exist in Git.

Audit trails, and checking changes before they land#

GitOps moves security earlier in the process, because the change exists as a reviewable file before it exists in the cluster.

Every intended change arrives as a pull request, so a security team can run tools like Conftest or Kyverno against the YAML before it is merged. A pull request that asks for root in a container can be blocked automatically, rather than discovered later in a running Pod.

It also gives you an audit trail for free: who changed what, when, and who approved it, is the Git history. That is useful evidence under a framework like SOC 2 — evidence, not compliance, which no single practice confers.

Be careful about one thing this does not do. GitOps does not physically stop somebody with cluster access from running kubectl edit. It makes that change visible as drift, and reverts it if self-healing is on. The control is detection and correction, not prevention — which is why the failure below exists at all.

When it breaks#

Symptom → evidence → hypothesis → test → fix. These are failures of the model rather than of any tool, and they show up whichever reconciler you run.

Git says one thing, the cluster does another, and both are "right"#

Symptom. The repository describes the intended state, the cluster does not match, and nothing reports an error.

Evidence. Whether the reconciler is actually watching that path and that revision — a branch renamed, a directory moved, or an Application pointed at a path that no longer exists reconciles nothing, successfully.

Hypothesis. A reconciler that cannot find manifests has nothing to compare, which is not the same as finding no differences.

Test. Change something trivial in Git and watch whether the cluster follows within the sync interval. This is the only end-to-end proof that the loop is closed.

Fix. Correct the path or revision. Then keep testing the loop deliberately — a reconciler nobody has watched reconcile is in the same category as the untested scan gate.

Someone fixed production by hand#

Symptom. The cluster works, Git does not describe it, and the next sync reverts the fix — or worse, nobody notices until a rebuild loses it.

Evidence. The diff between desired and live state.

Hypothesis. Under GitOps, a manual change is drift by definition, however good the reason.

Test. Ask what would happen if the namespace were deleted and reconciled from scratch. If the answer is "we would lose that change", it exists nowhere durable.

Fix. Put the change in Git, even during an incident — especially during an incident, because that is when it is most likely to be forgotten. Break-glass access should be possible, loud, and followed by a commit.

The secret cannot go in Git, so the loop is broken for secrets#

Symptom. Everything is declarative except the secrets, which someone applies by hand.

Evidence. A namespace that cannot be rebuilt from the repository alone.

Hypothesis. Plaintext secrets in Git are unacceptable, and a manual step outside the loop is also unacceptable — it is the one thing nobody can reproduce under pressure.

Test. Try to rebuild the namespace from Git in a scratch cluster.

Fix. Keep the reference in Git and the value in something the cluster can resolve: an encrypted manifest, or an operator that fetches from a secret store. The rule to preserve is that the repository fully describes the desired state, not that every byte of it is in the repository.

Interview Questions#

Beginner#

Q: If a hacker logs into the Kubernetes cluster and deletes a Deployment, what happens in a GitOps system? A: On its next reconciliation, Argo CD sees that the Deployment is missing from the cluster but still present in Git, and — if the Application has selfHeal enabled — recreates it. Both conditions matter. Without selfHeal it reports OutOfSync and waits for a human. And reconciliation is periodic, so recovery takes as long as the sync interval unless a webhook triggers one sooner. It is a strong control against unauthorised change; it is not an instantaneous one, and an attacker who can reach the cluster can usually also delete the thing again.

Intermediate#

Q: Explain the concept of "Configuration Drift". A: Configuration Drift happens when the actual running state of an environment differs from the documented or desired state (e.g., an engineer manually patches a server in the middle of the night but forgets to update the codebase). GitOps eliminates drift through continuous reconciliation.

Senior#

Q: How do you handle environment promotion (Dev -> Staging -> Prod) in a GitOps architecture? A: You use branching or folder structures. A common pattern is folder-based: namespaces/dev, namespaces/staging, namespaces/prod. When Jenkins tests v2.0 in Dev, it updates the YAML in the dev/ folder. A human engineer then opens a Pull Request to copy that YAML change into the prod/ folder. When the PR is approved and merged, ArgoCD detects the change in the prod/ folder and deploys it.

Principal/Architect#

Q: Explain the architectural trade-offs between "Push-based CI/CD" (e.g., Jenkins running helm upgrade) versus "Pull-based GitOps" (e.g., ArgoCD) regarding network security and IAM blast radius. A: In a Push-based model, the CI/CD server sits outside the cluster (e.g., Jenkins in a management VPC). To deploy, the Kubernetes API server must be reachable from Jenkins, requiring inbound firewall rules. Furthermore, Jenkins must hold highly privileged cluster-admin credentials. If Jenkins is compromised, the attacker owns the cluster. In a Pull-based GitOps model, ArgoCD lives inside the cluster. It polls Git via outbound HTTPS only. The cluster API requires zero inbound firewall ports from the CI system, and the CI system holds zero cluster credentials, drastically reducing the IAM blast radius and shrinking the network attack surface. Contents | Continuous Delivery (ArgoCD) |