Hands-On Labs

Why labs, and why these ones#

Reading about a rolling update teaches you the vocabulary. Watching one stall because you gave it a broken image teaches you the system.

Every lab here follows the same shape:

• Goal — one sentence
• Do — exact commands
• Observe — what you should see, and why
• Verify — a check that either passes or fails
• Break it — the failure mode, deliberately triggered

The Break it step is the point. Anyone can follow a happy path. Interviews, and 3am pages, are about the other one.

Cost note: these run against dev (~$180/month). Run terraform destroy when you finish a session.

Lab 1 · Container fundamentals#

~45 min · no AWS spend · Docker

Goal#

Prove you understand what a multi-stage build actually removes.

Do#

cd CloudDevOpsProject/src/roadmap-service
docker build -t roadmap-service:multi .
 
# Now build a deliberately bad single-stage version
cat > /tmp/Dockerfile.bad <<'EOF'
FROM maven:3.9.11-eclipse-temurin-21
WORKDIR /app
COPY . .
RUN mvn -B clean package -DskipTests
CMD ["java", "-jar", "target/roadmap-service-1.0.0.jar"]
EOF
 
docker build -f /tmp/Dockerfile.bad -t roadmap-service:single .
docker image ls | grep roadmap-service

Observe#

The single-stage image is roughly 3× larger. Find out what is in it:

docker run --rm roadmap-service:single  which mvn git   # present
docker run --rm roadmap-service:multi   which mvn git   # absent
docker run --rm roadmap-service:single  id              # uid=0(root)
docker run --rm roadmap-service:multi   id              # uid=10001

The single-stage image ships Maven, Git, a JDK and your source code into production. Every one of those is attack surface that does nothing at runtime.

Verify#

docker run --rm roadmap-service:multi id | grep -q 'uid=0' && echo FAIL || echo PASS

Break it#

Delete the USER ivolve line and rebuild. Keep that image — Lab 5 uses it to show the restricted Pod Security Standard rejecting it at admission.

Lab 2 · Terraform: read a plan properly#

~40 min · no spend if you stop before apply · Terraform

Goal#

Learn to spot a destructive change before it destroys something.

Do#

cd CloudDevOpsProject/02-Terraform
terraform init
terraform plan -out=tfplan
terraform show -json tfplan | jq -r '
  .resource_changes[] | select(.change.actions[0] != "no-op") |
  "\(.change.actions|join(","))  \(.address)"' | sort | head -30

Observe#

Answer these from the plan, not from the code:

• How many resources will be created?
• What instance_types is the EKS node group configured to use?
• How many NAT gateways? (This is the cost decision.)

Break it — the important half#

Change something immutable and see what Terraform proposes:

# Edit terraform.tfvars, change the project_name, then:
terraform plan | grep -E '^\s+#.*must be replaced' | head

You should see must be replaced on the VPC or EKS cluster depending on the change. In production that is your data. -/+ means destroy-then-create — this is exactly the diff people miss by skimming a plan.

Revert the change before applying.

Verify#

You can state, without running anything, which resources in this plan are destroy-and-recreate rather than update-in-place.

Lab 3 · Observe the EKS control plane abstraction#

~20 min · Kubernetes, Cluster Administration

Goal#

Experience what a managed service actually abstracts away from you.

Do#

Build the EKS infrastructure using Terraform:

cd CloudDevOpsProject
make tf-apply

While it builds (it takes ~15 minutes), read 02-Terraform/modules/eks/main.tf.

Observe#

Notice what you didn't do. You didn't configure etcd, you didn't disable swap on worker nodes, you didn't generate TLS certificates for the API server, and you didn't write an Ansible playbook to run kubeadm init. EKS handled the entire control plane.

Verify#

When Terraform finishes, connect to the cluster:

aws eks update-kubeconfig --region us-east-1 --name ivolve-dev-eks
kubectl get nodes

The nodes are immediately Ready because the AWS VPC CNI was pre-installed by the managed service.

Break it#

What happens if you accidentally delete a node?

# Terminate an EC2 worker node directly from AWS
aws autoscaling terminate-instance-in-auto-scaling-group \
  --instance-id <worker-instance-id> --no-should-decrement-desired-capacity
 
watch kubectl get nodes

The EKS Managed Node Group detects the failure and automatically spins up a replacement EC2 instance to maintain the desired capacity. You did not have to run kubeadm join on the new server.

Lab 4 · Make a rolling update fail safely#

~45 min · Kubernetes · the single most valuable lab here

Goal#

Prove that a bad deploy cannot take down the service.

Do#

kubectl -n ivolve get pods -l app.kubernetes.io/name=roadmap-service -w   # leave this running in terminal 2

Terminal 1 — deploy an image that does not exist:

kubectl -n ivolve set image deploy/roadmap-service roadmap-service=roadmap-service:does-not-exist
kubectl -n ivolve rollout status deploy/roadmap-service --timeout=60s

Observe#

The rollout stalls. It does not fail catastrophically:

kubectl -n ivolve get rs          # two ReplicaSets: old at 3, new at 1
kubectl -n ivolve get endpointslices -l kubernetes.io/service-name=roadmap-service
curl -sf https://dev.<your-domain>/api/v1/status   # still works

This is maxUnavailable: 0 doing its job. The new pod cannot become ready, so no old pod is removed. Users see nothing.

Now try it with a working image but a broken readiness probe:

kubectl -n ivolve rollout undo deploy/roadmap-service
kubectl -n ivolve patch deploy roadmap-service --type=json \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/path","value":"/nope"}]'
kubectl -n ivolve get endpointslices -l kubernetes.io/service-name=roadmap-service -w

Watch the endpoint list empty as pods fail readiness. That is Kubernetes draining traffic from pods that say they cannot serve.

Verify#

kubectl -n ivolve rollout undo deploy/roadmap-service
kubectl -n ivolve rollout status deploy/roadmap-service

Understand before moving on#

Why did the first failure leave users unaffected while the second one emptied the endpoints? What is different about the two failure modes?

Lab 5 · Security controls, tested not assumed#

~60 min · Kubernetes Security, Network Policies

Goal#

Confirm each control actually blocks what it claims to.

Do — Pod Security Standards#

# The root image from Lab 1
kubectl -n ivolve run rooty --image=busybox:1.36 --restart=Never

It is rejected at admission with a message naming the violated policy. Not reported later — refused.

Do — NetworkPolicy#

# A pod that is not the storefront or the API tries to reach the database
kubectl -n ivolve run intruder --rm -it --restart=Never \
  --image=busybox:1.36 -- sh -c 'nc -zv -w5 mysql-0.mysql.ivolve.svc.cluster.local 3306'

It hangs and times out. Compare with the legitimate path:

kubectl -n ivolve exec deploy/frontend -- nc -zv -w5 mysql-0.mysql.ivolve.svc.cluster.local 3306

Note the difference between "timed out" and "connection refused." Timed out means a firewall silently dropped it — a NetworkPolicy working correctly.

Do — metadata endpoint#

kubectl -n ivolve exec deploy/roadmap-service -- \
  timeout 5 wget -qO- http://169.254.169.254/latest/meta-data/ || echo "BLOCKED"

This is the SSRF-to-credential-theft path. It should fail.

Do — RBAC#

kubectl auth can-i --list --as=system:serviceaccount:ivolve:ci-verifier -n ivolve
kubectl auth can-i delete deployments --as=system:serviceaccount:ivolve:ci-verifier -n ivolve
# → no

Verify#

All four controls block. If any succeeds, that control is not working — find out why before continuing.

Lab 6 · The full pipeline, including its failures#

~90 min · Continuous Integration, DevSecOps

Goal#

See each gate fire. A gate you have never seen fire is a gate you do not trust.

Do — the happy path first#

git switch -c lab/pipeline-test
# make a trivial change to the API
git commit -am "lab: trivial change" && git push -u origin lab/pipeline-test

Watch every stage in Jenkins. Note how long each takes.

Break it — three ways, one at a time#

1. Syntax error

// in src/roadmap-service/src/main/java/com/ivolve/roadmapservice/RoadmapServiceApplication.java
// Delete a semicolon at the end of a line

Push. The Jenkins pipeline stops at the Build stage. Nothing is pushed.

2. Vulnerable dependency

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.14.1</version>   <!-- Log4Shell -->
</dependency>

Push. Trivy stops it. Nothing is built. Read the report — it names the CVE and the fixed version.

3. Coverage drop

Delete most of the test class. Push. The SonarQube quality gate aborts the pipeline.

Verify#

git switch main && git branch -D lab/pipeline-test

You have watched three independent gates fail closed. Each one prevented a different category of bad change from reaching a registry.

Lab 7 · GitOps, and the moment it clicks#

~45 min · The GitOps Philosophy, Continuous Delivery

Goal#

Experience selfHeal reverting you.

Do#

kubectl -n ivolve get deploy roadmap-service    # note the replicas
kubectl -n ivolve scale deploy roadmap-service --replicas=7
kubectl -n ivolve get deploy roadmap-service    # 7
 
# Wait up to 3 minutes
watch kubectl -n ivolve get deploy roadmap-service

Observe#

It goes back. You changed production and the platform undid it, because git said otherwise.

Watch ArgoCD notice:

kubectl -n argocd logs deploy/argocd-application-controller --tail=30 | grep -i sync
argocd app diff ivolve-app

Then do it the right way#

$EDITOR 04-Kubernetes/manifests/06-roadmap-service.yaml   # set replicas: 3
git commit -am "chore: scale api to 3 in dev" && git push
watch kubectl -n ivolve get deploy roadmap-service

Same outcome. Completely different property: this one has an author, a diff, a review and a revert.

Verify#

git log --oneline -3 -- 04-Kubernetes/manifests/

That output is your deployment history.

Lab 8 · Break the cluster and recover it#

~90 min · Cluster Administration, Disaster Recovery · do this one last

Goal#

Survive failures you have caused deliberately, so the real ones are familiar.

Experiment 1 — Kill the database#

Hypothesis: The StatefulSet controller brings it back, and the EBS CSI driver automatically reattaches the exact same volume to the new Pod. No data is lost.

# Insert a test row first
kubectl -n ivolve exec -it mysql-0 -- mysql -u root -p ivolve -e "CREATE TABLE test (id INT); INSERT INTO test VALUES (1);"
 
# Murder the pod
kubectl -n ivolve delete pod mysql-0 --force --grace-period=0
 
watch kubectl -n ivolve get pods

When it returns to Running, check if your data survived:

kubectl -n ivolve exec -it mysql-0 -- mysql -u root -p ivolve -e "SELECT * FROM test;"

Experiment 2 — Break the GitOps loop#

ArgoCD enforces the desired state. What happens if you delete ArgoCD itself?

kubectl delete namespace argocd

The GitOps loop is now dead. If someone makes a manual change to the cluster with kubectl edit, it will no longer be reverted.

To recover, simply re-apply the installation manifest:

cd CloudDevOpsProject
make argo-install
make argo-apply

ArgoCD spins back up, reads its configuration from the cluster, immediately notices any drift that happened while it was offline, and syncs everything back to the Git state.

Verify#

./scripts/health-check.sh dev    # exits 0

Write it up#

For each experiment: hypothesis, what actually happened, what surprised you. That document is worth more in an interview than any certification.

Capstone#

If you haven't built the Cloud DevOps Capstone yet, now is the time.

Follow the START-HERE.md guide in the repository to build the entire production system end-to-end:

1. make tf-apply (Terraform EKS)
2. make ansible-run (Jenkins build host)
3. make k8s-apply and make argo-apply (GitOps Workloads)

When you are done, tear it all down with a single command to prove it's automated:

cd CloudDevOpsProject
make tf-destroy

If you can destroy and rebuild it reliably, you have automated the platform. If you cannot, you have found a manual step you forgot you performed — which is exactly what bites a team at 2am.

Before you destroy anything for the last time: take the screenshots. See screenshots/README.md. Once the environment is gone, they are gone.

Progress tracker#