11 · Operating itLab 56 / 59
Guided labsreDestructive
Chaos: Failure Injection & Recovery
Kill things deliberately, measure how long recovery takes, and find the assumption that was wrong.
- Time
- 50 min
- Level
- Advanced
- Objectives
- 4 objectives
- Cost
- Free
Where this fits in the platform
Already built
This lab adds
- Evidence of how the system behaves when you break it on purpose
Which lets you
—
Before you start
You will need
- kind or minikube
- kubectl 1.28+
You do not need these already — the lab environment below provides them.
You will be able to
- Form a hypothesis before injecting a failure
- Measure recovery rather than observing it
- Recognise a self-healing gap that only appears under failure
Cost — Free
— a local Kubernetes cluster.
Nothing to pay in the browser. Open the terminal runs this against a simulated cloud — the same API calls and the same commands, with no account and no bill. The figure above applies only if you build it in your own.
The scenario#
The architecture diagram says the system is highly available. Nobody has tested it.
Chaos engineering is not breaking things at random — it is stating what you believe will happen, then checking.
This lab deletes running workloads. Use a throwaway cluster.
Hands-on environment
Run this lab in a real terminal, free and in your browser. The environment is temporary and yours alone — break it as much as you like.
Open the terminalOpens in Killercoda, in a new tab — keep this page open for the steps.
Run it on your own machine
Run this lab on your own machine. One command starts the environment, with everything the lab needs already installed:
You will need:
- docker
git clone https://github.com/EgyKode/EgyKode-lab.git
cd EgyKode-lab
./egykode start
./egykode shellYou need Docker and Git installed. Everything else runs inside the environment. The first start downloads it and takes a few minutes; later starts are seconds.
Not sure what you already have? Run: npm run doctor — it checks and changes nothing.
Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.
Something to break
Step 1 of 6
The method#
Every experiment has four parts, and the first is the one people skip:
- Hypothesis — "killing one of three replicas causes no failed requests."
- Blast radius — one namespace, one Deployment, and a way to stop.
- Inject — the smallest failure that tests the hypothesis.
- Measure — was the hypothesis right? If yes, make it harsher.
An experiment without a hypothesis is just an outage you caused.
What you are proving: You can put a workload under steady traffic, so a failure is something you observe rather than infer
This step settles no success criterion on its own.
Terminal
kubectl create deployment web --image=nginx:1.27-alpine --replicas=3
kubectl expose deployment web --port=80
kubectl set resources deployment web --requests=cpu=10m,memory=16MiGenerate steady traffic in a second terminal:
Terminal
kubectl run load --rm -it --image=curlimages/curl --restart=Never -- \
sh -c 'while true; do curl -s -o /dev/null -w "%{http_code} " http://web; sleep 0.2; done'What you are proving: You can kill a Pod, measure how long the replacement took, and count the requests that failed
Marking this settles success criterion 1.
Hypothesis: no failed requests; a replacement is Ready within 30 seconds.
Destructive — This removes real resources. Check which environment you are in first.
Terminal
time kubectl delete pod -l app=web --field-selector=status.phase=Running --wait=false | head -1
kubectl get pods -wWatch the traffic terminal. Count non-200 responses.
If you saw failures, the hypothesis was wrong, and that is the useful outcome:
the Service kept the dying Pod in its endpoints until it was fully terminated.
The fix is a preStop hook and a readiness probe that fails first.
What you are proving: You can measure the real recovery time for a bad deploy, on purpose and in daylight
Marking this settles success criterion 2.
Hypothesis: requests fail until Pods return, and recovery is automatic.
Terminal
kubectl scale deployment web --replicas=0
sleep 10
kubectl scale deployment web --replicas=3Measure the gap. This is a controlled version of a bad deploy, and the number you get is your real recovery time for one.
What you are proving: You can drain a node and discover where your replicas actually were
This step settles no success criterion on its own.
Hypothesis: draining a node moves the Pods without downtime.
Terminal
kubectl get nodes
kubectl drain <node> --ignore-daemonsets --delete-emptydir-dataOn a single-node cluster everything becomes Pending — which is the finding:
there is nowhere to reschedule. On multi-node, watch whether all replicas were on
the same node, which quietly defeats the point of having three.
Terminal
kubectl uncordon <node>What you are proving: You can write a PodDisruptionBudget that refuses an eviction which would cause an outage
Marking this settles success criterion 3.
yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web
spec:
minAvailable: 2
selector:
matchLabels:
app: webTerminal
kubectl apply -f pdb.yaml
kubectl drain <node> --ignore-daemonsets
# evicting pod web-...
# error when evicting pod: Cannot evict pod as it would violate the disruption budgetA PDB does not stop a crash. It stops voluntary disruption — drains, upgrades, autoscaler scale-downs — which is precisely the category that causes self-inflicted outages during maintenance.
Spreading matters too:
yaml
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: webThree replicas on one node is one node failure away from zero.
What you are proving: You can write a hypothesis before each experiment, and treat the ones it disproves as the result
Marking this settles success criterion 4.
For each experiment: the hypothesis, what happened, and what you changed.
The experiments that disprove a hypothesis are the entire value. An experiment that confirms what you already believed has told you nothing you did not know.
Requests fail when a single Pod is deleted
The Pod stayed in endpoints while terminating. Add a readiness probe and a preStop sleep so it leaves the Service before the process stops.
drain hangs forever
Something cannot be evicted — often a bare Pod with no controller, or a PDB that cannot be satisfied. The message names it.
All replicas are on one node
The scheduler had no reason to spread them. That is what topologySpreadConstraints is for.
The PDB blocks every drain
minAvailable equals the replica count leaves no room for disruption. It must be lower than replicas.
Clean up#
Run this even if you did not finish.
Destructive — This removes real resources. Check which environment you are in first.
Terminal
kubectl delete deployment web --ignore-not-found
kubectl delete svc web --ignore-not-found
kubectl delete pdb web --ignore-not-found
kubectl uncordon $(kubectl get nodes -o name)Cost of this lab: Free — a local Kubernetes cluster.
Success criteria
0 of 4
The concept behind it
Next up
Lab 56 of 59 on the project path