11 · Operating itLab 56 / 59
ChallengesreDestructive
Chaos: Failure Injection & Recovery — Challenge
Kill things deliberately, measure how long recovery takes, and find the assumption that was wrong.
- Time
- 25 min
- Level
- Advanced
- Objectives
- 4 objectives
- Cost
- Free
Where this fits in the platform
Already built
This lab adds
- Evidence of how the system behaves when you break it on purpose
Which lets you
—
Before you start
You will need
- kind or minikube
- kubectl 1.28+
You do not need these already — the lab environment below provides them.
You will be able to
- Form a hypothesis before injecting a failure
- Measure recovery rather than observing it
- Recognise a self-healing gap that only appears under failure
The goal#
Achieve the same outcome as Chaos: Failure Injection & Recovery, from an empty starting point, without the steps.
The architecture diagram says the system is highly available. Nobody has tested it.
Chaos engineering is not breaking things at random — it is stating what you believe will happen, then checking.
Hands-on environment
Run this lab in a real terminal, free and in your browser. The environment is temporary and yours alone — break it as much as you like.
Start the challengeOpens in Killercoda, in a new tab — keep this page open for the steps.
This is the guided lab's environment — the same machine, with its walkthrough on the left. Work from the task above and leave those steps alone until you are done, or you are reading the answers.
Run it on your own machine
Run this lab on your own machine. One command starts the environment, with everything the lab needs already installed:
You will need:
- docker
git clone https://github.com/EgyKode/EgyKode-lab.git
cd EgyKode-lab
./egykode start
./egykode shellYou need Docker and Git installed. Everything else runs inside the environment. The first start downloads it and takes a few minutes; later starts are seconds.
Not sure what you already have? Run: npm run doctor — it checks and changes nothing.
Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.
What must be true when you are done
Step 1 of 3
What must be true when you are done#
- A killed Pod is replaced automatically, and you measured how long it took.
- You found at least one case where recovery did not happen as expected.
- A PodDisruptionBudget prevents an eviction that would have caused an outage.
- Each experiment had a written hypothesis before it was run.
Rules#
- Do not open the guided lab until you are finished, or until the same problem has held you up for 20 minutes.
- Documentation is allowed and encouraged.
- Verify every criterion with a command whose output you can read.
If you get stuck#
- What did you expect, exactly?
- What happened instead — the error text, not a paraphrase?
- Which layer is that error from?
- What is the smallest command that proves the layer below is fine?
You are done when
0 of 4
The concept behind it
Next up
Lab 56 of 59 on the project path