11 · Operating itLab 52 / 59
Incidentkubernetes
Incident: CrashLoopBackOff
A container starts, dies, and restarts forever. The current logs are empty. Find out what the one that died said.
- Time
- 35 min
- Level
- Advanced
- Objectives
- 3 objectives
- Cost
- Free
Where this fits in the platform
Already built
This lab adds
- A pod diagnosed from its own evidence
Which lets you
—
Before you start
You will need
- kind or minikube
- kubectl 1.28+
You do not need these already — the lab environment below provides them.
You will be able to
- Read the logs of a container that has already exited
- Map an exit code to a cause
- Separate a crash from a failing health check
Cost — Free
— runs on kind or minikube.
Nothing to pay in the browser. Open the terminal runs this against a simulated cloud — the same API calls and the same commands, with no account and no bill. The figure above applies only if you build it in your own.
The incident#
A Pod has restarted 14 times in six minutes and sits in CrashLoopBackOff. kubectl logs prints nothing.
Find out why, and fix it.
Hands-on environment
Run this lab in a real terminal, free and in your browser. The environment is temporary and yours alone — break it as much as you like.
Start the challengeOpens in Killercoda, in a new tab — keep this page open for the steps.
Run it on your own machine
Run this lab on your own machine. One command starts the environment, with everything the lab needs already installed:
You will need:
- docker
- kubectl
- kind
git clone https://github.com/EgyKode/EgyKode-lab.git
cd EgyKode-lab
./egykode start k8s
./egykode shellYou need Docker and Git installed. Everything else runs inside the environment. The first start downloads it and takes a few minutes; later starts are seconds.
Not sure what you already have? Run: npm run doctor — it checks and changes nothing.
Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.
Reproduce it
Step 1 of 9
Reproduce it#
Terminal
kubectl create namespace incident-02
kubectl apply -n incident-02 -f https://raw.githubusercontent.com/EgyKode/EgyKode-Academy/master/content/labs/fixtures/incident-02.yamlBy hand: deploy any image whose entrypoint reads a required environment variable that is not set, and let it exit non-zero.
What CrashLoopBackOff actually means#
It is not an error in itself. It means the container exited, Kubernetes restarted it, it exited again, and the kubelet is now backing off — waiting longer between attempts (10s, 20s, 40s…) so a broken container cannot spin the node.
The useful information is in the container that already died.
The one command people miss#
Terminal
kubectl -n incident-02 logs <pod> # the container that just started
kubectl -n incident-02 logs <pod> --previous # the one that died--previous is the whole lab. The current container has been alive for two
seconds and knows nothing; the evidence is in its predecessor.
Then the exit code#
Terminal
kubectl -n incident-02 describe pod <pod>Look at Last State: Terminated and its Exit Code:
| Exit code | Means | Look at |
|---|---|---|
0 | Exited cleanly | A one-shot command in a Deployment — it needs to be a Job |
1 | Application error | The previous logs |
2 | Shell misuse | The command or args in the manifest |
126 / 127 | Not executable / not found | The entrypoint path |
137 | SIGKILL — almost always OOMKilled | Memory limit versus what it needs |
143 | SIGTERM — asked to stop | Something else is terminating it |
137 is the one worth recognising instantly: the container exceeded its memory
limit and the kernel's OOM killer terminated it. describe says OOMKilled
explicitly in Last State.
This is the cgroup limit you met in Linux and set with --memory in
Docker — Kubernetes declares it in resources.limits, and the kernel enforces
it. Same mechanism, three names. Which is why the evidence is also on the node:
dmesg -T | grep -i oom shows the kill from the kernel's side, and it is worth
looking at once so the chain stops being abstract.
Rule out the probes#
A container that is running fine but failing its liveness probe restarts in a loop that looks identical from the outside:
Terminal
kubectl -n incident-02 describe pod <pod> | grep -A5 Liveness
kubectl -n incident-02 get events --sort-by=.lastTimestamp | tail -20An event saying Liveness probe failed means the application is alive and the
check is wrong — often a slow-starting process with no startupProbe, being
killed before it ever finishes booting.
Before you change anything#
Finish this sentence: "The container exits because ___, which I know from ___."
The second half matters as much as the first.
The candidates#
- A required environment variable or config file is missing, so the process exits at startup.
- A referenced ConfigMap or Secret does not exist — the Pod never starts and
describesays so in Events. - The memory limit is below what the process needs — exit
137,OOMKilled. - The command or args are wrong — exit
127. - It is a one-shot task in a Deployment: it succeeds, exits
0, and gets restarted because a Deployment expects a long-running process. - A liveness probe is killing a healthy but slow-starting container.
--previous says there is no previous container
The Pod has not restarted yet, or was just recreated. Wait for one more restart, or check kubectl get events.
Restart count climbs but the logs look normal
Suspect a liveness probe rather than a crash. describe will show Liveness probe failed in Events.
The Pod never starts at all
That is not CrashLoopBackOff — it is CreateContainerConfigError or ImagePullBackOff. describe names which.
Check your reasoning#
Read this after you have fixed it, or after a genuine attempt. Being handed the answer costs you the only thing this tier teaches.
Root cause: the container requires an environment variable that the manifest never sets. It exits
1at startup, before writing anything to the log the current container would show.The command that found it:
kubectl logs <pod> --previous— the dead container's output names the missing variable directly.The fix: add the variable via
envorenvFrom. If it belongs in a ConfigMap, note that editing that ConfigMap later will not restart the Pod —envFromvalues are read once at container start.Why
kubectl logsalone showed nothing: by the time you ran it, the kubelet had already started a replacement that had not reached the failure yet, or had produced no output at all.
Clean up#
Destructive — This removes real resources. Check which environment you are in first.
Terminal
kubectl delete namespace incident-crSuccess criteria
0 of 3
The concept behind it
Next up
Lab 52 of 59 on the project path