01 · FoundationsLab 2 / 59
Guided lablinux
Linux Processes, Services & Logs
Find the process, read what it actually said, and restore a service that will not start.
- Time
- 45 min
- Level
- Beginner
- Objectives
- 4 objectives
- Cost
- Free
Where this fits in the platform
Already built
This lab adds
- The ability to find why a service died
Which lets you
Before you start
You will need
- Linux with systemd
- sudo access
You do not need these already — the lab environment below provides them.
You will be able to
- Locate a process by port, name or open file
- Read a service's own logs rather than guessing from its status
- Tell a crash apart from a configuration error
Cost — Free
— any Linux machine, VM or container.
Nothing to pay in the browser. Open the terminal runs this against a simulated cloud — the same API calls and the same commands, with no account and no bill. The figure above applies only if you build it in your own.
The scenario#
A service is down. systemctl status says failed, which tells you that it failed and nothing about why.
This lab builds the sequence that gets from that word to the cause.
Hands-on environment
Run this lab in a real terminal, free and in your browser. The environment is temporary and yours alone — break it as much as you like.
Open the terminalOpens in Killercoda, in a new tab — keep this page open for the steps.
Run it on your own machine
Run this lab on your own machine. One command starts the environment, with everything the lab needs already installed:
Run these against the node1 container rather than the controller — it is the machine with systemd. From the controller: `ssh node1`.
You will need:
- bash
git clone https://github.com/EgyKode/EgyKode-lab.git
cd EgyKode-lab
./egykode start
./egykode shell
ssh node1You need Docker and Git installed. Everything else runs inside the environment. The first start downloads it and takes a few minutes; later starts are seconds.
Not sure what you already have? Run: npm run doctor — it checks and changes nothing.
Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.
What is running, and what holds the port
Step 1 of 5
What you are proving: You can name the process holding a given port, and the command that told you
Marking this settles success criterion 1.
Terminal
ps aux --sort=-%mem | head -10 # heaviest processes first
sudo ss -ltnp | grep ':8080' # who is listening
sudo lsof -i :8080 # the same, with more detail
pgrep -a nginx # PIDs by name, with their command liness -ltnp is the one to memorise: listening, tcp, numeric, with the
process. If nothing is listening, the service is not running and the network
was never involved.
What you are proving: You can read a service's own logs, and tell failed, inactive and activating apart
Marking this settles success criteria 2 and 4.
Terminal
systemctl status nginx # what systemd thinks happened
journalctl -u nginx -n 50 --no-pager # what the application said
journalctl -u nginx -f # follow it live
journalctl -u nginx --since "10 min ago" -p errstatus gives you the exit code and the last few lines. journalctl -u gives
you everything the unit wrote. The second is where the cause usually is, and it
is the step people skip.
The states mean different things:
| State | Means |
|---|---|
active (running) | Working |
inactive (dead) | Stopped, and nothing tried to start it |
failed | It tried and exited non-zero — read the logs |
activating | Still starting, or stuck in a start loop |
What you are proving: You can diagnose a service that refuses to start from its logs, and restore it
Marking this settles success criterion 3.
Terminal
sudo sed -i 's/^user /usr /' /etc/nginx/nginx.conf # a deliberate typo
sudo systemctl restart nginx
systemctl status nginxNow work it properly:
Terminal
journalctl -u nginx -n 20 --no-pager
sudo nginx -t # most services have a config testnginx -t names the file and line. Many daemons have an equivalent — sshd -t,
apachectl configtest, postgres --check. Reach for it before restarting
anything, because a service that fails to start on a bad config will keep
failing no matter how many times you restart it.
Terminal
sudo sed -i 's/^usr /user /' /etc/nginx/nginx.conf
sudo systemctl restart nginx && systemctl is-active nginxWhat you are proving: You can choose between reload and restart knowing which one drops live connections
This step settles no success criterion on its own.
Terminal
sudo systemctl reload nginx # SIGHUP — re-read config, keep connections
sudo systemctl restart nginx # stop then start — drops connections
kill -TERM <pid> # ask politely
kill -9 <pid> # last resort; no cleanup happensreload and restart are not interchangeable. On a busy server restart drops
every in-flight request; reload re-reads the configuration without doing so —
if the service supports it.
What you are proving: You can explain why a service's journal is empty and find where its output actually went
This step settles no success criterion on its own.
Terminal
journalctl -u myapp --since today | wc -l
sudo journalctl --disk-usage
systemctl cat myapp | grep -E 'StandardOutput|StandardError'A unit with StandardOutput=null writes nothing to the journal, and its output
is wherever the application was told to put it. systemctl cat shows the unit
as systemd actually sees it, including drop-ins you did not know existed.
systemctl status shows failed with no useful output
journalctl -u <unit> -n 50 — status truncates. If that is empty too, check StandardOutput in systemctl cat.
The service restarts in a loop
Restart=always with a config error. Fix the config; the loop is systemd doing what it was told.
Port already in use
sudo ss -ltnp | grep <port> names the holder. Often an old instance that did not exit.
Changes to the unit file do nothing
sudo systemctl daemon-reload after editing a unit, then restart it.
Maintained by others, on Killercoda. Useful for extra repetition on one tool — it does not complete this lab or settle any criterion above.
Success criteria
0 of 4
The concept behind it
Next up
Lab 2 of 59 on the project path