Linux Networking & Troubleshooting
Work a connection failure from the outside in: DNS, route, port, firewall, application — and know which layer you are on.
- Time
- 50 min
- Level
- Beginner
- Objectives
- 4 objectives
- Cost
- Free
Where this fits in the platform
Already built
This lab adds
- A method for following a connection to the layer that broke it
Which lets you
Before you start
You will need
- Linux with `dig`, `ss`, `curl`, `ip`
- sudo access
You do not need these already — the lab environment below provides them.
You will be able to
- Resolve a name and read the TTL that explains a stale answer
- Read a routing table and say where a packet would go
- Prove whether a port is open, listening, or filtered
- Work a failure in layers instead of guessing
Cost — Free
— no cloud resources
Nothing to pay in the browser. Open the terminal runs this against a simulated cloud — the same API calls and the same commands, with no account and no bill. The figure above applies only if you build it in your own.
The scenario#
An application cannot reach its database. The developer says "the network is down". It almost never is.
This lab builds the sequence that finds the real cause in under two minutes, instead of restarting things until something changes.
Hands-on environment
Run this lab in a real terminal, free and in your browser. The environment is temporary and yours alone — break it as much as you like.
Open the terminalOpens in Killercoda, in a new tab — keep this page open for the steps.
Run it on your own machine
Run this lab on your own machine. One command starts the environment, with everything the lab needs already installed:
You will need:
- bash
- curl
git clone https://github.com/EgyKode/EgyKode-lab.git
cd EgyKode-lab
./egykode start
./egykode shellYou need Docker and Git installed. Everything else runs inside the environment. The first start downloads it and takes a few minutes; later starts are seconds.
Not sure what you already have? Run: npm run doctor — it checks and changes nothing.
Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.
Does the name resolve?
Step 1 of 4
The order matters#
Work outside-in. Each step rules out a layer, so you never guess:
name resolves? -> route exists? -> port open? -> app answers?
dig ip route ss / curl logsWhat you are proving: You can state the address a name resolves to, and how long that answer is valid
Marking this settles success criterion 1.
dig +short db.internal.example.com
dig db.internal.example.com | grep -A2 "ANSWER SECTION"The number before the record type is the remaining TTL. If the value is
wrong and the TTL is large, you are looking at a cached answer, not at your
configuration — and no amount of restarting will fix it. dig @8.8.8.8 <name>
asks a resolver that has no local cache, which tells you whether the problem is
yours or upstream.
What you are proving: You can name the interface and gateway a packet would leave by
Marking this settles success criterion 2.
ip route
ip route get 10.20.5.10ip route get is the direct answer: it names the interface and gateway the
kernel would use for that exact destination. If it says the wrong interface, the
problem is routing, and nothing downstream is worth checking yet.
What you are proving: You can tell a refused connection from a timed-out one, and say what each implies
Marking this settles success criterion 3.
Locally:
ss -ltnp | grep 5432ss -ltnp — listening, TCP, numeric, with the process. If nothing is listening,
the application is not running, and the network was never involved.
Remotely:
curl -v --max-time 5 telnet://db.internal.example.com:5432The two failures mean different things, and the distinction is the whole point:
| Result | Meaning | Look at |
|---|---|---|
Connection refused | Something answered and said no | The service — it is down or bound to 127.0.0.1 |
Connection timed out | Nothing answered at all | A firewall or security group silently dropping it |
| Connects, then hangs | Reached it; the app is not replying | Application logs, slow queries |
A refused connection is good news: routing and firewalls are fine, and the problem is a process you control.
What you are proving: You can name the failing layer before changing anything
Marking this settles success criterion 4.
The most common false alarm:
ss -ltnp | grep 5432
# LISTEN 0 244 127.0.0.1:5432 <- only localhost
# LISTEN 0 244 0.0.0.0:5432 <- every interfaceA service bound to 127.0.0.1 works perfectly from the machine itself and is
unreachable from anywhere else. curl from the server succeeds, the developer
says "it works here", and the connection still fails from the app.
The failure is where the learning is. These are the ones that actually happen:
dig returns the old IP after a DNS change
The record is cached for its TTL. Check the TTL in the answer, and query @8.8.8.8 to compare with an uncached resolver.
curl times out but the security group looks correct
Check the outbound rules on the source and the NACL on the subnet — a NACL is stateless and needs the return path allowed explicitly.
Works from the server, fails from anywhere else
The service is bound to 127.0.0.1. Look at the ss -ltnp output, not at the firewall.
ss shows nothing on the port
The process is not running. This is not a network problem — check the service and its logs.
Success criteria
0 of 4
The concept behind it
Next up
Lab 4 of 59 on the project path