Incident: Service-to-Service Calls Fail

The incident#

The API Pods are Running and READY. The database Pods are Running and READY. The API logs show connection failures to the database, and nobody has changed either application.

Find out why they cannot talk.

Reproduce it#

kubectl create namespace incident-03
kubectl apply -n incident-03 -f https://raw.githubusercontent.com/EgyKode/EgyKode-Academy/master/content/labs/fixtures/incident-03.yaml

By hand: deploy two services in one namespace, then apply a default-deny egress NetworkPolicy that does not allow UDP 53 to kube-system.

Your CNI must enforce NetworkPolicy or this will not reproduce.

kind's default CNI (kindnet) accepts policies and silently ignores them, so the calls will simply succeed and you will conclude the lab is broken. Use a cluster with Calico or Cilium:

TerminalCopy command

minikube start --cni=calico
# or kind with disableDefaultCNI: true, then apply Calico

Confirm enforcement is real before trusting any result:

TerminalCopy command

kubectl get pods -n kube-system -l k8s-app=calico-node

This is worth knowing beyond the lab: a NetworkPolicy that is accepted but not enforced is a security control that exists only on paper, and nothing warns you.

Test DNS before anything else#

More than half of "the network is broken" reports in Kubernetes are name resolution.

kubectl -n incident-03 exec deploy/api -- nslookup db
kubectl -n incident-03 exec deploy/api -- nslookup db.incident-03.svc.cluster.local
kubectl -n incident-03 exec deploy/api -- cat /etc/resolv.conf

Three distinct outcomes, three different faults:

That third case is the one that surprises people, because everything looks healthy.

Is CoreDNS itself alright?#

kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=20

If the name resolves, test the connection#

kubectl -n incident-03 exec deploy/api -- wget -qO- --timeout=5 http://db:5432 || echo "failed"
kubectl -n incident-03 get endpoints db

Separate the two failures deliberately:

• Refused — something answered and said no. Routing and policy are fine; look at the service.
• Timed out — nothing answered. Suspect a NetworkPolicy, or a Service with no endpoints.

NetworkPolicies#

kubectl -n incident-03 get networkpolicy
kubectl -n incident-03 describe networkpolicy

Two properties cause most of these incidents:

• A Pod selected by no policy is unrestricted. Security starts only when something selects it — which is why a default-deny is usually the first policy written, and why adding one breaks things that used to work.
• Policies are additive with no deny rule. Traffic is allowed if any policy allows it, so you cannot fix this by adding a deny — you widen an allow.

The classic mistake: a default-deny egress policy that permits traffic to the database but forgets UDP 53 to CoreDNS. Every hostname lookup in the namespace then times out, while the policy looks correct because the database rule is right there.

The rule it needs, and note that it names both protocols — DNS falls back to TCP for responses too large for one datagram, so a UDP-only rule leaves a few lookups hanging while most succeed:

  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Before you change anything#

"The call fails because ___, which I proved by ___."

Check your reasoning#

Read this after you have fixed it, or after a genuine attempt. Being handed the answer costs you the only thing this tier teaches.

Root cause: a default-deny egress NetworkPolicy allows traffic to the database but not to CoreDNS. The API cannot resolve db, so it never opens a connection at all — the database rule is correct and never gets used.

The command that found it: nslookup db from inside the calling Pod timed out rather than returning NXDOMAIN. A timeout points at reachability of the resolver, not at a missing record.

The fix: add an egress rule permitting UDP 53 to the kube-system namespace.

Why everything looked healthy: the policy does exactly what it says, and both workloads are genuinely fine. Nothing reports an error, because from Kubernetes' point of view nothing is wrong.

Clean up#

kubectl delete namespace incident-cl