10 · ObservabilityLab 51 / 59

Guided lablogging

Centralised Logging with Loki and Grafana

Ship every Pod's logs somewhere they survive the Pod, then answer a real question with them.

Time: 50 min
Level: Intermediate
Objectives: 4 objectives
Cost: Free

Start the lab Challenge

Where this fits in the platform

Already built

Deploying Kube-Prometheus-Stack on AWS EKS

This lab adds

Logs you can search after the pod that wrote them is gone

Which lets you

—

Before you start

You will need

kind or minikube
Helm 3.14+
kubectl

You do not need these already — the lab environment below provides them.

You will be able to

Query logs by label instead of grepping a stream
Correlate a metric spike with the log lines behind it
Explain why Loki indexes labels rather than content

Cost — Free

on kind or minikube. On a cloud cluster Loki requests a persistent volume, billed per GB-month — see cleanup.

Nothing to pay in the browser. Open the terminal runs this against a simulated cloud — the same API calls and the same commands, with no account and no bill. The figure above applies only if you build it in your own.

How to clean up →

The scenario#

A Pod crashed at 3am. It has been replaced, and kubectl logs shows the new one. The evidence went with the old container.

Metrics told you that something broke. Logs are how you find out why — but only if they left the node before the Pod did.

Hands-on environment

Run this lab in a real terminal, free and in your browser. The environment is temporary and yours alone — break it as much as you like.

Open the terminal

Opens in Killercoda, in a new tab — keep this page open for the steps.

Run it on your own machine

Run this lab on your own machine. One command starts the environment, with everything the lab needs already installed:

You will need:

docker
kubectl
kind

git clone https://github.com/EgyKode/EgyKode-lab.git
cd EgyKode-lab
./egykode start k8s
./egykode shell

You need Docker and Git installed. Everything else runs inside the environment. The first start downloads it and takes a few minutes; later starts are seconds.

Not sure what you already have? Run: npm run doctor — it checks and changes nothing.

Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.

Install the stack

Step 1 of 6

What you are proving: You can install a logging stack and query logs from every namespace in one place

Marking this settles success criterion 1.

Terminal

helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install loki grafana/loki-stack \
  -n monitoring --create-namespace \
  --set grafana.enabled=true \
  --set promtail.enabled=true \
  --wait
 
kubectl port-forward -n monitoring svc/loki-grafana 3000:80

Three parts, and the split matters:

Promtail runs as a DaemonSet — one per node — and tails /var/log/pods, attaching Kubernetes labels to every line.
Loki stores it, indexing only the labels.
Grafana queries it.

What you are proving: You can explain why selecting by label first is what makes the query affordable, and what happens to a deleted Pod's logs with and without Loki

Marking this settles success criterion 4.

Elasticsearch indexes the full text of every line, which is powerful and expensive — the index often exceeds the logs. Loki indexes only labels (namespace, pod, container) and compresses the rest, then brute-force searches within the selected streams.

The practical consequence: always select by label first.

logql

{namespace="production"}                             # fast
{namespace="production", app="api"} |= "ERROR"       # fast, then scan
{} |= "ERROR"                                        # scans everything, slow

That last query is the one people write first and the one that times out.

What you are proving: You can filter to one Deployment's errors over a time window

Marking this settles success criterion 2.

logql

{namespace="production", app="api"} |= "ERROR"
 
{namespace="production"} |= "ERROR" != "healthcheck"
 
{app="api"} | json | status >= 500
 
sum(rate({namespace="production"} |= "ERROR" [5m])) by (app)

That last one turns logs into a metric, so an error rate derived from log lines can be graphed beside Prometheus data and alerted on the same way.

What you are proving: You can cause a crash and find its cause from the logs alone, including the container that already died

Marking this settles success criterion 3.

Terminal

kubectl create deployment crasher --image=busybox -- \
  sh -c 'echo "FATAL: config missing" >&2; exit 1'
kubectl get pods -w      # CrashLoopBackOff

Now compare the two ways of looking:

Terminal

kubectl logs deploy/crasher              # the current container: nothing useful
kubectl logs deploy/crasher --previous   # the one that died: the message

Then in Grafana:

logql

{app="crasher"}

Every restart is there, in order. kubectl logs --previous gives you one container back; Loki gives you all of them, including the ones from before the Pod was rescheduled onto a different node — which is the case --previous cannot reach at all.

What you are proving: You can bound how long logs are kept before the bill decides it for you

This step settles no success criterion on its own.

yaml

loki:
  config:
    limits_config:
      retention_period: 168h        # 7 days
    compactor:
      retention_enabled: true

Logs grow without limit by default. Seven days is a reasonable start: long enough for an investigation, short enough to bound the volume.

Clean up#

Run this even if you did not finish.

Destructive — This removes real resources. Check which environment you are in first.

Terminal

helm uninstall loki -n monitoring
kubectl delete pvc --all -n monitoring   # PVCs survive uninstall
kubectl delete namespace monitoring --ignore-not-found