10 · ObservabilityLab 50 / 59
Guided labprometheus
Custom Prometheus Alert Rules & Grafana Dashboards
Write alerts that fire on conditions worth waking someone for, and a dashboard that shows why they fired.
- Time
- 23 min
- Level
- Advanced
- Objectives
- 4 objectives
- Cost
- Low cost
Where this fits in the platform
Already built
This lab adds
- Alerts a human can act on, and dashboards kept in git
Before you start
Cost — Low cost
Depends on an existing cluster. Alert rules and dashboards are configuration and cost nothing; the Prometheus stack under them holds EBS volumes.
Nothing to pay in the browser. Open the terminal runs this against a simulated cloud — the same API calls and the same commands, with no account and no bill. The figure above applies only if you build it in your own.
The scenario#
Prometheus is collecting metrics nobody looks at. There are no alerts, so problems are found by users; the one dashboard shows CPU, which has never once explained an outage.
Collecting metrics and being able to answer a question with them are different things.
Hands-on environment
Run this lab in a real terminal, free and in your browser. The environment is temporary and yours alone — break it as much as you like.
Open the terminalOpens in Killercoda, in a new tab — keep this page open for the steps.
Run it on your own machine
Run this lab on your own machine. One command starts the environment, with everything the lab needs already installed:
You will need:
- docker
- kubectl
- kind
- helm
git clone https://github.com/EgyKode/EgyKode-lab.git
cd EgyKode-lab
./egykode start k8s
./egykode shellYou need Docker and Git installed. Everything else runs inside the environment. The first start downloads it and takes a few minutes; later starts are seconds.
Not sure what you already have? Run: npm run doctor — it checks and changes nothing.
Run it on AWS
This lab builds real cloud infrastructure, so it needs your own AWS account. Follow the cost and cleanup notes above — the resources are yours, and so is the bill.
Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.
A PrometheusRule
Step 1 of 3
What you are building#
Alerts on symptoms users feel, not on causes you guessed:
text
RED, for a request-driven service
Rate requests per second
Errors the proportion that failed
Duration how long they took (p95, p99)High CPU is not an incident. Users cannot feel CPU. They feel errors and latency — so alert on those, and use CPU to explain them once you are already looking.
Build it#
What you are proving: You can write an alert, cause it to fire, and explain what for does to pager noise
Marking this settles success criteria 1 and 4.
yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: platform-alerts
namespace: monitoring
labels: { release: monitoring } # or the Operator ignores it
spec:
groups:
- name: platform.rules
interval: 30s
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{namespace="platform",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{namespace="platform"}[5m]))
> 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "5xx rate is {{ $value | humanizePercentage }} in platform"
description: >-
More than 5% of requests have failed for 5 minutes.
Current rate {{ $value | humanizePercentage }}.
runbook_url: "https://runbooks.example.com/high-error-rate"
- alert: HighLatency
expr: |
histogram_quantile(0.95,
sum(rate(http_request_duration_seconds_bucket{namespace="platform"}[5m])) by (le)
) > 1
for: 10m
labels: { severity: warning }
annotations:
summary: "p95 latency is {{ $value | humanizeDuration }}"
runbook_url: "https://runbooks.example.com/high-latency"
- alert: PodCrashLooping
expr: |
increase(kube_pod_container_status_restarts_total{namespace="platform"}[15m]) > 3
for: 5m
labels: { severity: critical }
annotations:
summary: "{{ $labels.pod }} restarted {{ $value }} times in 15 minutes"
runbook_url: "https://runbooks.example.com/crashloop"
- alert: PersistentVolumeFillingUp
expr: |
kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.10
for: 15m
labels: { severity: warning }
annotations:
summary: "{{ $labels.persistentvolumeclaim }} is {{ $value | humanizePercentage }} free"for: is the difference between an alert and pager noise. The expression
becomes true, the alert goes Pending, and only if it is still true after
the for: duration does it fire. A five-second blip during a rollout resolves
itself; without for:, it wakes someone up.
A ratio, not a count. sum(rate(...5xx)) > 10 fires on a busy Tuesday and
stays silent during an outage at 3am when traffic is low. The proportion is
what users experience.
Every alert has a runbook_url. Someone woken at 3am should not have to
reconstruct what the alert means. If you cannot write the runbook, the alert is
probably not actionable — which is itself a useful signal.
What you are proving: You can route an alert so it arrives carrying enough context to act on without opening a dashboard
Marking this settles success criterion 2.
yaml
alertmanager:
config:
route:
group_by: ["alertname", "namespace"]
group_wait: 30s # collect related alerts before the first send
group_interval: 5m
repeat_interval: 4h
receiver: default
routes:
- matchers: ['severity="critical"']
receiver: pager
repeat_interval: 1h
receivers:
- name: default
slack_configs:
- channel: "#alerts"
title: '{{ .CommonAnnotations.summary }}'
text: '{{ .CommonAnnotations.description }}\n{{ .CommonAnnotations.runbook_url }}'
- name: pager
pagerduty_configs:
- service_key: "..."
inhibit_rules:
# A node being down explains every Pod on it. Say it once.
- source_matchers: ['alertname="NodeDown"']
target_matchers: ['severity="warning"']
equal: ["node"]group_wait and inhibit_rules are what stop one failure producing forty
notifications. A node failing should page once, not once per Pod.
What you are proving: You can build a dashboard of request rate, error rate and latency, and keep it in Git
Marking this settles success criterion 3.
yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: platform-dashboard
namespace: monitoring
labels:
grafana_dashboard: "1" # the sidecar imports it
data:
platform.json: |
{ "title": "Platform — RED", "panels": [ ... ] }The Grafana sidecar watches for ConfigMaps carrying grafana_dashboard: "1"
and imports them. A dashboard edited in the UI is lost when the Pod is
replaced; one in a ConfigMap is version-controlled, reviewable and
reproducible.
The three panels worth having before any others:
promql
# Rate
sum(rate(http_requests_total{namespace="platform"}[5m])) by (service)
# Errors
sum(rate(http_requests_total{namespace="platform",status=~"5.."}[5m])) by (service)
/ sum(rate(http_requests_total{namespace="platform"}[5m])) by (service)
# Duration
histogram_quantile(0.95,
sum(rate(http_request_duration_seconds_bucket{namespace="platform"}[5m])) by (le, service))Verify it worked#
Terminal
# The Operator loaded your rules
kubectl get prometheusrule -n monitoring
curl -s localhost:9090/api/v1/rules | jq -r '.data.groups[].name'
# The expression returns something. An alert on a metric that does not exist
# is silent forever and looks identical to an alert that is not firing.
curl -sG localhost:9090/api/v1/query \
--data-urlencode 'query=sum(rate(http_requests_total{namespace="platform"}[5m]))' \
| jq '.data.result | length' # must be > 0Now make one fire — this is the lab.
Terminal
# Cause real restarts
kubectl set image deploy/api api=nginx:does-not-exist -n platform
# Watch it move through the states
watch -n5 'curl -s localhost:9090/api/v1/alerts | jq -r ".data.alerts[] | \"\(.labels.alertname) \(.state)\""'
# inactive -> pending (during `for:`) -> firing
# It reached Alertmanager
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-alertmanager 9093:9093
curl -s localhost:9093/api/v2/alerts | jq -r '.[].labels.alertname'
# Restore, and confirm it resolves
kubectl rollout undo deploy/api -n platformAn alert that has never fired is a hypothesis. Watching it go
pending → firing and then resolve is the only way to know the expression, the
for:, the labels and the routing all work together.
The rule does not appear in Prometheus
The release label is missing, or ruleSelectorNilUsesHelmValues was left
true. kubectl logs -n monitoring prometheus-operator shows what it loaded.
An alert never fires even though the condition is true
Run the expression in the Prometheus UI. Almost always it returns no data —
the metric name is wrong, or the label selector matches nothing. No data is not
false; it is silence.
Everything fires at once after a deploy
for: is too short, or missing. Rollouts produce brief spikes by design.
One node failure produces forty pages
No inhibit_rules and no group_by.
The dashboard disappeared
It was created in the UI on a Pod with no persistence. Put it in a ConfigMap.
{{ $value }} renders as a long float
Use humanize, humanizePercentage or humanizeDuration.
Clean up#
Destructive — This removes real resources. Check which environment you are in first.
Terminal
kubectl delete prometheusrule platform-alerts -n monitoring
kubectl delete configmap -l grafana_dashboard=1 -n monitoringCost of this lab: Free on top of the monitoring stack. Rules and dashboards are configuration; the Prometheus and Grafana volumes underneath them are what bills.
Success criteria
0 of 4
The concept behind it
Next up
Lab 50 of 59 on the project path