10 · ObservabilityLab 49 / 59

Guided labprometheus

Deploying Kube-Prometheus-Stack on AWS EKS

Get metrics out of the cluster and into Grafana, so 'is it healthy' has an answer that is not a guess.

Time: 31 min
Level: Advanced
Objectives: 4 objectives
Cost: Low cost

Start the lab Challenge

Where this fits in the platform

Already built

This lab adds

Metrics from every pod and node, and somewhere to see them

Which lets you

Before you start

Cost — Low cost

Depends on an existing cluster. kube-prometheus-stack requests persistent volumes — EBS at ~$0.08/GB-month — which survive `helm uninstall` because the PVCs are retained deliberately.

Nothing to pay in the browser. Open the terminal runs this against a simulated cloud — the same API calls and the same commands, with no account and no bill. The figure above applies only if you build it in your own.

How to clean up →

The scenario#

"Is it healthy?" is answered by kubectl get pods and a guess. There is no history, so nobody can say whether last night was unusual, and the first sign of a problem is a person noticing.

Hands-on environment

Run this lab in a real terminal, free and in your browser. The environment is temporary and yours alone — break it as much as you like.

Open the terminal

Opens in Killercoda, in a new tab — keep this page open for the steps.

Run it on your own machine

Run this lab on your own machine. One command starts the environment, with everything the lab needs already installed:

The stack is a Helm chart and installs on kind unchanged. What you lose locally is the EKS control-plane metrics — node and workload metrics, the alerting rules and every Grafana dashboard in the lab still work.

You will need:

docker
kubectl
kind
helm

git clone https://github.com/EgyKode/EgyKode-lab.git
cd EgyKode-lab
./egykode start k8s
./egykode shell

You need Docker and Git installed. Everything else runs inside the environment. The first start downloads it and takes a few minutes; later starts are seconds.

Not sure what you already have? Run: npm run doctor — it checks and changes nothing.

Run it on AWS

This lab builds real cloud infrastructure, so it needs your own AWS account. Follow the cost and cleanup notes above — the resources are yours, and so is the bill.

Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.

Install, with values that matter

Step 1 of 3

What you are building#

text

  ServiceMonitor  (a CRD: "scrape any Service with these labels")
        |  the Operator reads it and rewrites the Prometheus config
        v
  Prometheus  --scrapes-->  /metrics on node-exporter, kube-state-metrics, your app
        |
        +--> Alertmanager   routing and silencing
        +--> Grafana        dashboards

The Operator is the part worth understanding. Plain Prometheus has one configuration file listing every scrape target — in a cluster where Pods come and go, that file is wrong immediately. The Operator watches ServiceMonitor and PrometheusRule objects and regenerates the configuration, so adding monitoring to a new service is creating an object, not editing a central file and reloading it.

Build it#

What you are proving: You can install a monitoring stack whose metrics survive a restart of Prometheus itself

Marking this settles success criterion 4.

yaml

# values.yaml
prometheus:
  prometheusSpec:
    retention: 15d
    # Without this, ONLY ServiceMonitors carrying the release label are picked
    # up — the most common reason a new one is ignored in silence.
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
 
    resources:
      requests: { cpu: 200m, memory: 2Gi }
      limits:   { memory: 4Gi }
 
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi
 
grafana:
  adminPassword: ""            # set it, or read the generated Secret
  persistence:
    enabled: true
    size: 5Gi
  defaultDashboardsTimezone: browser
 
alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          accessModes: ["ReadWriteOnce"]
          resources: { requests: { storage: 5Gi } }

Terminal

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace -f values.yaml --atomic --wait --timeout 10m

storageSpec is not optional in any cluster you care about. The default is emptyDir, so every Prometheus restart loses all history — and you discover that during the first incident, which is exactly when history was the point.

The three ...NilUsesHelmValues: false lines are the single most useful thing in this file. Left at their default, Prometheus only discovers ServiceMonitors labelled with this Helm release, so a ServiceMonitor you create later is ignored with no error anywhere.

What you are proving: You can have your own application scraped without editing Prometheus configuration

Marking this settles success criterion 2.

Your app needs a /metrics endpoint and a Service with a named port:

yaml

apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: platform
  labels: { app: api }
spec:
  selector: { app: api }
  ports:
    - name: http          # the name is what the ServiceMonitor references
      port: 80
      targetPort: 8000
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api
  namespace: platform
  labels: { release: monitoring }
spec:
  selector:
    matchLabels: { app: api }
  namespaceSelector:
    matchNames: ["platform"]
  endpoints:
    - port: http          # the PORT NAME, not the number
      path: /metrics
      interval: 30s

port here is the Service's port name. Putting 8000 there is the most common ServiceMonitor mistake, and it fails by simply never appearing in Targets.

What you are proving: You can confirm in Targets and in Grafana that the cluster and your workload are actually being scraped

Marking this settles success criteria 1 and 3.

Terminal

kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80
kubectl get secret -n monitoring monitoring-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo
 
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

Verify it worked#

Destructive — This removes real resources. Check which environment you are in first.

Terminal

# Everything is up
kubectl get pods -n monitoring
 
# Your target is being scraped and is UP — the actual test
curl -s localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.labels.job=="api") | "\(.health) \(.scrapeUrl) \(.lastError)"'
 
# A query returns data for your workload
curl -sG localhost:9090/api/v1/query \
  --data-urlencode 'query=sum(rate(container_cpu_usage_seconds_total{namespace="platform"}[5m])) by (pod)' \
  | jq '.data.result | length'
 
# History survives a restart — this is what storageSpec buys
kubectl delete pod -n monitoring prometheus-monitoring-kube-prometheus-prometheus-0
kubectl wait --for=condition=Ready pod/prometheus-monitoring-kube-prometheus-prometheus-0 -n monitoring --timeout=300s
# re-run the query and confirm the older data is still there

kubectl get pods showing Running proves the stack installed. The Targets query proves it is monitoring something, which is a different claim.

Clean up#

Destructive — This removes real resources. Check which environment you are in first.

Terminal

helm uninstall monitoring -n monitoring
kubectl delete pvc --all -n monitoring          # PVCs SURVIVE uninstall
kubectl delete namespace monitoring
kubectl get crd | grep coreos                   # CRDs also survive

Cost of this lab: Low on top of the cluster. Three EBS volumes totalling 30 GB is about $2.40/month, and they keep billing after helm uninstall unless you delete the PVCs.

Success criteria

0 of 4

The concept behind it

Metrics (Prometheus)

Ready to try it without help?Do the challenge

Next up

Lab 49 of 59 on the project path

Custom Prometheus Alert Rules & Grafana DashboardsWrite alerts that fire on conditions worth waking someone for, and a dashboard that shows why they fired.Why next: Metrics from every pod and node, and somewhere to see them23 minAdvanced

Previous: GitOps Delivery with Argo CD: Sync, Drift & Self-Heal