11 · Operating itLab 55 / 59

Guided labsreDestructive

Backup & Disaster Recovery Drill

Lose the database on purpose, restore it, and write down the RTO and RPO you actually achieved.

Time: 55 min
Level: Advanced
Objectives: 4 objectives
Cost: Free

Start the lab Challenge

Where this fits in the platform

Already built

This lab adds

A restore you have performed, not a backup you assume works

Which lets you

Production Capstone: Build, Deploy & Operate the Platform

Before you start

You will need

Docker
Docker Compose
psql

You do not need these already — the lab environment below provides them.

You will be able to

Verify a backup by restoring it, not by checking it exists
Measure RTO and RPO rather than asserting them
Write a runbook someone else can follow

Cost — Free

— runs locally with Docker Compose and PostgreSQL.

Nothing to pay in the browser. Open the terminal runs this against a simulated cloud — the same API calls and the same commands, with no account and no bill. The figure above applies only if you build it in your own.

How to clean up →

The scenario#

The backup job has reported success every night for eight months. Nobody has restored one.

Today you find out whether it works — on a database you can afford to lose.

This lab destroys a database on purpose. Use the Compose stack below and nothing that matters.

Hands-on environment

Run this lab in a real terminal, free and in your browser. The environment is temporary and yours alone — break it as much as you like.

Open the terminal

Opens in Killercoda, in a new tab — keep this page open for the steps.

Run it on your own machine

Run this lab on your own machine. One command starts the environment, with everything the lab needs already installed:

You will need:

docker

git clone https://github.com/EgyKode/EgyKode-lab.git
cd EgyKode-lab
./egykode start
./egykode shell

You need Docker and Git installed. Everything else runs inside the environment. The first start downloads it and takes a few minutes; later starts are seconds.

Not sure what you already have? Run: npm run doctor — it checks and changes nothing.

Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.

Something to lose

Step 1 of 6

What you are proving: You can create data you would genuinely mind losing, so the drill is not theatre

This step settles no success criterion on its own.

yaml

services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_PASSWORD: labonly
      POSTGRES_DB: shop
    volumes:
      - pgdata:/var/lib/postgresql/data
volumes:
  pgdata:

sql

CREATE TABLE orders (id serial PRIMARY KEY, customer text, total numeric, created_at timestamptz DEFAULT now());
INSERT INTO orders (customer, total)
  SELECT 'customer-' || i, i * 10 FROM generate_series(1, 5000) i;

What you are proving: You can write a backup that checks itself rather than one that only appears to have run

Marking this settles success criterion 1.

Terminal

#!/usr/bin/env bash
set -euo pipefail
 
STAMP=$(date +%Y-%m-%dT%H-%M-%S)
OUT="backups/shop-${STAMP}.dump"
mkdir -p backups
 
# Custom format: compressed, and restorable selectively with pg_restore.
docker compose exec -T db pg_dump -U postgres -Fc shop > "${OUT}.partial"
 
size=$(stat -c %s "${OUT}.partial")
[ "$size" -gt 4096 ] || { echo "FATAL: dump is ${size} bytes"; exit 1; }
 
# Verify it parses before trusting it. This is the step that was missing for
# eight months: every command exited 0 and the dump was empty.
docker compose exec -T db pg_restore --list /dev/stdin < "${OUT}.partial" > /dev/null   || { echo "FATAL: dump is not readable by pg_restore"; exit 1; }
 
mv "${OUT}.partial" "$OUT"
echo "ok: $OUT ($size bytes)"

Three things make this a backup rather than a file:

.partial then rename — a rename is atomic, so an interrupted dump can never be mistaken for a good one.
A size floor — an empty dump is a successful command and a failed backup.
pg_restore --list — proves the archive is readable. A corrupt dump that nobody parses is discovered during the incident.

What you are proving: You can destroy the data deliberately and start the clock at the decision

Marking this settles success criterion 1.

Terminal

docker compose down -v      # the volume, and every byte in it, is gone
docker compose up -d

Start the timer.

What you are proving: You can restore into a clean database from a written runbook, and verify by querying it

Marking this settles success criteria 1 and 3.

Terminal

docker compose exec -T db psql -U postgres -c "CREATE DATABASE shop;"
docker compose exec -T db pg_restore -U postgres -d shop --no-owner < backups/shop-<stamp>.dump
docker compose exec -T db psql -U postgres -d shop -c "SELECT count(*) FROM orders;"

Stop the timer. That number is your RTO, and it is the honest one — including the time you spent finding the right file and remembering the flags.

What you are proving: You can detect a corrupted backup before you are relying on it

Marking this settles success criterion 4.

Destructive — This removes real resources. Check which environment you are in first.

Terminal

cp backups/shop-<stamp>.dump /tmp/corrupt.dump
dd if=/dev/urandom of=/tmp/corrupt.dump bs=1 seek=500 count=200 conv=notrunc
docker compose exec -T db pg_restore --list /dev/stdin < /tmp/corrupt.dump
# pg_restore: error: did not find magic string in file header

Your backup script already runs that check. This is what it catches.

What you are proving: You can state the RTO you measured and the RPO your schedule implies

Marking this settles success criterion 2.

	Definition	Yours
RTO	Time from decision to service restored	measured above
RPO	Maximum data loss, in time	your backup interval

A nightly backup means an RPO of up to 24 hours. If that is unacceptable, the answer is continuous archiving (WAL shipping), not a more frequent pg_dump.

The runbook is the deliverable, not the script. Write the restore procedure so that someone who has never done it can follow it at 3am — then have someone else run it, because the only real test of a runbook is a person who did not write it.

Clean up#

Run this even if you did not finish.

Destructive — This removes real resources. Check which environment you are in first.

Terminal

docker compose down -v
rm -rf backups /tmp/corrupt.dump

Cost of this lab: Free — runs locally with Docker Compose and PostgreSQL.

Success criteria

0 of 4

The concept behind it

Ready to try it without help?Do the challenge

Next up

Lab 55 of 59 on the project path

Chaos: Failure Injection & RecoveryKill things deliberately, measure how long recovery takes, and find the assumption that was wrong.50 minAdvanced

Previous: Incident: Service-to-Service Calls Fail