04 · Infrastructure as CodeLab 27 / 59

Guided labterraform

Jenkins EC2 Instance, S3 Backend & AWS Backup Vault

Move Terraform state off your laptop into locked remote storage, and put the build server under a backup plan.

Time: 23 min
Level: Intermediate
Objectives: 4 objectives
Cost: Low cost

Start the lab Challenge

Where this fits in the platform

Already built

This lab adds

A bare build host, and backups of the state that describes it

Which lets you

Ansible Architecture, Configuration & Automated Inventory

Before you start

Cost — Low cost

— a `t3.micro` EC2 instance and S3 storage are inside the 12-month allowance. Outside it, expect ~$8/month for the instance if left running. AWS Backup charges for stored recovery points.

How to clean up →

The scenario#

Terraform state is a file on one laptop. Two people ran apply at the same time last week and the state now disagrees with reality in ways nobody has fully mapped.

The Jenkins server has no backups and its public IP changes every time it is stopped, which breaks every webhook.

Hands-on environment

Run it on AWS

This lab builds real cloud infrastructure, so it needs your own AWS account. Follow the cost and cleanup notes above — the resources are yours, and so is the bill.

Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.

The backend, created locally

Step 1 of 4

What you are building#

The bootstrap problem first: the backend that stores state cannot itself be created by the configuration that uses it. So this is two stages, and that is not an accident of tooling.

text

  stage 1  →  create the bucket + lock table with LOCAL state
  stage 2  →  every other stack uses them as a remote backend

What you are proving: You can create the backend that holds state, and recover a deleted state file from versioning

Marking this settles success criterion 2.

hcl

resource "aws_s3_bucket" "state" {
  bucket = "platform-tfstate-${data.aws_caller_identity.current.account_id}"
}
 
resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration { status = "Enabled" }
}
 
resource "aws_dynamodb_table" "lock" {
  name         = "platform-tfstate-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
}

Two properties doing two different jobs:

Versioning protects against a bad write or a deletion. Every previous state is retrievable.
The lock table protects against two people applying at once. Terraform writes a lock item before touching state and removes it afterwards; a second run sees the item and refuses.

PAY_PER_REQUEST costs effectively nothing at this volume — a few writes per apply.

What you are proving: You can move state into S3 with locking, so a second concurrent apply is refused rather than racing

Marking this settles success criterion 1.

hcl

terraform {
  backend "s3" {
    bucket         = "platform-tfstate-111122223333"
    key            = "production/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "platform-tfstate-lock"
    encrypt        = true
  }
}

Terminal

terraform init -migrate-state

The key is a path inside the bucket. One bucket holding production/…, staging/… and network/… keeps each stack's state separate while sharing one lock table — which is what you want, because a lock is per state file.

The backend block cannot use variables. It is read before Terraform evaluates anything, so these values are literals or come from -backend-config flags. This surprises everyone once.

What you are proving: You can give a host an address that survives a stop and start

Marking this settles success criterion 3.

hcl

resource "aws_instance" "jenkins" {
  ami                    = data.aws_ami.al2023.id
  instance_type          = "t3.medium"
  subnet_id              = var.public_subnet_id
  vpc_security_group_ids = [aws_security_group.jenkins.id]
  iam_instance_profile   = aws_iam_instance_profile.jenkins.name
 
  metadata_options {
    http_tokens = "required"        # IMDSv2
  }
 
  root_block_device {
    volume_size = 30
    volume_type = "gp3"
    encrypted   = true
  }
 
  user_data = <<-EOF
    #!/bin/bash
    dnf install -y java-21-amazon-corretto docker git
    systemctl enable --now docker
    curl -fsSL https://pkg.jenkins.io/redhat-stable/jenkins.io-2026.key       -o /etc/pki/rpm-gpg/jenkins.key
    rpm --import /etc/pki/rpm-gpg/jenkins.key
    dnf install -y jenkins
    systemctl enable --now jenkins
  EOF
 
  tags = { Name = "jenkins", Backup = "daily" }
}
 
resource "aws_eip" "jenkins" {
  instance = aws_instance.jenkins.id
  domain   = "vpc"
}

A public IP is released when an instance stops and a different one is assigned on start. An Elastic IP stays. That matters here because GitHub webhooks point at an address, and every stop would otherwise mean reconfiguring them.

An EIP is free while attached and billed while unattached, which is the reverse of what people assume and a common small mystery charge.

What you are proving: You can select resources for backup by tag, and verify a recovery point actually exists

Marking this settles success criterion 4.

hcl

resource "aws_backup_vault" "main" {
  name = "platform"
}
 
resource "aws_backup_plan" "daily" {
  name = "daily-30d"
  rule {
    rule_name         = "daily"
    target_vault_name = aws_backup_vault.main.name
    schedule          = "cron(0 5 * * ? *)"      # 05:00 UTC
    start_window      = 60
    completion_window = 180
    lifecycle { delete_after = 30 }
  }
}
 
resource "aws_backup_selection" "tagged" {
  name         = "tagged-daily"
  plan_id      = aws_backup_plan.daily.id
  iam_role_arn = aws_iam_role.backup.arn
 
  selection_tag {
    type  = "STRINGEQUAL"
    key   = "Backup"
    value = "daily"
  }
}

Selecting by tag rather than by resource ID means a new instance carrying Backup = daily is protected the moment it exists. Selecting by ID means somebody has to remember, and eventually will not.

Verify it worked#

Terminal

# State is remote, and locking works
terraform state list | head
terraform plan &                       # hold a lock
terraform plan                         # must report: Error acquiring the state lock
wait
 
# A deleted state file is recoverable
aws s3api list-object-versions --bucket <state-bucket> --prefix production/terraform.tfstate \
  --query 'Versions[].[VersionId,LastModified]' --output table
 
# The address survives a stop/start
aws ec2 stop-instances --instance-ids <id> && aws ec2 wait instance-stopped --instance-ids <id>
aws ec2 start-instances --instance-ids <id> && aws ec2 wait instance-running --instance-ids <id>
aws ec2 describe-addresses --query 'Addresses[].[PublicIp,InstanceId]' --output table
 
# A recovery point actually exists — not just a plan that says it will
aws backup list-recovery-points-by-backup-vault --backup-vault-name platform \
  --query 'RecoveryPoints[].[CreationDate,Status,ResourceArn]' --output table

That last one is the difference between "backups are configured" and "backups happened". A plan with no recovery points is a plan that has never run, and you find that out either now or during a restore.

Clean up#

Destructive — This removes real resources. Check which environment you are in first.

Terminal

# Backups outlive the instance — delete recovery points first
aws backup list-recovery-points-by-backup-vault --backup-vault-name platform \
  --query 'RecoveryPoints[].RecoveryPointArn' --output text
terraform destroy -auto-approve
aws s3 rm s3://<state-bucket> --recursive        # only when finished for good

Cost of this lab: Billable. A t3.medium is about $0.042/hour (~$30/month). S3, DynamoDB on-demand and the EIP while attached are cents. Recovery points bill until deleted and survive terraform destroy.

Success criteria

0 of 4

The concept behind it

Infrastructure as Code (Terraform)

Ready to try it without help?Do the challenge

Next up

Lab 27 of 59 on the project path

Terraform Validation, Linting & CIBuild the gate that runs before every apply: format, validate, lint, scan, and a plan a human approves.50 minIntermediate

Previous: Amazon RDS PostgreSQL & AWS Secrets Manager Integration