04 · Infrastructure as CodeLab 25 / 59

Guided labterraform

Amazon ECR Container Registry & S3 Storage Buckets

Stand up the registry your images live in and the buckets your application writes to, both private by default.

Time: 23 min
Level: Intermediate
Objectives: 4 objectives
Cost: Low cost

Start the lab Challenge

Where this fits in the platform

Already built

AWS VPC, Subnets, Gateways & Route Tables

This lab adds

An ECR repository with immutable tags, and artifact buckets

Which lets you

Before you start

Cost — Low cost

— 500 MB of ECR storage and 5 GB of S3 are free for 12 months. Container images are large, so delete the repository rather than leaving a few GB of layers behind.

Nothing to pay in the browser. Open the terminal runs this against a simulated cloud — the same API calls and the same commands, with no account and no bill. The figure above applies only if you build it in your own.

How to clean up →

The scenario#

Images are pushed to Docker Hub with the tag latest, so nobody can say which commit is in production. Nothing scans them. The registry has eleven months of untagged layers nobody can delete safely because nobody knows what references them.

The buckets were created by hand, and one of them is public.

Hands-on environment

Run this lab in a real terminal, free and in your browser. The environment is temporary and yours alone — break it as much as you like.

Open the terminal

Opens in Killercoda, in a new tab — keep this page open for the steps.

Run it on AWS

This lab builds real cloud infrastructure, so it needs your own AWS account. Follow the cost and cleanup notes above — the resources are yours, and so is the bill.

Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.

A registry that scans and prunes

Step 1 of 4

What you are building#

Two storage services that get grouped together and behave nothing alike:

ECR — a private Docker registry with scanning on push and lifecycle rules for expiring images.
S3 — object storage, here for Terraform state and load balancer access logs.

Build it#

What you are proving: You can create a registry that scans what you push and expires what you stop tagging

Marking this settles success criterion 2.

hcl

resource "aws_ecr_repository" "app" {
  name                 = "platform/api"
  image_tag_mutability = "IMMUTABLE"
 
  image_scanning_configuration {
    scan_on_push = true
  }
 
  encryption_configuration {
    encryption_type = "KMS"
  }
}

IMMUTABLE is the setting to argue for. With mutable tags, api:1.4.2 can be overwritten, so the image you tested and the image running in production share a name and differ in content. Immutable tags make that impossible, and they force the habit of tagging by commit SHA.

hcl

resource "aws_ecr_lifecycle_policy" "app" {
  repository = aws_ecr_repository.app.name
  policy = jsonencode({
    rules = [
      {
        rulePriority = 1
        description  = "Expire untagged images after 7 days"
        selection = {
          tagStatus   = "untagged"
          countType   = "sinceImagePushed"
          countUnit   = "days"
          countNumber = 7
        }
        action = { type = "expire" }
      },
      {
        rulePriority = 2
        description  = "Keep the 30 most recent tagged images"
        selection = {
          tagStatus     = "tagged"
          tagPrefixList = ["v"]
          countType     = "imageCountMoreThan"
          countNumber   = 30
        }
        action = { type = "expire" }
      }
    ]
  })
}

Rules are evaluated in priority order and the first match wins, so an overly broad rule at priority 1 makes everything below it dead code. Untagged layers accumulate on every rebuild and are the usual answer to "why is the registry bill growing".

What you are proving: You can create buckets that are private by construction and refuse plaintext requests

Marking this settles success criterion 3.

hcl

resource "aws_s3_bucket" "state" {
  bucket = "platform-tfstate-${data.aws_caller_identity.current.account_id}"
}
 
resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
 
resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration { status = "Enabled" }
}
 
resource "aws_s3_bucket_server_side_encryption_configuration" "state" {
  bucket = aws_s3_bucket.state.id
  rule {
    apply_server_side_encryption_by_default { sse_algorithm = "AES256" }
  }
}
 
resource "aws_s3_bucket_policy" "state_tls_only" {
  bucket = aws_s3_bucket.state.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid       = "DenyInsecureTransport"
      Effect    = "Deny"
      Principal = "*"
      Action    = "s3:*"
      Resource  = [aws_s3_bucket.state.arn, "${aws_s3_bucket.state.arn}/*"]
      Condition = { Bool = { "aws:SecureTransport" = "false" } }
    }]
  })
}

Both ARNs in the policy. Bucket-level actions such as ListBucket apply to the bucket ARN; object actions apply to /*. A policy with only one is the most common S3 policy bug, and it half-works — which is worse than failing.

Versioning on the state bucket is not optional. A deleted state file is one command from recovery with it, and a full manual re-import of every resource without it. That is a lab of its own later in the path.

What you are proving: You can push an image and read the scan findings it produced

Marking this settles success criterion 1.

Terminal

aws ecr get-login-password --region us-east-1 \
  | docker login --username AWS --password-stdin \
    "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"
 
SHA=$(git rev-parse --short HEAD)
REG="$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"
 
docker build -t "$REG/platform/api:$SHA" .
docker push "$REG/platform/api:$SHA"

Tag with the commit SHA. latest cannot answer "what is running in production", which is the question you will be asked during an incident.

What you are proving: You can recover a deleted object from a versioned bucket, rather than trusting that versioning would have worked

Marking this settles success criterion 4.

Versioning was enabled in step 2. That is a setting, not a demonstration — and a backup nobody has restored from is a belief, not a control. Prove it.

Terminal

BUCKET=$(terraform output -raw state_bucket)
 
echo "recover me" > test.txt
aws s3 cp test.txt "s3://$BUCKET/test.txt"
aws s3 rm "s3://$BUCKET/test.txt"
 
# Gone from the listing...
aws s3 ls "s3://$BUCKET/test.txt"
 
# ...but not actually deleted. The delete added a marker on top.
aws s3api list-object-versions --bucket "$BUCKET" --prefix test.txt \
  --query '{versions:Versions[].VersionId,markers:DeleteMarkers[].VersionId}'

A delete in a versioned bucket writes a delete marker. The object is still there, hidden behind it. Removing the marker is the restore:

Destructive — This removes real resources. Check which environment you are in first.

Terminal

MARKER=$(aws s3api list-object-versions --bucket "$BUCKET" --prefix test.txt \
  --query 'DeleteMarkers[0].VersionId' --output text)
 
aws s3api delete-object --bucket "$BUCKET" --key test.txt --version-id "$MARKER"
aws s3 cp "s3://$BUCKET/test.txt" - # "recover me"

Note what delete-object did there: given a version id it removed that version — the marker — rather than the object. Without the version id it would have written another marker.

This is the same mechanism the state-recovery lab relies on later in the path, where the object that came back is a Terraform state file rather than a text file.

Verify it worked#

Terminal

# The scan ran, and you can read it
aws ecr describe-image-scan-findings \
  --repository-name platform/api --image-id imageTag="$SHA" \
  --query 'imageScanFindings.findingSeverityCounts'
 
# Immutable tags are enforced — this must FAIL
docker push "$REG/platform/api:$SHA"     # ImageTagAlreadyExistsException
 
# The lifecycle policy does what you think
aws ecr get-lifecycle-policy-preview --repository-name platform/api \
  --query 'previewResults[].{tag:imageTags[0],action:action.type}' --output table
 
# No bucket is public
aws s3api get-public-access-block --bucket "$(terraform output -raw state_bucket)"

get-lifecycle-policy-preview is worth knowing: it shows what the policy would expire without waiting for it to run.

Clean up#

Destructive — This removes real resources. Check which environment you are in first.

Terminal

aws ecr delete-repository --repository-name platform/api --force
aws s3 rm "s3://$(terraform output -raw state_bucket)" --recursive
terraform destroy -auto-approve

Cost of this lab: Low. ECR is $0.10/GB-month and S3 about $0.023/GB — a few images and a state file are cents. The --force on the repository is required because it holds images.

Success criteria

0 of 4

The concept behind it

Infrastructure as Code (Terraform)

Ready to try it without help?Do the challenge

Next up

Lab 25 of 59 on the project path

Amazon RDS PostgreSQL & AWS Secrets Manager IntegrationRun a Multi-AZ database that survives losing an availability zone, with a password no human ever types.31 minIntermediate

Previous: IAM Roles, IRSA Policies & Security Groups