Container Registry (AWS ECR)

Why this comes after the compute chapters#

You can now run containers on machines that scale behind a load balancer. One thing is still missing, and it is the thing you actually built back in the Docker chapter: the image. It exists on your laptop, and a node in a private subnet has no way to reach your laptop.

When Jenkins finishes compiling your code, it runs docker build to create a Docker Image. But Jenkins is just a temporary robot. Where does it store the Image so that Kubernetes can download it later?

It pushes the image to a Container Registry. In our platform, we use Amazon Elastic Container Registry (ECR).

In the capstone, this is the registry the pipeline pushes to and the cluster pulls from.

Level 1 — Beginner#

What is a Container Registry?#

Imagine you write a hit song.

• You record it in your studio (Docker Build).
• But to let millions of people listen to it, you can't just keep it on your laptop. You upload it to Spotify.
• People on their phones download the song from Spotify to listen to it.

A Container Registry is Spotify for Docker Images.

• Jenkins uploads the finished "song" (Docker Image) to ECR.
• Kubernetes downloads the "song" from ECR to play it on the servers.

ASCII Diagram: The ECR Workflow#

[ Jenkins ] ---> `docker push` ---> [ AWS ECR (Cloud Locker) ]
                                                |
                                                | `docker pull`
                                                v
                                      [ Kubernetes Cluster ]

Level 2 — Intermediate#

ECR vs. Docker Hub#

Why don't we just use Docker Hub? It's free!

• Private by default: Docker Hub offers private repositories too, but its default and its culture are public, and pushing a proprietary image to a public repository means anyone can pull it — including whatever credentials or source ended up baked into a layer. An ECR repository is private unless you deliberately make it public.
• Locality and transfer cost: a registry in the same region as the cluster is reached over the AWS network rather than the internet, so pulls are faster and you are not paying to move the image in from outside. How much faster depends on the image size and the node's bandwidth — a large image is still tens of seconds of pulling, which is why image size matters and why nodes cache layers.
• Rate Limits: Docker Hub limits how many times you can download an image per hour (to prevent spam). If your Kubernetes cluster autoscales and tries to download the image 200 times, Docker Hub will block you. ECR has massive enterprise rate limits.

How to use ECR (The Commands)#

To push an image to ECR, Jenkins must first prove who it is.

# 1. Get a temporary password from AWS and log into Docker
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 111122223333.dkr.ecr.us-east-1.amazonaws.com
 
# 2. Tag the image with the ECR URL
docker tag ivolve-api:v2.0 111122223333.dkr.ecr.us-east-1.amazonaws.com/ivolve-api:v2.0
 
# 3. Upload it
docker push 111122223333.dkr.ecr.us-east-1.amazonaws.com/ivolve-api:v2.0

Level 3 — Advanced#

Analyzing the Actual Code (Line-by-Line Breakdown)#

Kubernetes needs permission to download the image from ECR. If ECR is private, how does the kubelet bypass the security guard?

In AWS, if the EC2 Worker Nodes have the correct IAM Instance Profile (IAM Role), the kubelet natively knows how to ask AWS for an ECR password. But if you are running Kubernetes outside of AWS (or using a strict IAM setup), you must create an imagePullSecret in Kubernetes.

Look at how a Pod requests a private image:

apiVersion: v1
kind: Pod
metadata:
  name: ivolve-api
spec:
  containers:
  - name: api
    image: 111122223333.dkr.ecr.us-east-1.amazonaws.com/ivolve-api:v2.0
  imagePullSecrets:
  - name: ecr-registry-secret

Line-by-Line Breakdown:

• image:: The full URL to the ECR registry. The 111122223333 is your AWS Account ID.
• imagePullSecrets:: This tells Kubernetes: "Before you try to download this image, look inside the Kubernetes Secret named ecr-registry-secret. You will find a Docker username and password in there. Send those credentials to ECR."

Practise: Amazon ECR & S3 Storage Buckets provisions the registry itself.

Image Tagging Strategies#

Never use the latest tag in production (e.g., image: ivolve-api:latest). If ArgoCD sees latest, it doesn't know if the image actually changed. It might not trigger an update. Furthermore, if you need to rollback, you don't know what the previous version was because it was overwritten by the new latest. Always tag images with the Git commit hash: ivolve-api:a1b2c3d.

Level 4 — Enterprise#

Enterprise Patterns: Immutable Repositories#

In a Fortune 500 company, if an image is deployed to production, it must never change. If a developer accidentally builds a broken image and tags it as v1.0, and pushes it, they will overwrite the working v1.0.

To prevent this, Enterprise ECR repositories are configured with Image Tag Mutability: IMMUTABLE. Once an image is pushed with a specific tag, the AWS API physically blocks anyone from ever pushing a different image with that same tag again.

ECR Image Scanning#

The CI pipeline you build later scans images with Trivy before they are pushed here — see DevSecOps (Container Security). But what if a new Zero-Day vulnerability is discovered tomorrow, long after the CI pipeline finished? Enterprise ECR repositories enable Continuous Scanning. AWS continuously scans all stored images against the latest CVE databases. If a new vulnerability is found in an old image that is currently running in your cluster, AWS EventBridge triggers an alert to PagerDuty to wake up the Security team.

Practise: Jenkins Pipeline: Build, Scan and Push an Image wires the scan into a pipeline, including making the gate actually fail.

Lifecycle Policies#

If Jenkins runs 50 times a day, it creates 50 new Docker images. At 500MB each, you will quickly owe AWS thousands of dollars in storage fees for images you no longer use. Enterprises configure ECR Lifecycle Policies (using Terraform):

{
    "rulePriority": 1,
    "description": "Keep only the 30 most recent images",
    "selection": {
        "tagStatus": "any",
        "countType": "imageCountMoreThan",
        "countNumber": 30
    },
    "action": {
        "type": "expire"
    }
}

This automatically deletes old, unused images, optimizing cloud costs automatically.

When it breaks#

Symptom → evidence → hypothesis → test → fix.

repository does not exist for a repository that does#

Symptom. A push or pull fails claiming the repository is absent.

Evidence. Read the registry hostname in the error. It encodes an account ID and a region:

<account>.dkr.ecr.<region>.amazonaws.com/<repo>

Hypothesis. ECR repositories are regional and account-scoped. A repository in eu-west-1 is invisible to a client configured for us-east-1, and the message is the same as if it never existed.

Test. aws ecr describe-repositories --region <region> in the region the URL names, not the one you assume.

Fix. Correct the region or the account in the image reference. ECR also does not create repositories on push the way Docker Hub does — the repository must exist first, which is why the Terraform module creates it.

The image the cluster needs has been deleted#

Symptom. A rollback or a node replacement fails with an image that cannot be pulled, for a version that was working.

Evidence. The repository's lifecycle policy, and whether the tag is still listed.

Hypothesis. A lifecycle policy that keeps the last N images counts images, not deployments. A busy pipeline can push past N in a day and remove the image that production is still running — which nothing notices until a Pod is rescheduled.

Test. aws ecr list-images and look for the digest the running Deployment references.

Fix. Write lifecycle rules that exclude what is deployed — commonly by protecting release-tagged images and expiring only untagged layers. The failure mode is delayed: everything is fine until the day a node is replaced.

Interview Questions#

Beginner#

Q: What is the difference between Docker Hub and AWS ECR? A: Both are container registries used to store Docker images. Docker Hub is the default, generic public registry. AWS ECR is Amazon's fully managed, highly secure private registry designed for enterprise AWS integration.

Intermediate#

Q: Why do we get an ImagePullBackOff error in Kubernetes? A: This usually means Kubernetes cannot download the image from the registry. The two most common causes are: a typo in the image name/tag, or Kubernetes lacks the authentication permissions (imagePullSecrets or IAM Roles) to access the private ECR repository.

Senior#

Q: Explain how you optimize ECR storage costs for a microservice that builds 100 times a day. A: You implement an ECR Lifecycle Policy. The policy evaluates rules daily, such as deleting untagged images older than 7 days, or keeping only the 50 most recently pushed images for tagged releases. This ensures the registry only retains artifacts actively used for production or immediate rollbacks.

Principal/Architect#

Q: You have a multi-region Active-Active Kubernetes architecture spanning us-east-1 and eu-west-1. How do you architect the ECR deployment to minimize cross-region data transfer costs and reduce deployment latency? A: You do not have the EU cluster pull images across the Atlantic from the US registry. That incurs high cross-region data transfer out (DTO) costs and slows down pod startup. Instead, you configure ECR Cross-Region Replication. You push the image once to us-east-1. ECR asynchronously replicates the image to an identical registry in eu-west-1. The EU Kubernetes cluster then pulls the image from its local EU ECR, keeping traffic on the AWS backbone and eliminating internet DTO costs while minimizing latency. Contents | Artifact Management (Nexus) |