High Availability & Load Balancing

Why this comes after storage#

Everything so far has been one of a thing: one VPC, one instance, one bucket. A single instance is also a single point of failure, and the fix creates a problem of its own.

If you deploy your website on a single server, and that server crashes, your website goes offline. To fix this, you deploy your website on 3 servers.

But now you have a new problem: If you have 3 servers with 3 different IP addresses, which IP address do you give to your customers? You can't give them all three. You give them a Load Balancer.

In the capstone, this is the ALB that the Load Balancer Controller provisions from the Ingress object.

Level 1 — Beginner#

What is a Load Balancer?#

Imagine a busy grocery store.

• The Cashiers: The AWS EC2 Servers running your application.
• The Store Manager (Load Balancer): Stands at the front door.

When a customer walks in, the Manager looks at the cashiers. If Cashier 1 has a long line, the Manager sends the customer to Cashier 2. If Cashier 3 falls asleep (crashes), the Manager stops sending people to them. The customer only ever talks to the Manager. They don't know or care how many cashiers are working in the back.

ASCII Diagram: Traffic Distribution#

       [ User types www.ivolve.com ]
                  |
                  v
       [ AWS LOAD BALANCER ]
        /         |         \
       v          v          v
[ Server 1 ] [ Server 2 ] [ Server 3 ]
 (33% load)   (33% load)   (33% load)

Level 2 — Intermediate#

Types of AWS Load Balancers#

AWS offers three main types, but in DevOps, we primarily care about two:

1. Application Load Balancer (ALB): Operates at Layer 7 (HTTP/HTTPS). It is smart. It looks at the actual URL. If the user goes to /api, it sends them to the Backend servers. If they go to /images, it sends them to the Frontend servers.
2. Network Load Balancer (NLB): Operates at Layer 4 (TCP/UDP). It is dumb, but blazingly fast. It doesn't look at URLs. It just takes raw network packets and blasts them to the backend servers at millions of requests per second. (We use this for databases or high-performance game servers).

Health Checks#

How does the Load Balancer know if a server is dead? It requests a specific URL (like /health) on a fixed interval. If the server answers with the expected status it keeps sending traffic. If the server times out or returns an error, the load balancer counts a failure — and marks the target Unhealthy only after a configured number of consecutive failures, not on the first one. Interval times threshold is therefore how long a broken instance keeps receiving traffic, and it is the number to look at when someone asks why requests still reached a dead node.

Level 3 — Advanced#

Analyzing the Actual Code (Line-by-Line Breakdown)#

In our Kubernetes cluster, we do not expose the EC2 Worker Nodes to the internet. We put an AWS Application Load Balancer in the Public Subnets, and point it at our NGINX Ingress Controller.

Look at 02-Terraform/modules/alb/main.tf:

resource "aws_lb" "this" {
  name               = "${var.name_prefix}-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb.id]
  subnets            = var.public_subnet_ids
}
 
resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.this.arn
  port              = 443
  protocol          = "HTTP"
 
  default_action {
    type = "redirect"
    redirect {
      port        = "443"
      protocol    = "HTTPS"
      status_code = "HTTP_301"
    }
  }
}

Line-by-Line Breakdown:

• internal = false: This tells AWS to give the ALB a Public IP address so the internet can reach it. (If it were true, only resources inside the VPC could talk to it).
• subnets = var.public_subnet_ids: The ALB must be placed in the Public Subnets we created in Cloud Networking (AWS VPC). Notice it takes a list of subnets. For High Availability, we deploy the ALB across 3 different Availability Zones simultaneously.
• default_action { type = "redirect" }: This is a security best practice. If a user types http://ivolve.com, the ALB doesn't even forward the traffic to Kubernetes. The ALB intercepts it and forces the user's browser to redirect to https:// (encrypted traffic), returning a 301 Permanent Redirect.

Level 4 — Enterprise#

SSL/TLS Termination#

In an enterprise, encrypting traffic is a legal requirement. But decrypting HTTPS traffic requires heavy CPU math. If you have 10,000 Pods, and each Pod is trying to decrypt HTTPS traffic, you waste millions of dollars on CPU overhead.

The Solution: SSL Termination at the Edge. We attach an AWS ACM (Certificate Manager) SSL Certificate directly to the Application Load Balancer. The ALB uses AWS's massive specialized hardware to decrypt the HTTPS traffic. It then forwards raw, unencrypted HTTP traffic over the private, secure AWS VPC backbone to the Kubernetes nodes. The Pods never have to do math.

Integration with Kubernetes (AWS Load Balancer Controller)#

In the old days, you had to write Terraform code to create an ALB, then manually link it to Kubernetes. Today, we use the AWS Load Balancer Controller (installed via Helm).

You deploy a simple Kubernetes Ingress object:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing

The Controller sees this YAML file, reaches out to the AWS API, provisions a real Application Load Balancer, configures the Target Groups, sets up the Health Checks, and attaches the SSL certificate—all completely automatically. This bridges the gap between Kubernetes YAML and AWS hardware.

When it breaks#

Symptom → evidence → hypothesis → test → fix. A load balancer returns a different code for each failure, and the code tells you where to look. Learning these three saves more time than any dashboard.

502, 503 and 504 mean different things#

Symptom. The load balancer returns an error while the targets look fine.

Evidence. The status code, exactly:

Hypothesis. 503 is a registration or health problem and the request never reached your code. 502 and 504 mean it did — so debugging health checks on a 502 is time spent in the wrong place.

Test. Bypass the load balancer entirely and request the target directly, on its own address and port. If that works, the fault is between the balancer and the target: usually the port mapping or the security group.

Fix. Per branch. The same distinction applies to a Kubernetes Ingress, which is why Incident: Ingress 502 is worth doing — the cause there is a Service pointing at the wrong container port, and the symptom is identical to this.

Every target is unhealthy and the application is fine#

Symptom. The target group shows all targets unhealthy; the app answers when you curl it from inside.

Evidence. The health check's path, port and expected status code, compared with what the application actually serves.

Hypothesis. The check asks for / and the app serves /healthz, or the check uses the listener port while the app listens on another, or the security group does not allow the balancer's subnet to reach the health check port.

Test. Make the health check request yourself, from the balancer's subnet, against the exact path and port configured.

Fix. Whichever the test found. A health check that returns 302 counts as unhealthy unless the expected codes say otherwise — an app that redirects / to a login page fails a naive check while being perfectly healthy.

Interview Questions#

Beginner#

Q: What happens to user traffic if one of the three backend servers crashes? A: The health check for that target starts failing, and once it has failed the configured number of consecutive times the load balancer marks the target unhealthy and stops routing to it — traffic then goes to the two remaining healthy servers. Between the crash and that threshold being reached, requests are still being sent to a dead server and are failing. Interval times unhealthy-threshold is the size of that window, and shrinking it trades faster ejection against evicting a target that was briefly slow rather than broken.

Intermediate#

Q: Why do we put the Load Balancer in a Public Subnet, but the EC2 servers in a Private Subnet? A: Security. If the servers were in the public subnet, hackers could bypass the Load Balancer and attack the servers directly via SSH or open ports. By putting the servers in a private subnet, the Load Balancer becomes the only possible way into the system.

Senior#

Q: Explain the difference between an AWS ALB (Application Load Balancer) and a Kubernetes Ingress Controller (like Nginx). Do you need both? A: An ALB is a physical/managed AWS resource that balances traffic across EC2 instances. An Ingress Controller is a software router running inside the cluster that balances traffic across Pods. In a production EKS/kubeadm setup, you typically use both: The ALB receives public traffic and routes it to the EC2 nodes on a NodePort. The Nginx Ingress Controller running on those nodes receives the traffic and uses internal Kubernetes DNS to route it to the specific Pods. Alternatively, using the AWS Load Balancer Controller with "IP Mode", the ALB can bypass Nginx entirely and route traffic directly to the Pod IPs via the CNI.

Principal/Architect#

Q: During a massive DDoS attack, your Application Load Balancer scales up to handle 500,000 requests per second, but your backend Kubernetes cluster is completely overwhelmed and dies. How do you architect the edge layer to protect the cluster? A: You must decouple the traffic from the compute layer using Edge caching and Web Application Firewalls (WAF).

1. Place AWS CloudFront (a CDN) in front of the ALB. CloudFront will cache static assets globally at the edge, absorbing 80% of the traffic before it even reaches the ALB.
2. Attach AWS WAF to the ALB or CloudFront. Configure rate-limiting rules (e.g., block any IP making >100 requests per 5 minutes) and enable the AWS Shield Advanced managed DDoS protection.
3. The ALB should only ever receive legitimate, dynamic traffic, protecting the fragile Kubernetes backend from volumetric attacks. Contents | Elasticity (AWS Auto Scaling) |