07 · PackagingLab 43 / 59

Guided labhelm

Managing EKS Cluster Add-ons with Helm & IRSA

Install the controllers a cluster needs to be useful, each with its own AWS identity instead of node credentials.

Time: 39 min
Level: Intermediate
Objectives: 4 objectives
Cost: Billable

Start the lab Challenge

Where this fits in the platform

Already built

This lab adds

Cluster add-ons holding AWS permissions through IRSA

Which lets you

Before you start

Cost — Billable

Cluster add-ons commonly provision real infrastructure — a load balancer, EBS volumes for persistent storage — that outlives `helm uninstall` if a finalizer fails.

How to clean up →

The scenario#

The Load Balancer Controller works because someone attached its policy to the node role. Every Pod on every node now has permission to create and delete load balancers, and nothing in the cluster shows that.

Hands-on environment

Run it on AWS

This lab builds real cloud infrastructure, so it needs your own AWS account. Follow the cost and cleanup notes above — the resources are yours, and so is the bill.

Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.

The role, trusted by one ServiceAccount

Step 1 of 3

What you are building#

An empty EKS cluster cannot do much: it cannot provision a load balancer from an Ingress, cannot report CPU for an HPA, and cannot attach a volume. Those come from controllers you install — and the interesting part is how they get AWS permissions.

text

  Pod
   |  projected token: "I am system:serviceaccount:kube-system:alb-controller"
   v
  EKS OIDC provider  ->  AWS STS  ->  temporary credentials for ONE role

Without IRSA there are two options and both are bad: an access key in a Secret, which never rotates, or a policy on the node role, which grants it to every Pod on the node. IRSA gives one workload one role, with credentials that expire.

Build it#

What you are proving: You can scope an IAM role to a single ServiceAccount rather than to the whole node

Marking this settles success criterion 1.

hcl

data "aws_iam_policy_document" "lbc_trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type        = "Federated"
      identifiers = [var.oidc_provider_arn]
    }
    condition {
      test     = "StringEquals"
      variable = "${var.oidc_host}:sub"
      values   = ["system:serviceaccount:kube-system:aws-load-balancer-controller"]
    }
    condition {
      test     = "StringEquals"
      variable = "${var.oidc_host}:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}
 
resource "aws_iam_role" "lbc" {
  name               = "alb-controller"
  assume_role_policy = data.aws_iam_policy_document.lbc_trust.json
}

Both conditions are required. Without sub, any ServiceAccount in the cluster can assume the role — which is worse than the node-role problem it was meant to fix, because it now looks like it was done properly. Without aud, a token minted for another audience is accepted.

oidc_host is the issuer URL with https:// stripped.

What you are proving: You can prove a Pod assumes its own role, and that one without the annotation is refused

Marking this settles success criteria 2 and 3.

yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: aws-load-balancer-controller
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/alb-controller

The EKS Pod Identity Webhook sees this annotation and injects AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE and a projected token into every Pod using the ServiceAccount. Every AWS SDK finds them with no configuration.

The injection happens at Pod creation. Annotating a ServiceAccount does not affect Pods that already exist — they must be restarted, and this is the reason IRSA "does not work" more often than any misconfigured trust policy.

What you are proving: You can install the add-ons that give kubectl top and an HPA a metric to read

Marking this settles success criterion 4.

Terminal

helm repo add eks https://aws.github.io/eks-charts
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
 
helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=platform \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region=us-east-1 \
  --set vpcId=vpc-0abc \
  --atomic --wait
 
helm upgrade --install metrics-server metrics-server/metrics-server \
  -n kube-system --atomic --wait

serviceAccount.create=false matters: letting the chart create the ServiceAccount produces one without your annotation, and the controller falls back to the node role — which usually works, so nothing looks wrong.

Pin chart versions with --version for the same reason you pin images.

Verify it worked#

Terminal

# The Pod has an identity of its own
kubectl exec -n kube-system deploy/aws-load-balancer-controller -- env | grep AWS_ROLE_ARN
kubectl exec -n kube-system deploy/aws-load-balancer-controller -- \
  ls /var/run/secrets/eks.amazonaws.com/serviceaccount/
 
# It IS that role, not the node role
kubectl run awscli --rm -it --image=amazon/aws-cli --restart=Never \
  -n kube-system --overrides='{"spec":{"serviceAccountName":"aws-load-balancer-controller"}}' \
  -- sts get-caller-identity
# Arn: .../alb-controller/botocore-session-...
 
# The negative test — a Pod WITHOUT the annotation
kubectl run awscli-plain --rm -it --image=amazon/aws-cli --restart=Never \
  -- sts get-caller-identity
# returns the NODE role — proves the scoping is real
 
# metrics-server actually serves metrics
kubectl top nodes
kubectl top pods -A | head
 
# The controller can do its job
kubectl logs -n kube-system deploy/aws-load-balancer-controller --tail=20

The negative test is the one that proves the point. Showing the controller works does not distinguish IRSA from a policy on the node role — showing that an unannotated Pod gets something different does.

Clean up#

Destructive — This removes real resources. Check which environment you are in first.

Terminal

helm uninstall aws-load-balancer-controller metrics-server -n kube-system
terraform destroy -target=aws_iam_role.lbc

Cost of this lab: Low. IAM roles are free. The controllers run on nodes you are already paying for; anything they provision is not.

Success criteria

0 of 4

The concept behind it

Package Management (Helm)

Ready to try it without help?Do the challenge

Phase complete · 07 Packaging

You can now: The application installs from a versioned chart, upgrades safely, and rolls back in seconds.

Next phase

Lab 43 of 59 on the project path

08 · Continuous deliveryJenkins Fundamentals & Role-Based AccessRun Jenkins in a container, build a job from a webhook, and stop every authenticated user being an administrator.50 minBeginner

Previous: Helm Upgrades, Rollbacks & Release Strategy