06 · KubernetesLab 36 / 59

Guided labterraform

Amazon EKS Cluster & Managed Node Group Provisioning

Provision the cluster everything else runs on, with worker nodes in private subnets and no public endpoint.

Time: 31 min
Level: Intermediate
Objectives: 4 objectives
Cost: Billable

Start the lab Challenge

Where this fits in the platform

Already built

This lab adds

An EKS cluster with workers in private subnets

Which lets you

Before you start

Cost — Billable

An EKS control plane is $0.10/hour (~$73/month) from the moment it exists, with no free tier — plus the node group's EC2 instances and any NAT Gateway. Budget a few dollars for an afternoon, and destroy the cluster the same day.

How to clean up →

The scenario#

You have a network, roles and a registry. Nothing is running on any of it.

This is the cluster — and the two permission systems that catch everybody the first time, because they look like one.

Hands-on environment

Run it on AWS

This lab builds real cloud infrastructure, so it needs your own AWS account. Follow the cost and cleanup notes above — the resources are yours, and so is the bill.

Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.

The cluster

Step 1 of 5

What you are building#

text

   AWS manages this                     You manage this
  +-------------------------+          +--------------------------+
  |  EKS control plane      |          |  managed node group      |
  |  apiserver, etcd,       | <------> |  2 x t3.medium, private  |
  |  scheduler, controllers |          |  subnets, autoscaling    |
  |  across 3 AZs           |          |  to 6                    |
  +-------------------------+          +--------------------------+
        $0.10/hour                          EC2 pricing

EKS runs the control plane you built by hand in the kubeadm lab: the API server, etcd and the controllers, replicated across availability zones and patched by AWS. You do not get a node to log into, and you do not get to break etcd.

A managed node group is still EC2. AWS handles the launch template, the draining on upgrade and the replacement of an unhealthy instance, but the nodes are yours, they sit in your subnets, and they bill at normal EC2 rates.

Build it#

What you are proving: You can provision a managed control plane and know which parts AWS runs for you

This step settles no success criterion on its own.

hcl

resource "aws_eks_cluster" "main" {
  name     = "platform"
  role_arn = aws_iam_role.cluster.arn
  version  = "1.31"
 
  vpc_config {
    subnet_ids              = concat(var.private_subnet_ids, var.public_subnet_ids)
    endpoint_private_access = true
    endpoint_public_access  = true
    public_access_cidrs     = ["203.0.113.4/32"]   # your address, not 0.0.0.0/0
  }
 
  encryption_config {
    provider { key_arn = aws_kms_key.eks.arn }
    resources = ["secrets"]
  }
 
  enabled_cluster_log_types = ["api", "audit", "authenticator"]
 
  access_config {
    authentication_mode                         = "API"
    bootstrap_cluster_creator_admin_permissions = true
  }
}

encryption_config is envelope encryption of Secrets in etcd with your KMS key. Without it, a Secret is base64 in etcd and AWS holds the only key. It can only be enabled at creation — there is no adding it later.

enabled_cluster_log_types is the only way to see the control plane. audit in particular answers "who did this", and turning it on after an incident is too late.

What you are proving: You can run nodes in private subnets and scale them without recreating the cluster

Marking this settles success criterion 4.

hcl

resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "workers"
  node_role_arn   = aws_iam_role.node.arn
  subnet_ids      = var.private_subnet_ids     # private, always
 
  instance_types = ["t3.medium"]
  capacity_type  = "ON_DEMAND"                 # or SPOT — see below
 
  scaling_config {
    desired_size = 2
    min_size     = 2
    max_size     = 6
  }
 
  update_config {
    max_unavailable = 1                        # one node at a time
  }
 
  lifecycle {
    ignore_changes = [scaling_config[0].desired_size]
  }
}

ignore_changes on desired_size matters once autoscaling is real. The Cluster Autoscaler changes the desired count; without this, the next terraform apply sets it back to 2 and evicts whatever had scaled up.

Nodes go in private subnets. They reach the internet through the NAT Gateway to pull images; nothing on the internet reaches them.

capacity_type = "SPOT" is up to 90% cheaper and gives two minutes' notice before reclamation — a good fit for stateless workloads, a bad one for the database.

What you are proving: You can create the OIDC provider that lets a ServiceAccount assume an IAM role

Marking this settles success criterion 2.

hcl

data "tls_certificate" "eks" {
  url = aws_eks_cluster.main.identity[0].oidc[0].issuer
}
 
resource "aws_iam_openid_connect_provider" "eks" {
  url             = aws_eks_cluster.main.identity[0].oidc[0].issuer
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.eks.certificates[0].sha1_fingerprint]
}

Without this, the only way to give a Pod AWS permissions is to attach them to the node role — which grants them to every Pod on that node. The IRSA lab builds on this.

What you are proving: You can install the add-ons a cluster needs before anything useful runs on it

This step settles no success criterion on its own.

hcl

resource "aws_eks_addon" "this" {
  for_each      = toset(["vpc-cni", "coredns", "kube-proxy", "aws-ebs-csi-driver"])
  cluster_name  = aws_eks_cluster.main.name
  addon_name    = each.value
  resolve_conflicts_on_update = "OVERWRITE"
}

A cluster without aws-ebs-csi-driver cannot bind a PersistentVolumeClaim, and the failure is a PVC that stays Pending with nothing obviously wrong.

What you are proving: You can reach the cluster and see Ready nodes, and explain why update-kubeconfig succeeding does not mean kubectl will work

Marking this settles success criteria 1 and 3.

Terminal

aws eks update-kubeconfig --name platform --region us-east-1
kubectl get nodes

Two permission systems, not one#

This is the part worth slowing down for, because the error message is unhelpful.

text

IAM          ->  may you call the EKS API? (DescribeCluster, ListClusters)
EKS access   ->  may you call the KUBERNETES API? (get pods, create deploy)

update-kubeconfig only needs the first. It writes a file. Getting a kubeconfig therefore tells you nothing about whether kubectl will work, and the failure appears one command later as:

text

error: You must be logged in to the server (Unauthorized)

Grant the second explicitly:

Terminal

aws eks create-access-entry --cluster-name platform \
  --principal-arn arn:aws:iam::111122223333:role/deployer \
  --type STANDARD
 
aws eks associate-access-policy --cluster-name platform \
  --principal-arn arn:aws:iam::111122223333:role/deployer \
  --access-scope type=namespace,namespaces=production \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy

On clusters older than 1.23 this is the aws-auth ConfigMap instead — the same idea with a much worse failure mode, since a malformed edit locks everyone out including you.

Verify it worked#

Destructive — This removes real resources. Check which environment you are in first.

Terminal

# Nodes are Ready, and in private subnets
kubectl get nodes -o wide
aws ec2 describe-instances --filters "Name=tag:eks:cluster-name,Values=platform" \
  --query 'Reservations[].Instances[].[InstanceId,PrivateIpAddress,PublicIpAddress]' --output table
# the PublicIpAddress column must be empty
 
# The OIDC provider exists and matches the cluster issuer
aws eks describe-cluster --name platform --query 'cluster.identity.oidc.issuer'
aws iam list-open-id-connect-providers
 
# Secrets are encrypted with your key
aws eks describe-cluster --name platform --query 'cluster.encryptionConfig'
 
# Something actually schedules
kubectl run smoke --image=nginx:alpine --restart=Never
kubectl wait --for=condition=Ready pod/smoke --timeout=90s && kubectl delete pod smoke

That last one is the real test. Nodes reporting Ready and a Pod actually running are different claims — a broken CNI gives you the first without the second.

Clean up#

Destructive — This removes real resources. Check which environment you are in first.

Terminal

kubectl delete svc --all-namespaces --field-selector spec.type=LoadBalancer
kubectl delete ingress --all -A
terraform destroy -auto-approve
aws eks list-clusters

Cost of this lab: Billable. The EKS control plane is $0.10/hour (~$73/month) whether or not anything runs on it, plus two t3.medium nodes at about $0.08/hour together. Destroy it the moment you finish.

Success criteria

0 of 4

The concept behind it

Infrastructure as Code (Terraform)

Ready to try it without help?Do the challenge

Next up

Lab 36 of 59 on the project path

Application Routing with K8s Ingress & AWS Load Balancer ControllerLet a Kubernetes manifest provision a real AWS load balancer, and make the application reachable from the internet.Why next: An EKS cluster with workers in private subnets23 minIntermediateBillable — destroy resources when you finish

Previous: Core Kubernetes Workloads, ConfigMaps & Secrets