Node Drain, Upgrade & Recovery

kubectl create deployment web --image=nginx:1.27-alpine --replicas=4
kubectl expose deployment web --port=80
kubectl get pods -o wide       # note the spread across nodes

Steady traffic, in another terminal:

kubectl run load --rm -it --image=curlimages/curl --restart=Never -- \
  sh -c 'while true; do curl -s -o /dev/null -w "%{http_code} " http://web; sleep 0.2; done'

kubectl cordon ops-worker
kubectl get nodes              # SchedulingDisabled — nothing new lands here

cordon stops new Pods arriving. Existing ones keep running, which makes it safe to run well before the maintenance window.

kubectl drain ops-worker --ignore-daemonsets --delete-emptydir-data

• --ignore-daemonsets — DaemonSet Pods are recreated on the same node by design, so a drain can never evict them and refuses to start without this.
• --delete-emptydir-data — acknowledges that emptyDir data on this node is destroyed. Say it deliberately.

Watch the traffic terminal. Count non-200 responses.

The Pod stayed in the Service's endpoints while it was terminating. The fix is in the workload, not in the drain:

      terminationGracePeriodSeconds: 30
      containers:
        - name: web
          readinessProbe:
            httpGet: { path: /, port: 80 }
            periodSeconds: 3
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 5"]

The preStop sleep is not superstition. On termination, two things happen concurrently: the Pod is removed from endpoints, and SIGTERM is sent. The five seconds let the endpoint removal propagate to every kube-proxy before the process starts shutting down, so no traffic is routed to a dying Pod.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: web

kubectl apply -f pdb.yaml
kubectl drain ops-worker2 --ignore-daemonsets 2>&1 | head -3
# Cannot evict pod as it would violate the disruption budget

A PDB governs voluntary disruption only — drains, upgrades, autoscaler scale-downs. It does nothing about a crash or a node failure, and that distinction is the whole point: it prevents self-inflicted outages during maintenance.

minAvailable must be lower than the replica count, or no drain can ever proceed.

kubectl uncordon ops-worker
kubectl get nodes
kubectl rollout restart deployment/web    # rebalance, once it is schedulable

Note that nothing moves back on its own. Kubernetes does not rebalance running Pods, so after a drain the remaining nodes stay loaded until something forces a reschedule.

cordon → drain → upgrade kubelet/kubeadm → uncordon → verify → next node

• Upgrade the control plane first, one node at a time.
• Never skip a minor version — 1.28 → 1.30 is two upgrades.
• Check API deprecations before starting: a removed API version breaks workloads on the new nodes only, so it looks like a partial outage.

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

That metric names what is still calling a deprecated API, which is exactly the list you want before an upgrade rather than after.