Troubleshooting Kubernetes

Chapter 7 built the tools. This chapter catalogs the actual failure patterns you'll run into repeatedly — the Kubernetes-specific counterpart to cloud2-5's own failure-mode catalog from Cloud Platforms.

A Troubleshooting Starting Discipline

Reusing Chapter 7's own tools as a fixed sequence for almost every pod-level problem: kubectl get pods first — what state is it actually in? — then kubectl describe for the specific reason/Events, then kubectl logs (and --previous, per Chapter 7) for the actual application output.

CrashLoopBackOff — The Container Keeps Dying

The container starts, exits, and Kubernetes keeps restarting it per Chapter 11's restart policy, with an increasing backoff delay between attempts — hence the name. Genuinely important: this isn't a special error state — it's the normal, expected behavior of the restart policy responding to a container that keeps failing. The root cause is almost always in the application itself, not Kubernetes. Check kubectl logs --previous (Chapter 7) first. Common real causes: an application config error, a Secret/ConfigMap (Chapter 7, Course 1) not actually mounted correctly, or the application failing a startup dependency check.

ImagePullBackOff / ErrImagePull — Kubernetes Can't Get the Image

The kubelet (Chapter 2, Course 1's architecture) can't successfully pull the specified image. Common real causes: a typo in the image name/tag, the image genuinely doesn't exist, the image is in a private registry and the pod lacks correct pull credentials — genuinely tying to Chapter 4's ServiceAccount material, since image pull secrets are attached via the ServiceAccount — or, less commonly, a registry rate limit. kubectl describe pod shows the specific pull error message in its Events, usually explaining exactly which of these applies.

Pending Pods — Never Even Started

A pod stuck in Pending (Chapter 3's own pod phase material) means the scheduler (Chapter 2) hasn't been able to place it on any node. Common real causes: insufficient cluster resource capacity for the pod's own requests (Chapter 10) — directly the same scenario Chapter 6's autoscaling chapter walked through, minus Cluster Autoscaler actually being available to fix it — a node selector/affinity rule no current node satisfies, or a PersistentVolumeClaim (Chapter 8) that can't be bound to any available PersistentVolume. kubectl describe pod again shows the specific scheduling failure reason.

Readiness Probe Failures — Running, But Never Actually Serving Traffic

A pod stuck showing 0/1 Ready despite being in the Running phase means its readiness probe (Chapter 11) is failing continuously — the pod is technically alive, but Kubernetes never adds it to a Service's Endpoints (Chapter 6), so it never receives real traffic. Common causes: the readiness probe's path/port is misconfigured — Chapter 6's own port/targetPort confusion recurring here — or the application genuinely never finishes its own startup dependency (can't reach the database, for instance). Worth naming a genuinely non-obvious cross-chapter connection: a NetworkPolicy (Chapter 5) blocking database access could be the actual underlying cause of a readiness probe that looks purely application-side.

DNS Resolution Failures Inside the Cluster

Revisiting Chapter 5's own DNS/NetworkPolicy gotcha as a cataloged, diagnosable symptom rather than just a preventive warning: a pod that can't resolve a Service name (Chapter 6) at all — check whether a NetworkPolicy is blocking egress DNS traffic first, then confirm the target Service actually exists and has matching Endpoints in the first place, rather than assuming DNS itself is broken.

RBAC "Forbidden" Errors — A Different Category Entirely

An application or kubectl command failing with a "Forbidden" error is not a networking or scheduling problem at all — it means the identity making the request (a ServiceAccount, Chapter 4's material) lacks the necessary Role/RoleBinding permissions for the specific action attempted. Genuinely worth recognizing the error message itself as the fastest signal for which category actually applies — "Forbidden" points straight at RBAC, not at networking, scheduling, or probes.

A Genuinely Useful Troubleshooting Decision Tree

Hands-On Exercises