Observability in Kubernetes

Chapter 6's autoscaling itself depends on metrics. This chapter covers how to actually see what's happening inside a cluster — revisiting cloud1-8's metrics/logs/traces framework, now specifically at the Kubernetes layer.

Revisiting Cloud1-8's Three Pillars, for Kubernetes Specifically

Metrics, logs, traces — cloud1-8's own framework, from Cloud Platforms. This chapter covers how each pillar actually works at the Kubernetes layer, distinct from the cloud-provider-level monitoring that chapter covered.

kubectl's Own Built-In Observability — First-Line Tools

• kubectl logs — a specific container's stdout/stderr; the --previous flag retrieves a crashed container's last logs before it restarted, genuinely important since a freshly-restarted container's live logs won't show what caused the previous crash at all.
• kubectl describe — a resource's full state, including recent Events — often the fastest way to see why a pod is stuck: a failed scheduling reason, an image pull failure.
• kubectl get events — a cluster-wide or namespace-scoped event stream, useful for correlating when something happened across multiple resources — directly foreshadowing Course 2's own k8s2-8 troubleshooting workflow.

The Limitation of Built-In Tools

Genuinely great for a single resource, right now — but they don't retain history long-term (pod logs disappear when the pod is deleted), don't aggregate across many pods/services easily, and provide no dashboards, alerting, or trend analysis. Exactly the gap dedicated observability tooling exists to fill — the same log-aggregation problem cloud1-8 already covered at the cloud-provider level, recurring here specifically inside Kubernetes.

Metrics Server — The Baseline

Prometheus — The De Facto Standard for Kubernetes Metrics

Prometheus is a metrics collection and storage system that pulls (scrapes) metrics from configured targets at regular intervals, rather than applications pushing metrics to it. Applications commonly expose a /metrics HTTP endpoint in a specific text format; Prometheus scrapes it periodically. Node-level metrics exporters (like node-exporter) commonly run as DaemonSets — directly reusing k8s2-2's own DaemonSet material, one exporter instance per node, for exactly the reason that chapter explained.

Grafana — Visualizing What Prometheus Collects

Prometheus stores and queries metrics, but its own native UI is minimal. Grafana is the dashboarding/visualization layer commonly paired with it — Prometheus as the data source, Grafana as the presentation layer. Genuinely worth being clear these are two separate tools with distinct jobs, not one combined product, since that's a common point of confusion for newcomers.

Logs at Scale — Beyond kubectl logs

Since kubectl logs only shows one pod at a time and loses history once a pod is gone, real clusters typically run a log-aggregation pipeline — commonly a DaemonSet-based log-shipping agent (Fluentd/Fluent Bit, k8s2-2's own concrete DaemonSet example, now with its actual real-world purpose fully explained) collecting every node's container logs and forwarding them to a centralized store (Elasticsearch, Loki, or a cloud provider's own logging service per cloud1-8's own material), where they persist beyond any individual pod's lifetime and can be searched across the whole cluster.

This Course's Own Scope, Honestly

Matching this course's recurring convention: this chapter is deliberately an orientation to Kubernetes observability, not a deep Prometheus/Grafana course — this site's own bucket list has a separate, still-outstanding Observability course topic for that depth. This chapter's job is making sure the Kubernetes-specific pieces (Metrics Server as an HPA prerequisite, DaemonSet-based collection patterns, the built-in-vs-dedicated-tooling gap) are understood — exactly what a dedicated Prometheus/Grafana course would otherwise need to re-explain from scratch in a Kubernetes context anyway.

Hands-On Exercises