Kubernetes¶

Prerequisites: HTTP & TCP, Load Balancing

← Cloud | Next: Observability →

Why This Exists¶

Your Deployment says 3/3 Ready. Grafana is green. Users get 504.

Kubernetes did not "lose the request." The request walked Client → Ingress → Service → Endpoints → Pod and fell off at a hop you did not look at. The most expensive K8s skill is not writing YAML — it is knowing which object is lying and which kubectl command makes it confess.

This page is a field guide: the objects, the path, the probes, then a diagnosis playbook for the failures that actually page you.

Abstraction Levels¶

The Objects That Matter¶

Requests / limits (the other page-causer):

• request — scheduler + bin-packing. Too low: noisy neighbor. Too high: Pending forever.
• limit — cgroup cap. CPU limit → throttle (latency). Memory limit → OOMKilled (restart).
• Unset limit: the node OOM killer picks a victim (maybe not you). Unset request: you are scheduled as if you were tiny.

Request Flow¶

flowchart LR
    U[Client] --> I[Ingress controller]
    I --> S[Service ClusterIP]
    S --> E[EndpointSlice]
    E --> P1[Pod A Ready]
    E --> P2[Pod B Ready]
    P3[Pod C Running unready] -.-> E
    style P3 fill:#e65100,color:#fff
    style E fill:#1565c0,color:#fff

1. DNS: api.shop.com → Ingress controller (or external LB).
2. Ingress matches host/path → Service name (not a pod).
3. kube-proxy (or dataplane: Cilium, Istio) NATs ClusterIP to a Ready endpoint.
4. Pod network: container port. App accepts.

If Endpoints is empty, the Service still has an IP. Packets go to a black hole or the LB returns 503. kubectl get pods can look perfect.

Interactive Simulation¶

Try: Fail readiness, then Send request. The pod is still "there." The Service will not send it work.

Probes — Three Different Questions¶

kubectl You Will Actually Type¶

kubectl get pods -o wide
kubectl describe pod $POD          # events at the bottom. Always the bottom.
kubectl logs $POD -c app
kubectl logs $POD --previous       # the crash you just missed
kubectl exec -it $POD -- sh
kubectl get events --sort-by=.lastTimestamp
kubectl top pod
kubectl get endpointslices -l kubernetes.io/service-name=$SVC
kubectl get svc,ing,ep -o wide

describe > dashboards when the object never became Ready. Events are the API server's diary: FailedScheduling, FailedMount, Unhealthy, Killing, Pulled.

Guided Diagnosis¶

Work top-down: schedule → pull → start → live → ready → route → serve.

Pending¶

• Look: describe pod → FailedScheduling.
• Causes: requests bigger than any node; taints; missing PVC; affinity; Insufficient cpu/memory.
• Move: kubectl describe nodes | grep -A5 Allocated; shrink requests or add nodes. Do not "just remove requests."

ImagePullBackOff¶

• Look: events: 401 Unauthorized, not found, tls.
• Causes: wrong tag, private registry without imagePullSecrets, rate limit (Docker Hub).
• Move: pull the exact image from a node; fix the secret; pin digest not :latest.

CrashLoopBackOff¶

• Look: logs --previous, describe restart count, exit code.
• Causes: bad config, missing secret, crash on boot, liveness too aggressive.
• Exit 137 often OOM; exit 1 is the app. Backoff is exponential — waiting is not a fix.

OOMKilled¶

• Look: Last State: Reason: OOMKilled, kubectl top, container limit.
• Causes: limit too low; leak; one request materializes a huge JSON.
• Move: raise limit and find the allocation. A higher limit without a request change can still evict neighbors.

Failed readiness probe¶

• Look: pod Running but 0/1 Ready; Endpoints empty.
• Causes: app still warming, wrong port/path, dependency down (if you wired it that way).
• Move: kubectl get ep; curl the probe from inside the pod. Users see 502; you see a green Deployment if minReadySeconds/replicas are sloppy.

DNS failure (in-cluster)¶

• Look: nslookup kubernetes.default from the pod; CoreDNS logs; ndots search path.
• Causes: CoreDNS down, network policy, using http://svc without namespace, Alpine musl + search domains.
• Move: FQDN svc.ns.svc.cluster.local; check kube-dns endpoints.

Unreachable Service¶

• Look: ClusterIP exists, get endpoints empty or endpoints exist but packets drop.
• Causes: selector labels ≠ pod labels (the classic typo); readiness; NetworkPolicy; kube-proxy / CNI bug.
• Move: diff labels. kubectl get pod --show-labels vs spec.selector.

Ingress failure¶

• Look: controller logs, Ingress address, HTTP 404 from nginx vs 502.
• Causes: wrong Service name/port (servicePort vs named port), no TLS secret, controller not watching the namespace, path Prefix vs Exact.
• Move: curl the controller pod directly; bypass DNS.

Where TLS terminates decides more than certificates. Terminate at the Ingress/Gateway and the connection from there to the Pod is plain HTTP by default — fine for most apps, wrong if you need mTLS all the way to the container (compliance, zero-trust) or if a downstream service trusts X-Forwarded-For/X-Real-IP without validating it came from a trusted proxy (that header is attacker-controlled if anything upstream of your edge doesn't strip and re-set it). Also check body-size and timeout defaults at the termination point — a Pod that accepts 50 MB uploads behind an Ingress controller defaulting to a 1 MB body limit fails with a controller-generated 413 that never reaches your app's logs.

High CPU¶

• Look: kubectl top pod, CPU throttle (container_cpu_cfs_throttled_seconds).
• Causes: limit too tight (throttle looks like "mystery latency"), real hot loop, HPA not firing (wrong metric).
• Move: throttle stats first; then profiles. HPA on CPU when you are I/O bound will not save you.

Memory pressure (node)¶

• Look: node condition MemoryPressure, evicted pods, describe node.
• Causes: sum of working sets > allocatable; cache; one burstable hog.
• Move: requests that match reality; PriorityClass for critical daemon; don't run CI on the same nodes as checkout.

Storage — Don't Treat Stateful Like Stateless¶

A Pod's local filesystem dies with the Pod. For anything that needs to survive a reschedule, three objects do the work:

Access modes matter more than the name suggests:

• ReadWriteOnce (RWO) — one node at a time. This is the default for block storage (EBS, PD). A StatefulSet pod that gets rescheduled to a new node has to wait for the volume to detach from the old node first — that's a real, sometimes multi-minute, availability gap.
• ReadWriteMany (RWX) — many nodes concurrently (NFS, EFS, Filestore). Needed for genuinely shared state; usually slower than block storage for random I/O.
• ReadOnlyMany (ROX) — many nodes, read-only. Good fit for shared config/reference data.

Volume expansion (allowVolumeExpansion: true on the StorageClass) lets you grow a PVC without recreating it — but the filesystem still needs an online resize (usually automatic on modern CSI drivers, but confirm for your provisioner) and expansion is one-directional; you can't shrink.

RBAC & Service Accounts¶

Every Pod runs as a ServiceAccount, and every ServiceAccount not explicitly configured runs as default in its namespace — which historically had broad-enough permissions in many clusters to be a real lateral-movement path if that Pod is compromised.

The least-privilege pattern:

1. Create a dedicated ServiceAccount per workload, not a shared one.
2. Grant a Role (namespace-scoped) or ClusterRole (cluster-scoped) with only the verbs/resources that workload actually needs — get/list/watch on its own ConfigMaps, not * on everything.
3. Bind them with a RoleBinding (or ClusterRoleBinding only when the access genuinely needs to span namespaces).
4. Set automountServiceAccountToken: false on Pods that never call the Kubernetes API at all — most application workloads don't need a token mounted.

What to audit: who can kubectl exec or port-forward into production Pods — those two verbs bypass every network-layer control (NetworkPolicy, Service, Ingress) and reach the container directly. Treat pods/exec RBAC grants with the same scrutiny as SSH access to a prod host, and log them — kubectl exec doesn't show up in application logs at all.

NetworkPolicy¶

By default, every Pod can talk to every other Pod in the cluster — there's no network isolation until you add a NetworkPolicy. This is genuinely useful for blast-radius containment (a compromised Pod in one namespace can't reach the payments database in another), and genuinely easy to get backwards.

The safe rollout pattern: start with an explicit allow-list — allow DNS egress, allow the specific dependencies a workload calls, allow ingress from the specific namespaces/Pods that call it — and apply it to one namespace at a time with monitoring, rather than a cluster-wide default-deny in one shot.

Realistic Example¶

Checkout: 3 replicas, readiness GET /ready → 200 after Redis ping. Redis flaps 2s.

• If readiness includes Redis: all 3 pods go unready → Endpoints empty → Ingress 502. You "failed closed" the whole app because a cache blinked.
• If liveness includes Redis: kubelet restarts all 3. Cold JVM + CrashLoop. Worse.
• If readiness is local (/ready = event loop alive) and Redis failures are handled in-process (degrade, timeout): traffic continues; checkout may be slower or use a fallback. That is usually what you wanted.

Failure Modes (Cluster Level)¶

Production Debugging¶

Symptom: Ingress 502, Deployment 3/3

1. kubectl get endpointslices -l kubernetes.io/service-name=checkout
   → empty? selector / readiness. not empty? hop is Ingress or CNI.
2. kubectl describe pod | tail
   → Unhealthy, OOMKilled, FailedMount
3. kubectl logs --previous
   → the crash that is no longer the current container
4. From a debug pod: curl checkout:80/ready  and  curl checkout.prod.svc:80
5. kubectl get ing -o yaml   vs  actual Service port name
6. Node: kubectl top node; describe node  (pressure, taints)

Trade-offs¶

Interview Questions¶

Key Takeaways¶

Previous: HTTP & TCP | Next: Sagas