Kubernetes¶

Prerequisites: HTTP & TCP, Load Balancing

← Cloud | Next: Observability →

Why This Exists¶

Your Deployment says 3/3 Ready. Grafana is green. Users get 504.

Kubernetes did not "lose the request." The request walked Client → Ingress → Service → Endpoints → Pod and fell off at a hop you did not look at. The most expensive K8s skill is not writing YAML — it is knowing which object is lying and which kubectl command makes it confess.

This page is a field guide: the objects, the path, the probes, then a diagnosis playbook for the failures that actually page you.

Abstraction Levels¶

The Objects That Matter¶

Requests / limits (the other page-causer):

• request — scheduler + bin-packing. Too low: noisy neighbor. Too high: Pending forever.
• limit — cgroup cap. CPU limit → throttle (latency). Memory limit → OOMKilled (restart).
• Unset limit: the node OOM killer picks a victim (maybe not you). Unset request: you are scheduled as if you were tiny.

Service Types¶

A Service is a stable name and, usually, a virtual IP. The type field changes who can open that name. It does not run the container.

Headless with a selector publishes each Ready pod as an address (db-0.db.ns.svc.cluster.local on a StatefulSet). Headless without a selector is how you attach your own Endpoints to an IP that lives outside the cluster. Neither case is load-balanced by kube-proxy. The client, or a StatefulSet peer, chooses.

LoadBalancer with no cloud controller stays <pending>. The ClusterIP still works from inside the cluster. The default NodePort range is 30000–32767 (--service-node-port-range). Do not memorize a port from a diagram onto a cluster that changed the range.

Ingress is the HTTP front door: host, path, TLS, then a Service name. A 404 from the controller never reached your process. Gateway API is the newer object with the same job; the hop you debug is still "which Service did this route name?"

Request Flow¶

flowchart LR
    U[Client] --> I[Ingress controller]
    I --> S[Service ClusterIP]
    S --> E[EndpointSlice]
    E --> P1[Pod A Ready]
    E --> P2[Pod B Ready]
    P3[Pod C Running unready] -.-> E
    style P3 fill:#e65100,color:#fff
    style E fill:#1565c0,color:#fff

1. DNS: api.shop.com → Ingress controller (or external LB).
2. Ingress matches host/path → Service name (not a pod).
3. kube-proxy (or dataplane: Cilium, Istio) NATs ClusterIP to a Ready endpoint.
4. Pod network: container port. App accepts.

If Endpoints is empty, the Service still has an IP. Packets go to a black hole or the LB returns 503. kubectl get pods can look perfect.

Interactive Simulation¶

Try: Fail readiness, then Send request. The pod is still "there." The Service will not send it work.

How the packet actually moves¶

Read the symptom against the panel, not against "Kubernetes networking" as one blob.

• Pod IP to pod IP fails, Service DNS still resolves. CNI or a NetworkPolicy. The Service object is not the broken part.
• The name resolves, curl to the ClusterIP times out. Empty EndpointSlice, or a policy that allows DNS and drops the app port.
• The FQDN works and the short name does not. Search path. ndots: 5 tries several svc.cluster.local suffixes before it treats a name as absolute.
• You can open the node and you cannot open the pod IP from your laptop. That is the design. Pod CIDRs are not the public address. HTTP from users goes through Ingress or a cloud load balancer, then a Service.

NetworkPolicy is off until a policy selects the pod. The first rule people forget is egress to CoreDNS on UDP and TCP 53. The error is lookup ... i/o timeout, and the policy object does not mention DNS.

Probes — Three Different Questions¶

kubectl You Will Actually Type¶

kubectl get pods -o wide
kubectl describe pod $POD          # events at the bottom. Always the bottom.
kubectl logs $POD -c app
kubectl logs $POD --previous       # the crash you just missed
kubectl exec -it $POD -- sh
kubectl get events --sort-by=.lastTimestamp
kubectl top pod
kubectl get endpointslices -l kubernetes.io/service-name=$SVC
kubectl get svc,ing,ep -o wide

describe > dashboards when the object never became Ready. Events are the API server's diary: FailedScheduling, FailedMount, Unhealthy, Killing, Pulled.

How a Pod Finds a Node¶

kubectl apply stores a pod with an empty spec.nodeName. Phase is Pending. Nothing is running.

1. Filter. Nodes that cannot take the pod are out: requests larger than allocatable CPU or memory, nodeSelector, required affinity, a taint with no toleration, a cordoned node, a volume that cannot attach in that zone.
2. Score. The survivors are ranked. Remaining capacity, topology spread, and preferred affinity are the usual reasons. A required rule that matches nobody never reaches this step.
3. Bind. The scheduler writes the node name. Assigned is not Running.
4. Kubelet. The kubelet on that node pulls the image, starts the containers, and runs probes. Ready is a probe result. The scheduler does not know if your process listens.

spec.schedulerName points at a custom scheduler. If that controller is down, the pod stays Pending and the default scheduler logs nothing, because it is not watching that pod.

Guided Diagnosis¶

Work top-down: schedule → pull → start → live → ready → route → serve. The picture above is the first three of those words.

Pending¶

• Look: describe pod → FailedScheduling.
• Causes: requests bigger than any node; taints; missing PVC; affinity; Insufficient cpu/memory.
• Move: kubectl describe nodes | grep -A5 Allocated; shrink requests or add nodes. Do not "just remove requests."

ImagePullBackOff¶

• Look: events: 401 Unauthorized, not found, tls.
• Causes: wrong tag, private registry without imagePullSecrets, rate limit (Docker Hub).
• Move: pull the exact image from a node; fix the secret; pin digest not :latest.

CrashLoopBackOff¶

• Look: logs --previous, describe restart count, exit code.
• Causes: bad config, missing secret, crash on boot, liveness too aggressive.
• Exit 137 often OOM; exit 1 is the app. Backoff is exponential — waiting is not a fix.

OOMKilled¶

• Look: Last State: Reason: OOMKilled, kubectl top, container limit.
• Causes: limit too low; leak; one request materializes a huge JSON.
• Move: raise limit and find the allocation. A higher limit without a request change can still evict neighbors.

Failed readiness probe¶

• Look: pod Running but 0/1 Ready; Endpoints empty.
• Causes: app still warming, wrong port/path, dependency down (if you wired it that way).
• Move: kubectl get ep; curl the probe from inside the pod. Users see 502; you see a green Deployment if minReadySeconds/replicas are sloppy.

DNS failure (in-cluster)¶

• Look: nslookup kubernetes.default from the pod; CoreDNS logs; ndots search path.
• Causes: CoreDNS down, network policy, using http://svc without namespace, Alpine musl + search domains.
• Move: FQDN svc.ns.svc.cluster.local; check kube-dns endpoints.

Unreachable Service¶

• Look: ClusterIP exists, get endpoints empty or endpoints exist but packets drop.
• Causes: selector labels ≠ pod labels (the classic typo); readiness; NetworkPolicy; kube-proxy / CNI bug.
• Move: diff labels. kubectl get pod --show-labels vs spec.selector.

Ingress failure¶

• Look: controller logs, Ingress address, HTTP 404 from nginx vs 502.
• Causes: wrong Service name/port (servicePort vs named port), no TLS secret, controller not watching the namespace, path Prefix vs Exact.
• Move: curl the controller pod directly; bypass DNS.

Where TLS terminates decides more than certificates. Terminate at the Ingress/Gateway and the connection from there to the Pod is plain HTTP by default — fine for most apps, wrong if you need mTLS all the way to the container (compliance, zero-trust) or if a downstream service trusts X-Forwarded-For/X-Real-IP without validating it came from a trusted proxy (that header is attacker-controlled if anything upstream of your edge doesn't strip and re-set it). Also check body-size and timeout defaults at the termination point — a Pod that accepts 50 MB uploads behind an Ingress controller defaulting to a 1 MB body limit fails with a controller-generated 413 that never reaches your app's logs.

High CPU¶

• Look: kubectl top pod, CPU throttle (container_cpu_cfs_throttled_seconds).
• Causes: limit too tight (throttle looks like "mystery latency"), real hot loop, HPA not firing (wrong metric).
• Move: throttle stats first; then profiles. HPA on CPU when you are I/O bound will not save you.

Memory pressure (node)¶

• Look: node condition MemoryPressure, evicted pods, describe node.
• Causes: sum of working sets > allocatable; cache; one burstable hog.
• Move: requests that match reality; PriorityClass for critical daemon; don't run CI on the same nodes as checkout.

Storage — Don't Treat Stateful Like Stateless¶

A Pod's local filesystem dies with the Pod. For anything that needs to survive a reschedule, three objects do the work:

Access modes matter more than the name suggests:

• ReadWriteOnce (RWO) — one node at a time. This is the default for block storage (EBS, PD). A StatefulSet pod that gets rescheduled to a new node has to wait for the volume to detach from the old node first — that's a real, sometimes multi-minute, availability gap.
• ReadWriteMany (RWX) — many nodes concurrently (NFS, EFS, Filestore). Needed for genuinely shared state; usually slower than block storage for random I/O.
• ReadOnlyMany (ROX) — many nodes, read-only. Good fit for shared config/reference data.

Volume expansion (allowVolumeExpansion: true on the StorageClass) lets you grow a PVC without recreating it — but the filesystem still needs an online resize (usually automatic on modern CSI drivers, but confirm for your provisioner) and expansion is one-directional; you can't shrink.

RBAC & Service Accounts¶

Every Pod runs as a ServiceAccount, and every ServiceAccount not explicitly configured runs as default in its namespace — which historically had broad-enough permissions in many clusters to be a real lateral-movement path if that Pod is compromised.

The least-privilege pattern:

1. Create a dedicated ServiceAccount per workload, not a shared one.
2. Grant a Role (namespace-scoped) or ClusterRole (cluster-scoped) with only the verbs/resources that workload actually needs — get/list/watch on its own ConfigMaps, not * on everything.
3. Bind them with a RoleBinding (or ClusterRoleBinding only when the access genuinely needs to span namespaces).
4. Set automountServiceAccountToken: false on Pods that never call the Kubernetes API at all — most application workloads don't need a token mounted.

What to audit: who can kubectl exec or port-forward into production Pods — those two verbs bypass every network-layer control (NetworkPolicy, Service, Ingress) and reach the container directly. Treat pods/exec RBAC grants with the same scrutiny as SSH access to a prod host, and log them — kubectl exec doesn't show up in application logs at all.

NetworkPolicy¶

By default, every Pod can talk to every other Pod in the cluster — there's no network isolation until you add a NetworkPolicy. This is genuinely useful for blast-radius containment (a compromised Pod in one namespace can't reach the payments database in another), and genuinely easy to get backwards.

The safe rollout pattern: start with an explicit allow-list — allow DNS egress, allow the specific dependencies a workload calls, allow ingress from the specific namespaces/Pods that call it — and apply it to one namespace at a time with monitoring, rather than a cluster-wide default-deny in one shot.

Realistic Example¶

Checkout: 3 replicas, readiness GET /ready → 200 after Redis ping. Redis flaps 2s.

• If readiness includes Redis: all 3 pods go unready → Endpoints empty → Ingress 502. You "failed closed" the whole app because a cache blinked.
• If liveness includes Redis: kubelet restarts all 3. Cold JVM + CrashLoop. Worse.
• If readiness is local (/ready = event loop alive) and Redis failures are handled in-process (degrade, timeout): traffic continues; checkout may be slower or use a fallback. That is usually what you wanted.

Failure Modes (Cluster Level)¶

Production Debugging¶

Symptom: Ingress 502, Deployment 3/3

1. kubectl get endpointslices -l kubernetes.io/service-name=checkout
   → empty? selector / readiness. not empty? hop is Ingress or CNI.
2. kubectl describe pod | tail
   → Unhealthy, OOMKilled, FailedMount
3. kubectl logs --previous
   → the crash that is no longer the current container
4. From a debug pod: curl checkout:80/ready  and  curl checkout.prod.svc:80
5. kubectl get ing -o yaml   vs  actual Service port name
6. Node: kubectl top node; describe node  (pressure, taints)

Trade-offs¶

Interview Questions¶

Key Takeaways¶

Previous: HTTP & TCP | Next: Sagas