supraj.dev

Interview questions & answers

Kubernetes

Pods, scheduling, networking, and the control plane.

  • 164 questions
  • 8 easy
  • 68 medium
  • 88 hard
  • 4 related articles

164 questions

Question 1Easy

What is the difference between a Deployment and a StatefulSet?

#
Show answer

A Deployment manages stateless, interchangeable replicas with random pod names and a shared scaling behavior. A StatefulSet gives each pod a stable, ordered identity (pod-0, pod-1…), stable network IDs, and stable persistent volumes — used for databases, Kafka, and anything that needs predictable identity or ordered rollout.

Question 2Easy

What is a Service and what are the main types?

#
Show answer

A Service gives a stable virtual IP/DNS name in front of a changing set of pods (selected by labels). Types: ClusterIP (internal only, default), NodePort (exposes a port on every node), LoadBalancer (provisions a cloud LB), and ExternalName (maps to a DNS CNAME). Ingress sits above Services for HTTP routing.

Question 3Medium

How does the scheduler decide where to place a pod?

#
Show answer

In two phases: filtering (predicates) removes nodes that can't run the pod — insufficient resources, taints without matching tolerations, node selectors/affinity, volume topology. Then scoring ranks the remaining nodes (spreading, least-requested, affinity weights) and the highest-scoring node wins. You influence it with requests, nodeAffinity, podAffinity/anti-affinity, taints/tolerations, and topology spread constraints.

Question 4MediumControl plane

What happens, step by step, when you run kubectl apply?

#
Show answer

kubectl sends the manifest to the API server, which authenticates, authorizes (RBAC), runs admission controllers, validates, and persists the desired state in etcd. The relevant controller (e.g. Deployment controller) notices the new desired state and creates ReplicaSets/Pods. The scheduler assigns each pod to a node. The kubelet on that node pulls images and starts containers via the container runtime, then reports status back to the API server.

The sequence described in this answer
  1. kubectl sends the manifest
  2. API server authenticates, authorizes, admits and persists
  3. Controller creates ReplicaSets and Pods
  4. Scheduler assigns a node
  5. Kubelet starts containers and reports status

Check your reasoning

  • Explain the API server checks before persistence.
  • Distinguish the controller, scheduler and kubelet responsibilities.
Question 5HardTroubleshooting

A pod is stuck in CrashLoopBackOff. How do you debug it?

#
Show answer

Start with kubectl describe pod to see events and the last state/exit code. kubectl logs <pod> --previous shows the crashed container's output. Common causes: bad command/entrypoint, missing config/secret, failing readiness vs liveness probe killing it too early, OOMKilled (check exit code 137 and memory limits), or a dependency it can't reach. Fix probes/limits, or run kubectl debug / an ephemeral container to inspect the filesystem and environment.

Command excerpt from the existing answer; replace the pod placeholder · shell
kubectl logs <pod> --previous
Question 6Hard

A pod is running but users see 503 errors. How do you debug it?

#
Show answer

A running pod only means the container is alive — not that the application is healthy. Start by checking: (1) Service endpoints — kubectl get endpoints to verify the pod is an endpoint target. (2) Pod labels vs Service selectors — mismatched labels mean traffic routes nowhere. (3) Readiness probes — if failing, the pod is removed from the Service even though it's running. (4) Ingress configuration — verify routing rules, TLS, and that the backend Service name/port match. (5) Application logs — the app may be crashing on each request or failing its startup dependency.

Question 7Hard

How does pod-to-pod networking work without NAT?

#
Show answer

Kubernetes requires a flat network where every pod gets a unique IP and can reach every other pod directly, no NAT. A CNI plugin (Calico, Cilium, etc.) implements this — assigning pod IPs and programming routes or an overlay (VXLAN/eBPF). Services are virtual: kube-proxy (or eBPF in Cilium) load-balances the Service ClusterIP to backing pod IPs via iptables/IPVS rules. NetworkPolicies then restrict which pods may talk to which.

Question 8Easy

What is the difference between a DaemonSet and a Deployment?

#
Show answer

A DaemonSet runs one pod on every eligible node, or a subset selected by node labels/affinity, and is used for node-level agents like log collectors, Node Exporter, CNI agents, and CSI node plugins. A Deployment manages a replica count for stateless application pods. Modern DaemonSet pods still go through the scheduler, so taints, tolerations, affinity, and resource requests matter. Rolling updates are node-oriented and controlled with maxUnavailable or maxSurge depending on cluster support and configuration.

Question 9Easy

What is a Namespace and why would you use multiple namespaces?

#
Show answer

A Namespace partitions a cluster into virtual sub-clusters. They provide scope for resource names (no DNS conflicts), RBAC boundaries (restrict who can access what), resource quotas (per-namespace CPU/memory caps), and NetworkPolicy isolation. Common patterns: separate namespaces per team (team-a, team-b), per environment (dev, staging, prod), or per application. Many objects (nodes, PVs) are cluster-scoped and don't belong to any namespace. kubectl get namespaces lists all.

Question 10Easy

What is the purpose of a ResourceQuota, and what resource types can it limit?

#
Show answer

A ResourceQuota sets aggregate hard limits on resource consumption within a namespace. It prevents a single team from starving others. Limits include: compute (requests.cpu, requests.memory, limits.cpu, limits.memory), storage (requests.storage, persistentvolumeclaims), object counts (pods, services, configmaps, secrets, replicationcontrollers), and extended resources (GPUs, FPGAs). Exceeding a quota causes the API server to reject the operation with a 403 Forbidden. View with kubectl describe resourcequota -n <ns>.

Question 11Easy

How do you restrict a container's resource usage, and what happens when it exceeds its limit?

#
Show answer

Set resources.requests (guaranteed minimum) and resources.limits (hard cap) per container. CPU is throttled above the limit; memory exceeding the limit causes OOMKill (container terminates, kubelet restarts it per restartPolicy). requests help the scheduler pick a node with enough capacity. The QoS class is determined by the ratio: Guaranteed (limits == requests), Burstable (requests < limits, or requests set but limits omitted), BestEffort (neither set). Use kubectl describe pod to see QoS class and OOM score.

Question 12Easy

What is a ConfigMap and how does it differ from a Secret?

#
Show answer

Both store configuration data as key-value pairs. ConfigMap stores plaintext data (max 1 MiB) intended for non-sensitive config like env vars, config files, or command-line arguments. Secret stores sensitive data (base64-encoded in YAML, automatically decrypted when injected) with optional encryption at rest if the cluster has a KMS provider configured. Secrets support additional access control via RBAC. Best practice: use ConfigMap for app config, Secrets for passwords/tokens/keys. Both can be mounted as volumes or env vars.

Question 13Medium

How does kube-proxy work and what are the differences between iptables, IPVS, and eBPF modes?

#
Show answer

kube-proxy watches the API server for Service and EndpointSlice changes, then programs network rules so traffic to a Service's ClusterIP reaches backing pods. iptables mode: creates a chain of iptables rules per Service — O(n) linear traversal for large clusters, updates require rewriting all rules (slow for 5000+ Services). IPVS mode (default in many distributions): uses Linux kernel IP Virtual Server tables, O(1) lookup per packet, much faster for large clusters. Supports more scheduling algorithms (round-robin, least-connection, source-hash). eBPF mode (via Cilium): replaces kube-proxy entirely, attaches BPF programs to network interfaces at the kernel level, bypassing iptables/IPVS entirely. Offers the lowest latency, per-packet policy enforcement, and native NetworkPolicy without separate iptables rules. Cilium's eBPF kube-proxy replacement is the modern recommended approach for greenfield clusters.

Question 14Medium

How do you restrict pod-to-pod traffic using NetworkPolicy? Walk through an example that blocks all ingress except from a specific label.

#
Show answer

NetworkPolicy is a Kubernetes resource that selects pods via podSelector and defines ingress/egress rules. Default is 'allow all' unless a policy selects a pod — once any policy matches a pod, only traffic conforming to the rules is allowed. Example blocking all ingress except from pods labeled app: frontend: kind: NetworkPolicy; metadata: {name: deny-all-except-frontend}; spec: {podSelector: {matchLabels: {app: api}}, ingress: [{from: [{podSelector: {matchLabels: {app: frontend}}}]}]}. The from can specify podSelector, namespaceSelector, ipBlock, or any combination. egress rules restrict outbound traffic similarly. CNI support varies: Calico, Cilium, and Weave Net support full enforcement; some CNIs (Flannel) don't support NetworkPolicy at all and require a separate network policy engine.

Question 15Medium

What are init containers, and what problems do they solve compared to regular containers?

#
Show answer

Init containers run sequentially before the app containers in a pod start. Each init container must exit successfully (exit code 0) before the next one starts. If any init container fails, Kubernetes restarts the entire pod. Use cases: (1) waiting for a dependency to be ready (e.g., until pg_isready; do sleep 1; done); (2) downloading or preparing files into shared volumes before the app starts; (3) running privileged setup operations (e.g., setting sysctls, modifying kernel params) while the app container runs with reduced capabilities. Init containers can have different resource limits, images, and volume mounts than app containers.

Question 16Medium

How do you implement canary deployments in Kubernetes without a service mesh?

#
Show answer

Without a service mesh, the simplest weighted canary is two Deployments behind one Service selector that matches both versions, for example app: api, while the pods carry different version labels. Traffic share is roughly proportional to ready pod count, so 9 stable pods and 1 canary pod is about 10% for new connections. This is coarse and connection-dependent. For explicit HTTP percentages, use an ingress controller feature such as NGINX canary annotations, or a progressive delivery controller like Argo Rollouts or Flagger with an ingress provider. Keep rollback simple: scale canary to zero or remove it from the Service selector.

Question 17Medium

What is the difference between a PersistentVolume and a PersistentVolumeClaim, and how does dynamic provisioning add storage on demand?

#
Show answer

A PersistentVolume (PV) is a storage resource in the cluster — an abstraction over actual storage (EBS, GCE PD, NFS, etc.). It has a capacity, access mode (ReadWriteOnce, ReadOnlyMany, ReadWriteMany), and reclaim policy (Retain, Delete, Recycle). A PersistentVolumeClaim (PVC) is a request for storage by a user — it specifies desired size and access mode. Kubernetes binds a matching PVC to a PV (or waits if none match). Dynamic provisioning uses a StorageClass: when a PVC references a StorageClass with a provisioner, the CSI driver automatically creates the underlying storage volume and a PV object. This eliminates manual PV pre-creation. The reclaimPolicy on the StorageClass determines what happens to the volume when the PVC is deleted.

Question 18Medium

How does the HorizontalPodAutoscaler (HPA) work, and what are the key parameters for tuning it?

#
Show answer

The HPA controller reads CPU, memory, custom, or external metrics through Kubernetes metrics APIs and computes desired replicas roughly as ceil(currentReplicas * currentMetric / targetMetric). It ignores small changes inside the tolerance band, commonly 10%, to avoid flapping. Important tuning lives under spec.behavior: scale-up policies limit how fast replicas can increase, scale-down policies and scaleDown.stabilizationWindowSeconds prevent rapid scale-down. The usual default scale-down stabilization is 300 seconds, while scale-up stabilization is commonly 0 unless configured. For custom metrics, you need a working adapter such as Prometheus Adapter or Datadog Cluster Agent, and requests must be set correctly for CPU utilization-based scaling.

Question 19Medium

What is a PodDisruptionBudget, and when should you use one for a stateful application?

#
Show answer

A PodDisruptionBudget (PDB) limits the number of replicas of a set that can be voluntarily disrupted at a time. It does NOT protect against involuntary disruptions (hardware failure, kernel panic). PDBs are configured with minAvailable (e.g., minAvailable: 80%) or maxUnavailable (e.g., maxUnavailable: 1). Before an eviction (e.g., node drain via kubectl drain, cluster autoscaler scale-down), the disruption controller checks if evicting the pod would violate the PDB. If so, the eviction is blocked. For stateful applications (databases, message queues), set a PDB with maxUnavailable: 1 to ensure quorum during rolling updates or node maintenance. The PDB interacts with the cluster autoscaler: if a node needs to drain and the PDB blocks eviction, the autoscaler may fail to scale down.

Question 20Medium

How does DNS resolution work inside a Kubernetes cluster? Describe the full lookup path for curl myservice.myns.svc.cluster.local.

#
Show answer

CoreDNS (default since K8s 1.13) runs as a cluster DNS Service (typically kube-dns or coredns). A pod's /etc/resolv.conf points nameserver to the CoreDNS ClusterIP and sets search domains: <pod-namespace>.svc.cluster.local, svc.cluster.local, cluster.local, <cloud-region>.compute.internal. For curl myservice.myns.svc.cluster.local, the OS resolver queries CoreDNS, which matches the cluster domain and returns the ClusterIP. CoreDNS uses Headless Service endpoints directly (A/AAAA records for each pod IP via endpoint slices). For ExternalName Services, CoreDNS returns a CNAME. The search domains allow short names: curl myservice resolves to myservice.<pod-ns>.svc.cluster.local. CoreDNS supports stub domains (custom upstreams), prometheus metrics, and autoscaling via cluster-proportional-autoscaler. Troubleshoot with kubectl run dnsutils --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3 then nslookup kubernetes.default.

Question 21Hard

How does etcd work under the hood — leader election, Raft consensus, quorum, and data storage?

#
Show answer

etcd is a distributed key-value store based on the Raft consensus algorithm. Leader election: peers elect a leader via randomized timer-based voting. The leader handles all client writes and sends heartbeats (default 100ms) to followers. Quorum: a write succeeds only if a majority of nodes (N/2 + 1) acknowledge it. For a 3-node cluster, 2 nodes must confirm; writes are durable once committed to the majority. Reads: by default, reads are served from the leader's state machine (linearizable, consistent). With --quorum-backend-read, followers can serve reads but may return stale data. Data storage: etcd stores data in a persistent bbolt database on disk. Keys are organized in a B+ tree; writes go to a write-ahead log (WAL) for durability, then the in-memory tree, then snapshots. MVCC: each modification creates a new revision; the --compact flag reclaims old revisions. The Kubernetes API server stores all cluster state in etcd under /registry/. Snapshot size grows with cluster state; etcd defragmentation (etcdctl defrag) reclaims space. Best practices: 3-5 nodes, SSD storage, regular backups (etcdctl snapshot save), TLS client-to-server and peer-to-peer encryption.

Question 22Hard

A node is in NotReady state. Walk through the full debugging process from kubectl to kernel diagnostics.

#
Show answer

Step 1: kubectl describe node <node> and inspect Conditions, events, taints, and last heartbeat. Ready=Unknown usually means the control plane stopped hearing from kubelet; Ready=False means kubelet reported a problem. Also check DiskPressure, MemoryPressure, and PIDPressure. Step 2: on the node, check systemctl status kubelet, journalctl -u kubelet, container runtime health with crictl ps, and certificate or auth errors. Step 3: check resources and kernel signals: df -h, inode usage, free -h, CPU load, process count, CNI logs, routes, firewall rules, and DNS/API reachability. Step 4: verify control-plane health through component pods or managed-service status rather than relying on deprecated kubectl get cs. Recover by fixing the specific issue, cordoning/draining if needed, or replacing the node. Delete the Node object only when you understand whether the underlying machine should rejoin or be rebuilt.

Question 23Hard

How would you design a Kubernetes cluster for multi-tenancy with strong workload isolation across tenants, including network, compute, and RBAC isolation?

#
Show answer

There is no single multi-tenancy feature in K8s — it's a layered approach. (1) Namespaces per tenant: each tenant gets one or more namespaces with a ResourceQuota and LimitRange. (2) RBAC: create a ClusterRole with limited permissions (read-only on nodes, ability to create Deployments/Services in own namespace) and bind it via RoleBinding in each tenant namespace. Use ClusterRoleBinding only for cluster-wide access. (3) Network isolation: a default-deny NetworkPolicy in every namespace plus permissive ingress/egress policies per tenant. For cross-tenant isolation, use a CNI with native policy enforcement (Cilium, Calico). Cilium supports CiliumNetworkPolicy with layer 7 rules (HTTP methods, paths). (4) Compute isolation: use ResourceQuota per tenant namespace for hard caps. For CPU/memory guarantees, set requests == limits (Guaranteed QoS) for critical workloads. Use RuntimeClass (gVisor, Kata Containers) for untrusted tenant workloads — each tenant pod runs in a lightweight VM for kernel isolation. (5) Storage isolation: each tenant PV/PVC should be in their namespace; CSI drivers can enforce volume ownership. (6) vCluster: for true control-plane isolation, each tenant gets a virtual cluster (loft.sh/vcluster) — a full K8s API server backed by the host cluster. This adds overhead but provides near-complete isolation. No single approach is perfect; choose isolation level based on threat model and complexity budget.

Question 24Hard

What happens when a node runs out of PID pressure or inode pressure? How does the kubelet detect and react to these conditions?

#
Show answer

The kubelet monitors system-level pressure indicators via its node condition controller. By default, it checks nodefs.inodesFree (inode pressure) and pid.available (PID pressure) via the cgroup stats. PID pressure: if the node's PID limit (kernel kernel.pid_max, typically 32768) is reached, the kernel cannot fork new processes — containers crash with "resource temporarily unavailable". The kubelet sets NodeCondition.PIDPressure = True and begins evicting pods (lowest priority first, using the same eviction order as memory/disk). Inode pressure: depletion of inodes means no new files can be created (even if disk space is available). Causes: a rogue container writing millions of small log files or a Docker overlay2 graph driver accumulating orphan layers. Detection: nodefs.inodesFree based on kubelet's eviction signals. Reaction: kubelet sets DiskPressure = True and evicts pods. Monitoring: kubectl top node doesn't show PID/inode metrics — use node_exporter Prometheus metrics (node_namespaces_pid, node_filesystem_files_free). Prevention: set kubelet --eviction-hard=nodefs.inodesFree<5%,pid.available<10% or similar thresholds. Pod-level PID limiting via --pod-max-pids (kubelet flag) or cgroup v2 pids.max controller.

Question 25Hard

How does the Kubernetes API server handle concurrent writes and what conflict resolution mechanism prevents lost updates?

#
Show answer

The API server uses optimistic concurrency based on resourceVersion (a monotonically increasing integer from etcd's mod revision). When a client creates an object, it gets a resourceVersion. On update, the PUT request must include the resourceVersion of the object being updated. The API server compares the resourceVersion in the request against the current version stored in etcd — if they differ (meaning another client updated the object in the meantime), the API server returns a 409 Conflict error. The client must re-fetch the object and retry the operation. This prevents lost updates: without it, two concurrent clients could overwrite each other's changes silently. The kubectl apply command handles this internally via server-side apply (SSA) — it uses PATCH with application/apply-patch+yaml content type and the field manager pattern. SSA tracks which field manager owns each field, so overlapping patches merge correctly rather than overwrite. The resourceVersion check is also used for watches: the watch request includes resourceVersion and the API server streams events only from that version onward, ensuring no events are missed. The watch reconnects with the latest version on timeout. For high-contention resources, use kubectl replace (replace entire object) or strategic merge patch (surgical field update without needing full object). The fieldValidation flag (K8s 1.25+) rejects unknown fields at the API server level.

Question 26Hard

What is the difference between a mutating admission webhook and a validating admission webhook? Walk through the full request lifecycle.

#
Show answer

Admission webhooks intercept API requests after authentication/authorization but before persistence. MutatingAdmissionWebhook can modify the object in flight: it receives an AdmissionReview with the full object, can patch it, and returns the modified version. Use cases: injecting sidecars (Istio, Linkerd), setting default resource limits, adding node selectors, validating that namespaces have correct labels. ValidatingAdmissionWebhook cannot modify the object — it only accepts or rejects. Use cases: enforcing policies (OPA/Gatekeeper, Kyverno), checking compliance (e.g., forbid latest image tags, require specific labels). Full lifecycle: (1) Authentication/Authorization (RBAC); (2) MutatingAdmissionWebhooks (run in order, can modify object); (3) Object schema validation; (4) ValidatingAdmissionWebhooks; (5) ResourceQuota and LimitRanger admission; (6) Persist to etcd. Webhooks receive: AdmissionReview JSON with request.operation (CREATE, UPDATE, DELETE, CONNECT), request.object (new state), request.oldObject (for UPDATE/DELETE). Response: response.allowed = true/false, response.patch (base64-encoded JSON patch, mutating only), response.status.message (error message on rejection). Control with failurePolicy: Ignore (cluster survives if webhook is down) or FailurePolicy: Fail (cluster blocks requests if webhook unreachable). reinvocationPolicy: IfNeeded re-runs mutating webhooks after other objects are created/modified. Ordering: MutatingWebhookConfiguration rules have failurePolicy and matchPolicy; ordering within multiple mutating webhooks is not guaranteed in K8s < 1.15 (fixed with webhook ordering via names). TLS: webhooks require a TLS server certificate signed by a CA the API server trusts. Deploy webhooks as a Service backed by pods; the API server connects to the webhook Service endpoint. The kube-apiserver or aggregator can be configured with --admission-control-config-file for fine-grained webhook routing. Use objectSelector to restrict which objects trigger the webhook (label-based filtering) to reduce latency overhead. Watch out for deadlock: a mutating webhook that modifies a ConfigMap that the webhook itself reads can cause infinite loops. Prevent by setting reinvocationPolicy: Never.

Question 27Hard

How do you migrate workloads from one node pool to another in a production cluster with zero downtime?

#
Show answer

Zero-downtime migration requires draining nodes gradually while ensuring enough capacity and respecting PDBs. Step 1: cordon all source pool nodes (kubectl cordon <node>) — no new pods are scheduled. Step 2: drain nodes one at a time (kubectl drain <node> --ignore-daemonsets --delete-emptydir-data). The --ignore-daemonsets flag keeps DaemonSet pods running (they'll be rescheduled by the DaemonSet controller on the new pool). --delete-emptydir-data acknowledges loss of ephemeral data. Step 3: before draining each node, verify the target pool has enough allocatable capacity — use kubectl describe nodes on target pool nodes. Step 4: ensure PodDisruptionBudgets are configured so stateful workloads aren't evicted below quorum. Step 5: for stateful workloads with PVCs, ensure the new pool has the same storage class and topology zones — otherwise PVC binding fails. Use azcopy or Velero for PV migration if the new pool is in a different zone. Step 6: monitor rollout with kubectl get pods -o wide -w and dashboards. Step 7: after all pods have migrated, scale down the source pool (cloud provider) or delete nodes. Alternatives: (a) PDB-respecting eviction API — write a script that iterates pods and calls kubectl evict which respects PDBs. (b) For cluster autoscaler managed pools: taint the source pool with CriticalAddonsOnly=true:NoSchedule and set up a temporary node group on the target pool. (c) Use descheduler to proactively move pods from source pool based on label/taint-based eviction strategies.

Question 28Hard

What is the role of the cloud-controller-manager (CCM), and how does it interact with nodes, Services, and routes?

#
Show answer

The cloud-controller-manager is a Kubernetes controller that embeds cloud-specific control logic into the control plane. It runs as a set of loops: Node controller: watches nodes and annotates them with cloud provider info (instance type, region, zone, provider ID). When a node is unreachable, the CCM checks the cloud API to determine if the VM is healthy — if deleted, the CCM deletes the Kubernetes node object. This prevents orphaned node objects. Route controller: configures cloud networking routes so pods on different nodes can communicate without overlay (used with legacy cloud provider route-based networking, e.g., GCP routes, AWS VPC route tables). For modern clusters using CNI (Calico, Cilium), the route controller is unnecessary. Service controller: provisions cloud load balancers (AWS NLB/ALB, GCP TCP/HTTP LB, Azure LB) when a Service of type LoadBalancer is created. It also manages health checks, security group rules (AWS), and firewall rules (GCP). CCM communicates with the cloud API via IAM/IRSA credentials. It does NOT manage compute resources (VM creation) — that's the cluster autoscaler or cloud provider. In-tree cloud providers (gce, aws, azure) were removed upstream in K8s 1.27+; all clusters must now run out-of-tree CCMs. The CCM interacts with the API server via normal informers and workqueues. Failure of the CCM means LoadBalancer Services stop provisioning and node lifecycle events (instance termination) aren't handled.

Question 29Hard

You deploy a ClusterIP Service, but pods can't reach it by name or ClusterIP. Walk through the full debugging flow.

#
Show answer

Step 1: verify the Service exists and has endpoints: kubectl get svc my-svc and kubectl get endpoints my-svc. If endpoints are empty, the pod selector doesn't match — check kubectl describe svc my-svc for Selector. Step 2: verify pods are running and have the matching labels: kubectl get pods --show-labels. Step 3: inside a pod, test connectivity: kubectl exec <pod> -- curl -v <cluster-ip>:<port>. If curl hangs (no route), check the service chain. Step 4: check kube-proxy: kubectl logs -n kube-system kube-proxy-<hash>. If kube-proxy is not running, iptables/IPVS rules aren't updated. Step 5: on a node, check iptables rules: iptables-save | grep my-svc or ipvsadm -L -n. For IPVS: verify the virtual server exists with correct pod endpoints. Step 6: check kernel conntrack: conntrack -L | grep <cluster-ip>. Conntrack entries can be stale. Step 7: check net.ipv4.ip_forward=1 on all nodes. Step 8: check bridge-nf-call-iptables=1 (sysctl net.bridge.bridge-nf-call-iptables) — required for traffic through bridge interfaces to be filtered by iptables. Step 9: check CNI plugin health — kubectl get pods -n kube-system | grep cilium|calico|weave. A misconfigured CNI drops packets before they reach kube-proxy rules. Step 10: test Service DNS: kubectl exec <pod> -- nslookup my-svc. If DNS fails, check CoreDNS: kubectl logs -n kube-system -l k8s-app=kube-dns. Step 11: enable kube-proxy debug logging: edit the DaemonSet, add --v=4 to the kube-proxy args. This prints all iptables updates.

Question 30Hard

How does Kubernetes garbage collection work for owner references? What happens when a parent object is deleted while child objects are still being created?

#
Show answer

Owner references (metadata.ownerReferences) establish parent-child relationships between API objects. When a parent object is deleted, the garbage collector (GC) automatically deletes all child objects. The GC runs as a controller in the kube-controller-manager. Deletion modes: Foreground (default for some resources): GC first deletes all dependents (or adds orphanDependents finalizer to parent), then deletes the parent. The parent remains visible with a deletionTimestamp and a foregroundDeletion finalizer until all children are deleted. Background (default): GC propagates deletion concurrently — parent is deleted immediately, children are deleted by a background worker. Orphan: parent is deleted without deleting children (set deleteOptions.orphanDependents=true). When a parent is deleted while children are being created: if the child is created after the parent's deletionTimestamp, the GC still sees the owner reference (via the parent's UID) and deletes the child. Race condition: if a parent is deleted and a controller creates a child referencing the deleted parent, the child may appear briefly and then be garbage-collected. Use DeletionPropagation with foreground to avoid orphan races. The GC controller uses informers with caches; there's a small window between parent deletion and child deletion (bounded by GC sync period, default 1h for some resourced). Immediate deletion: set --controllers=garbagecollector --concurrent-gc-syncs=20 on the controller-manager. The GC uses UID references, not name references — so renaming a parent doesn't affect children. Shared owners: a single child can have multiple owner references (e.g., a custom resource owned by both a user and a system); all owners must exist for the child to survive. Cascade deletion of Jobs creates Pods: deleting a Job deletes its Pods via the owner reference (set by the Job controller). Prevent accidental cascade: use kubectl delete --cascade=orphan for specific cleanup scenarios.

Question 31Hard

Explain the Kubernetes Control Plane high availability architecture. How does leader election work for the controller-manager and scheduler?

#
Show answer

An HA control plane runs multiple replicas of the API server, controller-manager, and scheduler, typically across 3 or 5 availability zones. API servers are stateless — they front etcd which runs as a Raft cluster (3 or 5 nodes). API servers are load-balanced via a TCP load balancer or round-robin DNS. Controller-manager and scheduler are stateless but use leader election — only one instance actively operates at a time. Leader election mechanism: each replica creates a Lease object (or uses Endpoint in older versions) in the kube-system namespace. The Lease has a holderIdentity, leaseDurationSeconds (default 15s), renewTime, and acquireTime. Each candidate tries to update the Lease with its identity using an optimistic lock (resourceVersion). The first to succeed becomes leader; it renews the Lease before leaseDurationSeconds expires. If the leader fails, other candidates acquire the Lease after leaseDeadline (default 15s) + retryPeriod (default 2s). The scheduler and controller-manager each have separate Leases. Config: --leader-elect=true (default), --leader-elect-lease-duration=15s, --leader-elect-retry-period=3s, --leader-elect-renew-deadline=10s. The API server does NOT use leader election — all API server instances are active simultaneously. They share a single etcd cluster. The API server's cache (watch-based informer) ensures consistency across instances. etcd itself runs a separate Raft leader election with --election-timeout (default 1000ms, i.e., 1 second). The API server only talks to the etcd leader via the etcd client's built-in redirect. If the etcd leader fails, a new leader is elected in ~1 second, and API writes are briefly blocked during this window.

Question 32Hard

Kubernetes incident: A Deployment rollout reaches 100% updated replicas but users get intermittent 503s. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check kubectl rollout status, ReplicaSet history, Service endpoints, readiness probe results, app metrics, and ingress/controller logs. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Use readiness probes that prove real serving health, set maxUnavailable: 0 and a controlled maxSurge for critical APIs, keep revisionHistoryLimit for rollback, and gate rollout on user-facing metrics. Do not treat updated replicas as proof of healthy traffic; verify endpoint membership and error rates.

Question 33Hard

Kubernetes architecture scenario: Design a deployment strategy for stateless APIs that supports fast rollback and zero-downtime releases. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Use readiness probes that prove real serving health, set maxUnavailable: 0 and a controlled maxSurge for critical APIs, keep revisionHistoryLimit for rollback, and gate rollout on user-facing metrics. Do not treat updated replicas as proof of healthy traffic; verify endpoint membership and error rates.

Question 34Medium

Kubernetes security scenario: Developers can patch live Deployments directly and bypass the approved release path. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Use readiness probes that prove real serving health, set maxUnavailable: 0 and a controlled maxSurge for critical APIs, keep revisionHistoryLimit for rollback, and gate rollout on user-facing metrics. Do not treat updated replicas as proof of healthy traffic; verify endpoint membership and error rates.

Question 35Hard

Kubernetes release scenario: Move a critical service from recreate deploys to rolling or canary deploys. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Use readiness probes that prove real serving health, set maxUnavailable: 0 and a controlled maxSurge for critical APIs, keep revisionHistoryLimit for rollback, and gate rollout on user-facing metrics. Do not treat updated replicas as proof of healthy traffic; verify endpoint membership and error rates.

Question 36Medium

Kubernetes reliability/cost scenario: Rollouts take too long because readiness gates and surge settings are conservative. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Use readiness probes that prove real serving health, set maxUnavailable: 0 and a controlled maxSurge for critical APIs, keep revisionHistoryLimit for rollback, and gate rollout on user-facing metrics. Do not treat updated replicas as proof of healthy traffic; verify endpoint membership and error rates.

Question 37Hard

Kubernetes incident: Pods restart during startup because the liveness probe fails before the application is ready. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check kubectl describe pod, previous container logs, probe events, application startup timings, and endpoint readiness changes. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Use startupProbe to protect slow boot, readiness to control traffic, and liveness only for unrecoverable deadlocks. Health endpoints should be cheap, deterministic, and sanitized. Tune periodSeconds, timeoutSeconds, and failure thresholds from measured startup and dependency behavior.

Question 38Hard

Kubernetes architecture scenario: Design health checks for an application with slow warmup and external dependencies. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Use startupProbe to protect slow boot, readiness to control traffic, and liveness only for unrecoverable deadlocks. Health endpoints should be cheap, deterministic, and sanitized. Tune periodSeconds, timeoutSeconds, and failure thresholds from measured startup and dependency behavior.

Question 39Medium

Kubernetes security scenario: Health endpoints expose dependency names, versions, or internal errors. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Use startupProbe to protect slow boot, readiness to control traffic, and liveness only for unrecoverable deadlocks. Health endpoints should be cheap, deterministic, and sanitized. Tune periodSeconds, timeoutSeconds, and failure thresholds from measured startup and dependency behavior.

Question 40Hard

Kubernetes release scenario: Introduce startup, readiness, and liveness probes to an existing production service. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Use startupProbe to protect slow boot, readiness to control traffic, and liveness only for unrecoverable deadlocks. Health endpoints should be cheap, deterministic, and sanitized. Tune periodSeconds, timeoutSeconds, and failure thresholds from measured startup and dependency behavior.

Question 41Medium

Kubernetes reliability/cost scenario: Aggressive probes create restart loops and increase load during partial outages. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Use startupProbe to protect slow boot, readiness to control traffic, and liveness only for unrecoverable deadlocks. Health endpoints should be cheap, deterministic, and sanitized. Tune periodSeconds, timeoutSeconds, and failure thresholds from measured startup and dependency behavior.

Question 42Hard

Kubernetes incident: Ingress returns 404 or 502 after a rule change, while the backend Service works inside the cluster. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check Ingress status, controller logs, generated load balancer config, backend Service ports, TLS secret, DNS records, and external health checks. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Separate DNS, certificate, routing, and backend health checks. Validate host/path matching and backend protocol, use staged DNS weights or parallel hostnames for migration, and restrict risky annotations with admission policy. Monitor both controller errors and external synthetic checks.

Question 43Hard

Kubernetes architecture scenario: Design external HTTP routing with TLS, path routing, and safe multi-team ownership. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Separate DNS, certificate, routing, and backend health checks. Validate host/path matching and backend protocol, use staged DNS weights or parallel hostnames for migration, and restrict risky annotations with admission policy. Monitor both controller errors and external synthetic checks.

Question 44Medium

Kubernetes security scenario: Ingress annotations allow unsafe snippets or expose internal services publicly. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Separate DNS, certificate, routing, and backend health checks. Validate host/path matching and backend protocol, use staged DNS weights or parallel hostnames for migration, and restrict risky annotations with admission policy. Monitor both controller errors and external synthetic checks.

Question 45Hard

Kubernetes release scenario: Migrate traffic from one ingress controller or gateway to another. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Separate DNS, certificate, routing, and backend health checks. Validate host/path matching and backend protocol, use staged DNS weights or parallel hostnames for migration, and restrict risky annotations with admission policy. Monitor both controller errors and external synthetic checks.

Question 46Medium

Kubernetes reliability/cost scenario: TLS renewal or load balancer changes cause avoidable downtime. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Separate DNS, certificate, routing, and backend health checks. Validate host/path matching and backend protocol, use staged DNS weights or parallel hostnames for migration, and restrict risky annotations with admission policy. Monitor both controller errors and external synthetic checks.

Question 47Hard

Kubernetes incident: A newly applied default-deny NetworkPolicy breaks database access for multiple services. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check CNI policy logs, flow logs, kubectl describe networkpolicy, DNS tests, connection tests from affected pods, and namespace labels. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Start with namespace-level default-deny in lower environments, then add explicit ingress and egress for DNS, telemetry, and known dependencies. Use CNI flow visibility such as Cilium Hubble or Calico flow logs. Roll out in audit or narrow namespaces first and avoid policies that accidentally block CoreDNS.

Question 48Hard

Kubernetes architecture scenario: Design namespace network isolation that still allows DNS, metrics, and required dependencies. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Start with namespace-level default-deny in lower environments, then add explicit ingress and egress for DNS, telemetry, and known dependencies. Use CNI flow visibility such as Cilium Hubble or Calico flow logs. Roll out in audit or narrow namespaces first and avoid policies that accidentally block CoreDNS.

Question 49Medium

Kubernetes security scenario: Pods can egress to the internet and call services outside their ownership boundary. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Start with namespace-level default-deny in lower environments, then add explicit ingress and egress for DNS, telemetry, and known dependencies. Use CNI flow visibility such as Cilium Hubble or Calico flow logs. Roll out in audit or narrow namespaces first and avoid policies that accidentally block CoreDNS.

Question 50Hard

Kubernetes release scenario: Move from allow-all networking to default-deny policies. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Start with namespace-level default-deny in lower environments, then add explicit ingress and egress for DNS, telemetry, and known dependencies. Use CNI flow visibility such as Cilium Hubble or Calico flow logs. Roll out in audit or narrow namespaces first and avoid policies that accidentally block CoreDNS.

Question 51Medium

Kubernetes reliability/cost scenario: NetworkPolicy troubleshooting is slow because denied flows are not observable. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Start with namespace-level default-deny in lower environments, then add explicit ingress and egress for DNS, telemetry, and known dependencies. Use CNI flow visibility such as Cilium Hubble or Calico flow logs. Roll out in audit or narrow namespaces first and avoid policies that accidentally block CoreDNS.

Question 52Hard

Kubernetes incident: Pods on different nodes cannot communicate after a node image or CNI upgrade. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check CNI DaemonSet health, node routes, eBPF maps or iptables rules, MTU, kubelet CNI errors, and node firewall settings. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Upgrade the CNI as critical infrastructure: one node pool or AZ at a time, with rollback manifests and node-level validation. Check MTU, route programming, policy enforcement, and kernel compatibility. Restrict write access to CNI resources and alert on agent restarts or datapath errors.

Question 53Hard

Kubernetes architecture scenario: Choose and operate a CNI for a multi-zone production cluster. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Upgrade the CNI as critical infrastructure: one node pool or AZ at a time, with rollback manifests and node-level validation. Check MTU, route programming, policy enforcement, and kernel compatibility. Restrict write access to CNI resources and alert on agent restarts or datapath errors.

Question 54Medium

Kubernetes security scenario: The CNI grants broad host privileges and its DaemonSet can be modified by many users. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Upgrade the CNI as critical infrastructure: one node pool or AZ at a time, with rollback manifests and node-level validation. Check MTU, route programming, policy enforcement, and kernel compatibility. Restrict write access to CNI resources and alert on agent restarts or datapath errors.

Question 55Hard

Kubernetes release scenario: Upgrade the CNI plugin without losing pod networking. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Upgrade the CNI as critical infrastructure: one node pool or AZ at a time, with rollback manifests and node-level validation. Check MTU, route programming, policy enforcement, and kernel compatibility. Restrict write access to CNI resources and alert on agent restarts or datapath errors.

Question 56Medium

Kubernetes reliability/cost scenario: Packet drops and conntrack pressure cause intermittent service failures. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Upgrade the CNI as critical infrastructure: one node pool or AZ at a time, with rollback manifests and node-level validation. Check MTU, route programming, policy enforcement, and kernel compatibility. Restrict write access to CNI resources and alert on agent restarts or datapath errors.

Question 57Hard

Kubernetes incident: Pods intermittently fail DNS lookups during traffic spikes. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check CoreDNS logs and metrics, resolv.conf, query rate, cache hit ratio, upstream latency, NodeLocal DNSCache status, and failed lookup samples. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Scale CoreDNS by query rate, enable caching, consider NodeLocal DNSCache, and avoid excessive search-domain expansion from short names. Test stub domain changes in a canary CoreDNS deployment or single cluster first. Use network policy and DNS policy carefully because DNS is a shared dependency.

Question 58Hard

Kubernetes architecture scenario: Design DNS resolution for high-query clusters and latency-sensitive services. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Scale CoreDNS by query rate, enable caching, consider NodeLocal DNSCache, and avoid excessive search-domain expansion from short names. Test stub domain changes in a canary CoreDNS deployment or single cluster first. Use network policy and DNS policy carefully because DNS is a shared dependency.

Question 59Medium

Kubernetes security scenario: Workloads can resolve or query internal domains they should not access. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Scale CoreDNS by query rate, enable caching, consider NodeLocal DNSCache, and avoid excessive search-domain expansion from short names. Test stub domain changes in a canary CoreDNS deployment or single cluster first. Use network policy and DNS policy carefully because DNS is a shared dependency.

Question 60Hard

Kubernetes release scenario: Change CoreDNS configuration for stub domains or conditional forwarding. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Scale CoreDNS by query rate, enable caching, consider NodeLocal DNSCache, and avoid excessive search-domain expansion from short names. Test stub domain changes in a canary CoreDNS deployment or single cluster first. Use network policy and DNS policy carefully because DNS is a shared dependency.

Question 61Medium

Kubernetes reliability/cost scenario: CoreDNS CPU rises and application latency follows DNS timeout patterns. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Scale CoreDNS by query rate, enable caching, consider NodeLocal DNSCache, and avoid excessive search-domain expansion from short names. Test stub domain changes in a canary CoreDNS deployment or single cluster first. Use network policy and DNS policy carefully because DNS is a shared dependency.

Question 62Hard

Kubernetes incident: HPA scales up too late during a traffic burst and scales down while requests are still queued. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check HPA status, metrics-server or adapter logs, Prometheus queries, queue depth, pod startup time, requests/limits, and workload latency. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Use the metric that matches demand: CPU for CPU-bound APIs, RPS or latency for services, and queue depth or lag for workers. Account for pod startup time with min replicas and scale-up policies. Use stabilization windows to prevent flapping, and protect metrics endpoints and labels from leaking sensitive data.

Question 63Hard

Kubernetes architecture scenario: Design autoscaling for APIs, workers, and queue consumers. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Use the metric that matches demand: CPU for CPU-bound APIs, RPS or latency for services, and queue depth or lag for workers. Account for pod startup time with min replicas and scale-up policies. Use stabilization windows to prevent flapping, and protect metrics endpoints and labels from leaking sensitive data.

Question 64Medium

Kubernetes security scenario: Autoscaling metrics expose tenant or customer identifiers. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Use the metric that matches demand: CPU for CPU-bound APIs, RPS or latency for services, and queue depth or lag for workers. Account for pod startup time with min replicas and scale-up policies. Use stabilization windows to prevent flapping, and protect metrics endpoints and labels from leaking sensitive data.

Question 65Hard

Kubernetes release scenario: Move from CPU-only HPA to custom or external metrics. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Use the metric that matches demand: CPU for CPU-bound APIs, RPS or latency for services, and queue depth or lag for workers. Account for pod startup time with min replicas and scale-up policies. Use stabilization windows to prevent flapping, and protect metrics endpoints and labels from leaking sensitive data.

Question 66Medium

Kubernetes reliability/cost scenario: Autoscaling increases cost but still misses latency SLOs. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Use the metric that matches demand: CPU for CPU-bound APIs, RPS or latency for services, and queue depth or lag for workers. Account for pod startup time with min replicas and scale-up policies. Use stabilization windows to prevent flapping, and protect metrics endpoints and labels from leaking sensitive data.

Question 67Hard

Kubernetes incident: Pods remain Pending even though cluster autoscaler is enabled. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check Pending pod events, autoscaler logs, node taints, affinity, topology spread, PDBs, quotas, and cloud capacity errors. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Pending pods must be schedulable onto some allowed node shape. Check requests, taints/tolerations, affinity, topology constraints, max node group size, cloud quotas, and capacity availability. Use separate pools for critical, batch, GPU, and spot workloads, with quotas and PDBs so scaling decisions do not violate availability.

Question 68Hard

Kubernetes architecture scenario: Design node pools for mixed workloads, GPUs, spot capacity, and critical services. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Pending pods must be schedulable onto some allowed node shape. Check requests, taints/tolerations, affinity, topology constraints, max node group size, cloud quotas, and capacity availability. Use separate pools for critical, batch, GPU, and spot workloads, with quotas and PDBs so scaling decisions do not violate availability.

Question 69Medium

Kubernetes security scenario: Workloads can force expensive node scale-ups by requesting large resources. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Pending pods must be schedulable onto some allowed node shape. Check requests, taints/tolerations, affinity, topology constraints, max node group size, cloud quotas, and capacity availability. Use separate pools for critical, batch, GPU, and spot workloads, with quotas and PDBs so scaling decisions do not violate availability.

Question 70Hard

Kubernetes release scenario: Introduce Karpenter or change node pool instance families. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Pending pods must be schedulable onto some allowed node shape. Check requests, taints/tolerations, affinity, topology constraints, max node group size, cloud quotas, and capacity availability. Use separate pools for critical, batch, GPU, and spot workloads, with quotas and PDBs so scaling decisions do not violate availability.

Question 71Medium

Kubernetes reliability/cost scenario: Scale-down evicts useful pods or is blocked by disruption constraints. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Pending pods must be schedulable onto some allowed node shape. Check requests, taints/tolerations, affinity, topology constraints, max node group size, cloud quotas, and capacity availability. Use separate pools for critical, batch, GPU, and spot workloads, with quotas and PDBs so scaling decisions do not violate availability.

Question 72Hard

Kubernetes incident: A node reports MemoryPressure or DiskPressure and evicts healthy pods. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check kubectl describe node, kubelet eviction events, cAdvisor metrics, container logs size, image GC state, and emptyDir usage. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Set realistic CPU, memory, and ephemeral-storage requests. Use limits for risky workloads, log rotation, image garbage collection, and namespace quotas. Evictions are based on node pressure and QoS/priority, so protect critical workloads with PriorityClass, Guaranteed or well-sized Burstable QoS, and enough node reserve.

Question 73Hard

Kubernetes architecture scenario: Design node sizing and eviction thresholds for production workloads. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Set realistic CPU, memory, and ephemeral-storage requests. Use limits for risky workloads, log rotation, image garbage collection, and namespace quotas. Evictions are based on node pressure and QoS/priority, so protect critical workloads with PriorityClass, Guaranteed or well-sized Burstable QoS, and enough node reserve.

Question 74Medium

Kubernetes security scenario: Pods can consume node disk through logs or emptyDir and affect other tenants. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Set realistic CPU, memory, and ephemeral-storage requests. Use limits for risky workloads, log rotation, image garbage collection, and namespace quotas. Evictions are based on node pressure and QoS/priority, so protect critical workloads with PriorityClass, Guaranteed or well-sized Burstable QoS, and enough node reserve.

Question 75Hard

Kubernetes release scenario: Introduce ephemeral-storage requests and limits across namespaces. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Set realistic CPU, memory, and ephemeral-storage requests. Use limits for risky workloads, log rotation, image garbage collection, and namespace quotas. Evictions are based on node pressure and QoS/priority, so protect critical workloads with PriorityClass, Guaranteed or well-sized Burstable QoS, and enough node reserve.

Question 76Medium

Kubernetes reliability/cost scenario: Overcommit improves utilization but causes noisy-neighbor evictions. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Set realistic CPU, memory, and ephemeral-storage requests. Use limits for risky workloads, log rotation, image garbage collection, and namespace quotas. Evictions are based on node pressure and QoS/priority, so protect critical workloads with PriorityClass, Guaranteed or well-sized Burstable QoS, and enough node reserve.

Question 77Hard

Kubernetes incident: A StatefulSet rollout is stuck because one ordinal never becomes Ready. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check StatefulSet events, PVC/PV binding, pod ordinal logs, readiness gates, quorum state, topology spread, and storage backend health. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Respect ordered identity and quorum. Back up first, upgrade one ordinal at a time, validate replication health, and understand whether the application supports parallel rollout. Use pod anti-affinity or topology spread, zone-aware storage, scoped secrets, and tested restore procedures.

Question 78Hard

Kubernetes architecture scenario: Design stateful workloads with stable identity, storage, backups, and failover. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Respect ordered identity and quorum. Back up first, upgrade one ordinal at a time, validate replication health, and understand whether the application supports parallel rollout. Use pod anti-affinity or topology spread, zone-aware storage, scoped secrets, and tested restore procedures.

Question 79Medium

Kubernetes security scenario: Database pods run with broad filesystem permissions and shared credentials. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Respect ordered identity and quorum. Back up first, upgrade one ordinal at a time, validate replication health, and understand whether the application supports parallel rollout. Use pod anti-affinity or topology spread, zone-aware storage, scoped secrets, and tested restore procedures.

Question 80Hard

Kubernetes release scenario: Upgrade a database StatefulSet image or configuration safely. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Respect ordered identity and quorum. Back up first, upgrade one ordinal at a time, validate replication health, and understand whether the application supports parallel rollout. Use pod anti-affinity or topology spread, zone-aware storage, scoped secrets, and tested restore procedures.

Question 81Medium

Kubernetes reliability/cost scenario: Stateful pods concentrate in one zone and fail during zonal impairment. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Respect ordered identity and quorum. Back up first, upgrade one ordinal at a time, validate replication health, and understand whether the application supports parallel rollout. Use pod anti-affinity or topology spread, zone-aware storage, scoped secrets, and tested restore procedures.

Question 82Hard

Kubernetes incident: Pods fail to start because PVCs are Pending or volumes cannot attach. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check PVC/PV status, StorageClass binding mode, CSI controller logs, VolumeAttachment objects, node zone labels, and cloud volume events. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Use WaitForFirstConsumer for zonal disks so scheduling and volume creation agree on topology. Check access modes, attach limits, reclaim policy, and CSI health. For migration, snapshot or replicate data, create new PVCs intentionally, validate application consistency, and avoid deleting retained PVs until restore is proven.

Question 83Hard

Kubernetes architecture scenario: Design Kubernetes storage for RWO, RWX, snapshots, and disaster recovery. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Use WaitForFirstConsumer for zonal disks so scheduling and volume creation agree on topology. Check access modes, attach limits, reclaim policy, and CSI health. For migration, snapshot or replicate data, create new PVCs intentionally, validate application consistency, and avoid deleting retained PVs until restore is proven.

Question 84Medium

Kubernetes security scenario: Applications mount volumes with permissions broader than required. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Use WaitForFirstConsumer for zonal disks so scheduling and volume creation agree on topology. Check access modes, attach limits, reclaim policy, and CSI health. For migration, snapshot or replicate data, create new PVCs intentionally, validate application consistency, and avoid deleting retained PVs until restore is proven.

Question 85Hard

Kubernetes release scenario: Migrate workloads to a new StorageClass or CSI driver. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Use WaitForFirstConsumer for zonal disks so scheduling and volume creation agree on topology. Check access modes, attach limits, reclaim policy, and CSI health. For migration, snapshot or replicate data, create new PVCs intentionally, validate application consistency, and avoid deleting retained PVs until restore is proven.

Question 86Medium

Kubernetes reliability/cost scenario: Volume attach limits or zone mismatch blocks scheduling during failover. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Use WaitForFirstConsumer for zonal disks so scheduling and volume creation agree on topology. Check access modes, attach limits, reclaim policy, and CSI health. For migration, snapshot or replicate data, create new PVCs intentionally, validate application consistency, and avoid deleting retained PVs until restore is proven.

Question 87Hard

Kubernetes incident: A service account gets Forbidden in production but works with cluster-admin in dev. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check kubectl auth can-i, audit logs, Role/ClusterRole rules, RoleBinding subjects, service account tokens, and controller error logs. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Start from observed API verbs/resources and grant only what is needed. Prefer namespaced Roles when possible, reserve ClusterRoles for cluster-scoped resources, and test with kubectl auth can-i --as=system:serviceaccount:ns:name. Roll out RBAC separately from workload changes so authorization failures are easy to isolate.

Question 88Hard

Kubernetes architecture scenario: Design least-privilege RBAC for controllers, apps, and platform teams. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Start from observed API verbs/resources and grant only what is needed. Prefer namespaced Roles when possible, reserve ClusterRoles for cluster-scoped resources, and test with kubectl auth can-i --as=system:serviceaccount:ns:name. Roll out RBAC separately from workload changes so authorization failures are easy to isolate.

Question 89Medium

Kubernetes security scenario: A namespace RoleBinding grants access to secrets or privileged resources unintentionally. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Start from observed API verbs/resources and grant only what is needed. Prefer namespaced Roles when possible, reserve ClusterRoles for cluster-scoped resources, and test with kubectl auth can-i --as=system:serviceaccount:ns:name. Roll out RBAC separately from workload changes so authorization failures are easy to isolate.

Question 90Hard

Kubernetes release scenario: Replace cluster-admin permissions with scoped Roles. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Start from observed API verbs/resources and grant only what is needed. Prefer namespaced Roles when possible, reserve ClusterRoles for cluster-scoped resources, and test with kubectl auth can-i --as=system:serviceaccount:ns:name. Roll out RBAC separately from workload changes so authorization failures are easy to isolate.

Question 91Medium

Kubernetes reliability/cost scenario: RBAC changes break controllers during a production deploy. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Start from observed API verbs/resources and grant only what is needed. Prefer namespaced Roles when possible, reserve ClusterRoles for cluster-scoped resources, and test with kubectl auth can-i --as=system:serviceaccount:ns:name. Roll out RBAC separately from workload changes so authorization failures are easy to isolate.

Question 92Hard

Kubernetes incident: A validating webhook outage blocks all pod creates in the cluster. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check admission webhook configuration, API server audit logs, webhook service endpoints, timeout settings, failurePolicy, and policy violation reports. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Admission webhooks must be highly available, fast, and scoped. Use failurePolicy: Fail only for controls that must block and have reliable webhook capacity; use namespace selectors and short timeouts. Prefer built-in Pod Security Admission for baseline restrictions and use policy engines for custom rules with staged enforcement.

Question 93Hard

Kubernetes architecture scenario: Design admission controls for image policy, security contexts, and required labels. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Admission webhooks must be highly available, fast, and scoped. Use failurePolicy: Fail only for controls that must block and have reliable webhook capacity; use namespace selectors and short timeouts. Prefer built-in Pod Security Admission for baseline restrictions and use policy engines for custom rules with staged enforcement.

Question 94Medium

Kubernetes security scenario: Teams can deploy privileged containers, hostPath mounts, or images from untrusted registries. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Admission webhooks must be highly available, fast, and scoped. Use failurePolicy: Fail only for controls that must block and have reliable webhook capacity; use namespace selectors and short timeouts. Prefer built-in Pod Security Admission for baseline restrictions and use policy engines for custom rules with staged enforcement.

Question 95Hard

Kubernetes release scenario: Move from PodSecurityPolicy-era controls to Pod Security Admission or policy engines. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Admission webhooks must be highly available, fast, and scoped. Use failurePolicy: Fail only for controls that must block and have reliable webhook capacity; use namespace selectors and short timeouts. Prefer built-in Pod Security Admission for baseline restrictions and use policy engines for custom rules with staged enforcement.

Question 96Medium

Kubernetes reliability/cost scenario: Admission latency slows deployments and causes API server request timeouts. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Admission webhooks must be highly available, fast, and scoped. Use failurePolicy: Fail only for controls that must block and have reliable webhook capacity; use namespace selectors and short timeouts. Prefer built-in Pod Security Admission for baseline restrictions and use policy engines for custom rules with staged enforcement.

Question 97Hard

Kubernetes incident: A rotated Secret does not update running pods and the application keeps using old credentials. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check Secret version, pod env/volume mounts, app reload behavior, External Secrets status, RBAC access, and audit logs. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Environment variable secrets do not update until pod restart; mounted Secret volumes update eventually but applications must reload them. Use external secret managers, scoped service accounts, encryption at rest, and rollout automation for apps that cannot reload. Rotate with overlapping credentials where the backend supports it.

Question 98Hard

Kubernetes architecture scenario: Design secret delivery and rotation for Kubernetes workloads. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Environment variable secrets do not update until pod restart; mounted Secret volumes update eventually but applications must reload them. Use external secret managers, scoped service accounts, encryption at rest, and rollout automation for apps that cannot reload. Rotate with overlapping credentials where the backend supports it.

Question 99Medium

Kubernetes security scenario: Secrets are exposed through environment variables, logs, or broad RBAC. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Environment variable secrets do not update until pod restart; mounted Secret volumes update eventually but applications must reload them. Use external secret managers, scoped service accounts, encryption at rest, and rollout automation for apps that cannot reload. Rotate with overlapping credentials where the backend supports it.

Question 100Hard

Kubernetes release scenario: Move from static Kubernetes Secrets to External Secrets or CSI secret delivery. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Environment variable secrets do not update until pod restart; mounted Secret volumes update eventually but applications must reload them. Use external secret managers, scoped service accounts, encryption at rest, and rollout automation for apps that cannot reload. Rotate with overlapping credentials where the backend supports it.

Question 101Medium

Kubernetes reliability/cost scenario: Secret rotation causes connection storms and failed logins. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Environment variable secrets do not update until pod restart; mounted Secret volumes update eventually but applications must reload them. Use external secret managers, scoped service accounts, encryption at rest, and rollout automation for apps that cannot reload. Rotate with overlapping credentials where the backend supports it.

Question 102Hard

Kubernetes incident: Controllers lag and kubectl calls intermittently time out during high churn. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check API server latency metrics, request rates by user-agent, audit logs, APF queues, watch counts, controller logs, and etcd latency. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Use informers and watches instead of polling, scope watches by namespace or label where possible, set client-side rate limits, and identify noisy user agents. Secure access with short-lived credentials and least privilege. For APF, classify critical controllers so noisy batch clients cannot starve control-plane operations.

Question 103Hard

Kubernetes architecture scenario: Design controller and operator interactions that do not overload the API server. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Use informers and watches instead of polling, scope watches by namespace or label where possible, set client-side rate limits, and identify noisy user agents. Secure access with short-lived credentials and least privilege. For APF, classify critical controllers so noisy batch clients cannot starve control-plane operations.

Question 104Medium

Kubernetes security scenario: Clients use long-lived admin kubeconfigs outside controlled systems. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Use informers and watches instead of polling, scope watches by namespace or label where possible, set client-side rate limits, and identify noisy user agents. Secure access with short-lived credentials and least privilege. For APF, classify critical controllers so noisy batch clients cannot starve control-plane operations.

Question 105Hard

Kubernetes release scenario: Introduce a new operator that watches many resources cluster-wide. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Use informers and watches instead of polling, scope watches by namespace or label where possible, set client-side rate limits, and identify noisy user agents. Secure access with short-lived credentials and least privilege. For APF, classify critical controllers so noisy batch clients cannot starve control-plane operations.

Question 106Medium

Kubernetes reliability/cost scenario: API Priority and Fairness throttles low-priority clients during incidents. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Use informers and watches instead of polling, scope watches by namespace or label where possible, set client-side rate limits, and identify noisy user agents. Secure access with short-lived credentials and least privilege. For APF, classify critical controllers so noisy batch clients cannot starve control-plane operations.

Question 107Hard

Kubernetes incident: A CRD upgrade breaks an operator and existing custom resources fail validation. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check CRD versions, storedVersions, conversion webhook health, controller logs, reconcile rate, finalizers, and failed custom resources. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Treat CRDs as APIs. Add new versions before removing old ones, maintain conversion webhooks, validate schemas carefully, and migrate stored versions. Controllers should be idempotent, rate-limited, and scoped to the minimum resources they own. Never remove fields or finalizers without a migration plan.

Question 108Hard

Kubernetes architecture scenario: Design CRDs and controllers with versioning, conversion, and backward compatibility. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Treat CRDs as APIs. Add new versions before removing old ones, maintain conversion webhooks, validate schemas carefully, and migrate stored versions. Controllers should be idempotent, rate-limited, and scoped to the minimum resources they own. Never remove fields or finalizers without a migration plan.

Question 109Medium

Kubernetes security scenario: A custom controller has broad permissions and can modify resources across all namespaces. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Treat CRDs as APIs. Add new versions before removing old ones, maintain conversion webhooks, validate schemas carefully, and migrate stored versions. Controllers should be idempotent, rate-limited, and scoped to the minimum resources they own. Never remove fields or finalizers without a migration plan.

Question 110Hard

Kubernetes release scenario: Upgrade a CRD schema and controller without corrupting existing resources. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Treat CRDs as APIs. Add new versions before removing old ones, maintain conversion webhooks, validate schemas carefully, and migrate stored versions. Controllers should be idempotent, rate-limited, and scoped to the minimum resources they own. Never remove fields or finalizers without a migration plan.

Question 111Medium

Kubernetes reliability/cost scenario: A controller reconcile loop creates an event storm and overloads the API server. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Treat CRDs as APIs. Add new versions before removing old ones, maintain conversion webhooks, validate schemas carefully, and migrate stored versions. Controllers should be idempotent, rate-limited, and scoped to the minimum resources they own. Never remove fields or finalizers without a migration plan.

Question 112Hard

Kubernetes incident: A Kubernetes minor version upgrade causes workloads using deprecated APIs to fail. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check deprecated API metrics, kubectl api-resources, admission warnings, add-on compatibility, node drain events, PDBs, and workload SLOs. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Run pre-upgrade scans for deprecated APIs and add-on compatibility, upgrade control plane first, then node pools gradually. Drain one failure domain at a time, respect PDBs, and keep rollback or surge capacity. Make upgrades routine so security patches do not become emergency migrations.

Question 113Hard

Kubernetes architecture scenario: Design a cluster upgrade process for many teams and namespaces. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Run pre-upgrade scans for deprecated APIs and add-on compatibility, upgrade control plane first, then node pools gradually. Drain one failure domain at a time, respect PDBs, and keep rollback or surge capacity. Make upgrades routine so security patches do not become emergency migrations.

Question 114Medium

Kubernetes security scenario: Old cluster versions carry known vulnerabilities but upgrades are repeatedly delayed. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Run pre-upgrade scans for deprecated APIs and add-on compatibility, upgrade control plane first, then node pools gradually. Drain one failure domain at a time, respect PDBs, and keep rollback or surge capacity. Make upgrades routine so security patches do not become emergency migrations.

Question 115Hard

Kubernetes release scenario: Upgrade control plane, node pools, and add-ons with minimal downtime. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Run pre-upgrade scans for deprecated APIs and add-on compatibility, upgrade control plane first, then node pools gradually. Drain one failure domain at a time, respect PDBs, and keep rollback or surge capacity. Make upgrades routine so security patches do not become emergency migrations.

Question 116Medium

Kubernetes reliability/cost scenario: Node upgrades evict too many pods or break workloads tied to node kernel behavior. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Run pre-upgrade scans for deprecated APIs and add-on compatibility, upgrade control plane first, then node pools gradually. Drain one failure domain at a time, respect PDBs, and keep rollback or surge capacity. Make upgrades routine so security patches do not become emergency migrations.

Question 117Hard

Kubernetes incident: Critical pods are Pending because affinity, taints, and topology constraints conflict. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check pod scheduling events, scheduler logs, node labels, taints/tolerations, affinity rules, topology spread constraints, and resource requests. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Use hard constraints only for requirements and soft preferences for optimization. Protect platform nodes with taints and admission policy, and use topology spread with realistic maxSkew and fallback behavior. Validate changes with representative workloads before applying them cluster-wide.

Question 118Hard

Kubernetes architecture scenario: Design scheduling rules for critical, batch, GPU, and zone-aware workloads. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Use hard constraints only for requirements and soft preferences for optimization. Protect platform nodes with taints and admission policy, and use topology spread with realistic maxSkew and fallback behavior. Validate changes with representative workloads before applying them cluster-wide.

Question 119Medium

Kubernetes security scenario: Untrusted workloads can schedule onto nodes meant for privileged platform components. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Use hard constraints only for requirements and soft preferences for optimization. Protect platform nodes with taints and admission policy, and use topology spread with realistic maxSkew and fallback behavior. Validate changes with representative workloads before applying them cluster-wide.

Question 120Hard

Kubernetes release scenario: Introduce taints, tolerations, and topology spread constraints to an existing cluster. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Use hard constraints only for requirements and soft preferences for optimization. Protect platform nodes with taints and admission policy, and use topology spread with realistic maxSkew and fallback behavior. Validate changes with representative workloads before applying them cluster-wide.

Question 121Medium

Kubernetes reliability/cost scenario: Strict spreading improves HA but prevents scheduling during partial capacity loss. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Use hard constraints only for requirements and soft preferences for optimization. Protect platform nodes with taints and admission policy, and use topology spread with realistic maxSkew and fallback behavior. Validate changes with representative workloads before applying them cluster-wide.

Question 122Hard

Kubernetes incident: Users report errors but cluster dashboards show all pods Running. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check service SLOs, ingress metrics, pod restarts, events, logs, traces, RED/USE metrics, and metric cardinality reports. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Running is not a health signal. Monitor request rate, errors, duration, saturation, dependency failures, and endpoint readiness. Add SLO burn-rate alerts tied to user impact, scrub sensitive fields from logs/traces, and control metric labels so cardinality does not break monitoring.

Question 123Hard

Kubernetes architecture scenario: Design observability for Kubernetes services from node to user journey. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Running is not a health signal. Monitor request rate, errors, duration, saturation, dependency failures, and endpoint readiness. Add SLO burn-rate alerts tied to user impact, scrub sensitive fields from logs/traces, and control metric labels so cardinality does not break monitoring.

Question 124Medium

Kubernetes security scenario: Logs and traces contain secrets, tokens, or customer data. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Running is not a health signal. Monitor request rate, errors, duration, saturation, dependency failures, and endpoint readiness. Add SLO burn-rate alerts tied to user impact, scrub sensitive fields from logs/traces, and control metric labels so cardinality does not break monitoring.

Question 125Hard

Kubernetes release scenario: Introduce SLO burn-rate alerts and service dashboards. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Running is not a health signal. Monitor request rate, errors, duration, saturation, dependency failures, and endpoint readiness. Add SLO burn-rate alerts tied to user impact, scrub sensitive fields from logs/traces, and control metric labels so cardinality does not break monitoring.

Question 126Medium

Kubernetes reliability/cost scenario: High-cardinality metrics make Prometheus expensive and slow. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Running is not a health signal. Monitor request rate, errors, duration, saturation, dependency failures, and endpoint readiness. Add SLO burn-rate alerts tied to user impact, scrub sensitive fields from logs/traces, and control metric labels so cardinality does not break monitoring.

Question 127Hard

Kubernetes incident: A rollout fails because nodes cannot pull a private image. How do you investigate, recover service, and prevent the same failure?

#
Show answer

Start by proving scope and the last change before editing resources. Check pod image pull events, registry auth, imagePullSecrets, node credential provider logs, admission policy, scan results, and image size history. Recover with the smallest reversible action, such as rollback, scaling, removing a bad endpoint, or temporarily relaxing a broken policy. The durable fix is: Use immutable image digests, private registries, short-lived pull credentials, vulnerability scanning, and signature verification with a staged exception process. Keep runtime images small and pre-pull critical images where needed. Do not rely on mutable tags for production rollbacks.

Question 128Hard

Kubernetes architecture scenario: Design secure image build, signing, scanning, and promotion for Kubernetes. What design would you choose and what tradeoffs would you explain?

#
Show answer

Design around failure domains, clear ownership, safe rollout, resource isolation, and observability. Prefer Kubernetes primitives first, then add controllers or platform tooling only when they solve an operational gap. For this scenario: Use immutable image digests, private registries, short-lived pull credentials, vulnerability scanning, and signature verification with a staged exception process. Keep runtime images small and pre-pull critical images where needed. Do not rely on mutable tags for production rollbacks.

Question 129Medium

Kubernetes security scenario: Clusters can run unsigned images or images from public registries. How do you harden it without breaking production?

#
Show answer

Inventory current consumers, add audit or report-only checks first where possible, then roll out namespace by namespace. Keep a rollback path and alert on denied traffic, rejected admission requests, or auth failures. For this scenario: Use immutable image digests, private registries, short-lived pull credentials, vulnerability scanning, and signature verification with a staged exception process. Keep runtime images small and pre-pull critical images where needed. Do not rely on mutable tags for production rollbacks.

Question 130Hard

Kubernetes release scenario: Enforce image registry and signature policy without blocking emergency fixes. How do you ship the change safely?

#
Show answer

Use a staged rollout with preflight checks, small blast radius, health gates, and an explicit rollback. Watch workload readiness, endpoint count, error rate, latency, saturation, and events while the change rolls forward. For this scenario: Use immutable image digests, private registries, short-lived pull credentials, vulnerability scanning, and signature verification with a staged exception process. Keep runtime images small and pre-pull critical images where needed. Do not rely on mutable tags for production rollbacks.

Question 131Medium

Kubernetes reliability/cost scenario: Large images slow node scale-up and increase cold-start time. What signals do you inspect and what changes do you make?

#
Show answer

Inspect requests, limits, throttling, memory pressure, pending pods, node utilization, disruption events, restart rate, and traffic shape. Tune the bottleneck instead of blindly adding nodes. For this scenario: Use immutable image digests, private registries, short-lived pull credentials, vulnerability scanning, and signature verification with a staged exception process. Keep runtime images small and pre-pull critical images where needed. Do not rely on mutable tags for production rollbacks.

Question 132HardService routing & ingress incidents

All Kubernetes pods report Ready, yet customers intermittently receive HTTP 503 from the public ingress. How do you find the exact failing hop and recover without restarting the entire cluster?

#

A new checkout release has 8/8 Ready replicas and no pod restarts. Calls inside the namespace sometimes succeed, but public requests through the ingress controller return 503, especially during rollout and scale events.

Show answer

Triage: Identify whether the HTTP 503 originates at the external load balancer, ingress gateway, service mesh, or application using request IDs, response headers, access logs and upstream error counters. Check Ingress/Gateway backend mappings, Service selector and port/targetPort, EndpointSlice membership, endpoints during rollout, target registration and health, readiness probe semantics, connection draining and idle/keepalive mismatches. Inspect version/AZ distribution and compare traffic paths: direct pod request, Service DNS/ClusterIP from a debug pod, ingress inside the cluster, then the external hostname. Confirm any NetworkPolicy, mesh policy, TLS or timeout mismatch on the failing hop.

Containment: If only the newly deployed version fails, stop promotion and shift traffic or roll back using the release mechanism, watching both old and new endpoints. If endpoints vanish during rollout, validate maxUnavailable, maxSurge, startup/readiness probes, termination grace and preStop/draining behavior before applying a targeted rollout correction. Fix a selector or targetPort mismatch only after comparing the live desired configuration with the service contract; do not delete healthy pods at random.

Prevention: Build release verification from outside the cluster, keep readiness checks representative of the application's ability to serve meaningful requests, alert on available EndpointSlice targets and ingress upstream 503s, and test scaling, termination and version skew. Kubernetes Ready status proves only configured readiness checks passed; it cannot certify the public request path.

Trade-offs

More aggressive readiness/draining protects users but can slow rollouts; broad restarts may temporarily improve symptoms while erasing evidence and increasing instability.

Common mistake

Assuming that 8/8 Ready means the ingress has registered and can actually route to all eight backends.

Follow-up

Pods are Ready and Service requests succeed, but ingress 503s spike only for 15 seconds during a rollout. Which endpoint and termination timing would you inspect?

Read-only Kubernetes routing checks · shell
kubectl -n checkout get ingress,svc,deploy -o wide
kubectl -n checkout get endpointslices -l kubernetes.io/service-name=checkout -o wide
kubectl -n checkout describe svc checkout
kubectl -n checkout rollout status deploy/checkout
Isolate the first failing hop
  1. Identify 503 generator with trace/request ID
  2. External LB → ingress upstreams
  3. Ingress → Service/EndpointSlice
  4. EndpointSlice → ready Pod/port
  5. Rollback or repair boundary; verify external requests

Check your reasoning

  • Explain how a pod can be Ready while an ingress has no usable upstream.
  • Show one way to compare the direct Service route to the public route.
  • Explain how termination and endpoint propagation affect short 503 bursts.
Question 133HardCoreDNS, NodeLocal DNS & resolution

Only some Kubernetes pods intermittently fail DNS lookups under load. What evidence would help you separate a CoreDNS outage from a node-level networking, resolver or upstream DNS problem?

#

An EKS cluster sees occasional 2-second DNS timeouts during peak QPS. Failures cluster on certain nodes; other pods resolve the same service successfully. Restarting CoreDNS temporarily appears to help but the issue returns.

Show answer

Triage: Gather a representative failing name and timestamp, failure type (timeout, SERVFAIL, NXDOMAIN), source pod, namespace, node and resolver path. From affected and unaffected nodes, inspect /etc/resolv.conf (nameserver, search, ndots), try absolute service FQDNs and external names, and compare UDP versus TCP queries where supported. Check kube-dns Service and EndpointSlices, CoreDNS health, restart counts, DNS request/error/latency metrics, configured forwarding/upstreams and autoscaling. Trace network policy, CNI/service routing, conntrack pressure, node-local DNS cache status, packet loss and upstream resolver health. High ndots/search retries can amplify lookup traffic.

Containment: Restore the specific broken resolver path—such as an unhealthy DNS endpoint, overloaded CoreDNS replicas, a bad forwarding configuration or node-level networking—rather than broadly deleting all DNS pods. Consider scaling DNS replicas and using supported NodeLocal DNSCache only after demonstrating a cache/conntrack bottleneck. Roll back a recent DNS or node upgrade if the symptoms correlate and the rollback is safer than ad-hoc modifications.

Prevention: Alert on DNS timeouts, response codes, latency, cache hit rate and per-node failure concentration. Load-test peak query patterns, review DNS search configuration and test node DNS behavior during rolling upgrades. Keep failure evidence and add a runbook with exact reproducible comparisons. A green CoreDNS Deployment does not prove every node can reach or use the DNS service reliably.

Trade-offs

Caching reduces repeat queries and conntrack traffic but can serve stale responses; additional replicas help DNS compute pressure but do not repair broken node-local networking.

Common mistake

Restarting CoreDNS every time without determining whether failures occur on particular nodes or involve external resolvers.

Follow-up

A fully qualified service name resolves over TCP but frequently times out over UDP only on two nodes. What would you check next?

Read-only DNS and endpoint checks · shell
kubectl -n kube-system get svc kube-dns
kubectl -n kube-system get endpointslices -l kubernetes.io/service-name=kube-dns
kubectl -n kube-system get pods -o wide
kubectl -n kube-system logs deploy/coredns --since=15m
Trace DNS from pod to authoritative source
  1. Compare failing and healthy pod/node observations
  2. Inspect resolver config and ndots/search path
  3. Check kube-dns Service/EndpointSlice/CoreDNS
  4. Check node network, conntrack and NodeLocal DNS
  5. Check external upstream and validate mitigation

Check your reasoning

  • Explain why TCP-success/UDP-failure points toward a different set of causes.
  • State how ndots and search domains can multiply DNS queries.
  • Describe why restarting all DNS pods may hide the real cause.
Question 134MediumNetworking

Ingress NGINX is retired. How would you plan the move to Gateway API?

#

Forty services sit behind the community ingress-nginx controller. Security has flagged that it no longer receives patches.

Show answer

Kubernetes SIG Network retired the community Ingress NGINX controller in March 2026, so it no longer gets fixes or security patches. Existing installs keep running, which is exactly why teams forget them.

Start with an inventory: every Ingress, and especially the nginx-specific annotations (rewrites, auth, rate limits, snippets) because those carry behaviour the Ingress API never modelled. Pick a Gateway API implementation, let the platform team own the Gateway and listeners, and let app teams own their HTTPRoutes. Use ingress2gateway for a first draft, then review it by hand. Run both paths in parallel, compare responses, and move DNS or the load balancer service by service with a rollback path.

Trade-offs

A like-for-like swap to another Ingress controller is faster; Gateway API is more work now but gives role separation and portable routing features.

Common mistake

Converting manifests mechanically and discovering in production that a snippet annotation was doing auth or header rewriting.

Follow-up

Which nginx annotations have no direct Gateway API equivalent in your chosen implementation, and how will you cover them?

Question 135MediumResources

In-place Pod resize is stable. When would you still recreate the Pod instead?

#

A StatefulSet replica needs more CPU during a reindex. The team wants to avoid a restart.

Show answer

Since Kubernetes 1.35 you can change CPU and memory requests and limits on a running Pod through the resize subresource. Each container's resizePolicy says whether a change needs a container restart. The kubelet applies the change if the node has room; otherwise the resize stays pending or is reported as infeasible.

You still recreate when the Pod must move to a bigger node, when the change would alter its QoS class, or when the process reads its size only at startup, such as a JVM with a fixed heap. Memory decreases also need care: lowering a limit below current usage is not a safe instant shrink.

Trade-offs

Resizing in place avoids restarts and lost caches, but a node can still run out of headroom, so it does not replace the scheduler.

Common mistake

Assuming the new limit took effect because the patch succeeded, without checking the Pod status and conditions.

Follow-up

How would you combine in-place resize with the Vertical Pod Autoscaler without fighting the HPA?

Resize a running container (kubectl 1.32+) · shell
kubectl patch pod web-0 --subresource resize --patch \
  '{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"800m"},"limits":{"cpu":"800m"}}}]}}'
Question 136HardWorkloads

A log-shipping sidecar keeps a Job from ever completing. Why, and how do native sidecars fix it?

#

The batch container exits 0 after ten minutes, but the Job stays Running because the fluent-bit container never stops.

Show answer

A sidecar declared as a normal container is just another app container, so the Pod keeps running until it exits too. Teams used to hack around this with shared files or kill signals.

Native sidecars, stable since Kubernetes 1.33, are init containers with restartPolicy: Always. They start before the main containers, keep running alongside them, and are stopped after the main containers finish, so the Job can complete. Because they start first, they are also ready to ship logs or proxy traffic from the first line the app writes.

Trade-offs

Native sidecars fix lifecycle ordering, but they still consume the Pod's resources and can delay startup if their probes are slow.

Common mistake

Adding restartPolicy: Always to a regular container entry; the field only creates a sidecar inside initContainers.

Follow-up

How does termination ordering change if the sidecar must flush logs after the main container exits?

Native sidecar inside a Job Pod template · yaml
spec:
  restartPolicy: Never
  initContainers:
    - name: log-shipper
      image: fluent/fluent-bit:3.2
      restartPolicy: Always
  containers:
    - name: batch
      image: registry.example.com/reindex:1.4.0
Question 137MediumPolicy

When would you use ValidatingAdmissionPolicy instead of a webhook or a policy engine like Kyverno or Gatekeeper?

#

You need three rules: images only from your registry, required team labels, and a check against an external inventory API.

Show answer

ValidatingAdmissionPolicy (stable since 1.30) evaluates CEL expressions inside the API server. There is no extra service to run or to fail, so it suits rules that look only at the object and its parameters, such as allowed registries and required labels. Bindings scope it, and validationActions let you start with Warn and Audit before Deny.

CEL cannot call external systems, so the inventory check needs a webhook. Policy engines add things VAP does not: mutation and resource generation, background scans and reports of existing resources, and policy libraries. Many teams use VAP for simple guardrails and an engine for the rest.

Trade-offs

In-process policies are fast and resilient but limited to what CEL can see; engines are richer but add components to operate.

Common mistake

Running a high-availability webhook just to check labels that CEL could validate in the API server.

Follow-up

How would you report which existing resources violate a new policy before enforcing it?

Question 138HardNetworking

kube-proxy IPVS mode is deprecated. How would you move a large cluster to nftables mode safely?

#

A 600-node cluster has run kube-proxy in ipvs mode for years because iptables rule syncs were too slow. The platform roadmap now says IPVS is going away.

Show answer

The ipvs mode has been deprecated since Kubernetes v1.35. The kube-proxy docs say it will be disabled by default from v1.40 (re-enabled only through the KubeProxyIPVS feature gate) and removed entirely in v1.43. The project recommends nftables as the replacement for both iptables and ipvs, because it scales better than either.

nftables is not a drop-in swap. It needs Linux kernel 5.13 or later. It defaults to --nodeport-addresses primary, so NodePort Services answer only on the node's primary IPs, and NodePorts on 127.0.0.1 do not work unless you enable a v1.37 alpha gate. Any node-local tool or health check that hits localhost:<nodePort> will break. If you run a local nftables-based firewall, you must allow Kubernetes traffic yourself.

My plan: first pin the mode explicitly in every kube-proxy config, because the default (iptables in v1.37) is going to change and you do not want an upgrade to switch backends silently. Audit kernels and localhost NodePort users. Then roll out by node pool: build a new pool whose kube-proxy runs nftables, move workloads with normal drains, and retire the old pool. Replacing nodes is cleaner than flipping modes in place, because the docs warn that rules left by the old mode may not be cleaned up until a reboot. Watch connection errors and kube-proxy sync latency per pool before continuing.

Trade-offs

nftables keeps you on the upstream path with better performance; an eBPF CNI that replaces kube-proxy is the other option, but it moves Service routing into a component with a different upgrade and debugging model.

Common mistake

Leaving the mode unset and letting a future default change pick the backend during an unrelated cluster upgrade.

Follow-up

How would you prove that no client reaches NodePorts through localhost before switching a pool?

Pin the proxy mode explicitly in the kube-proxy config · yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "nftables"
Question 139MediumNetworking

Cross-zone data transfer is a top line on the cloud bill. How would you use Service traffic distribution, and what can go wrong?

#

A chatty internal API runs evenly across three zones. Its callers pick any endpoint in the cluster, so most calls leave their zone.

Show answer

By default a Service spreads traffic evenly across all endpoints in the cluster. With endpoints in three zones, roughly two out of three calls leave the caller's zone, and you pay for each of those bytes.

Setting spec.trafficDistribution: PreferSameZone tells the service proxy to prefer endpoints in the client's zone. If the client's zone has no available endpoints, traffic falls back to the whole cluster. PreferSameNode goes one step further and prefers endpoints on the same node. The older value PreferClose is a deprecated alias for PreferSameZone.

The catch is load balance. The fallback triggers only when a zone has no available endpoints, not when its endpoints are overloaded. If zone A produces 60% of the calls but holds a third of the pods, those pods run hot while the others idle. So pair it with topology spread constraints that keep pods even across zones, make sure callers are also spread evenly, and watch per-pod request rate and latency by zone after the change. Also note precedence: an internalTrafficPolicy or externalTrafficPolicy of Local overrides it, and so does the older service.kubernetes.io/topology-mode: Auto annotation.

Trade-offs

Zone-local routing cuts cost and latency but gives up even load spreading; uneven zones turn into hot spots instead of a bigger bill.

Common mistake

Enabling it on a Service whose pods are concentrated in one zone, so the other zones fall back to cross-zone traffic and the hot zone overloads.

Follow-up

How would you detect, from metrics alone, that zone-local routing has created a hot zone?

Prefer endpoints in the caller's zone · yaml
apiVersion: v1
kind: Service
metadata:
  name: pricing
spec:
  selector:
    app: pricing
  ports:
    - port: 80
      targetPort: 8080
  trafficDistribution: PreferSameZone
Question 140MediumReliability

A node drain hangs for an hour because a PodDisruptionBudget guards a crash-looping app. How do you fix this properly?

#

A cluster upgrade stalls. kubectl drain keeps retrying evictions for a Deployment whose three pods are all in CrashLoopBackOff, protected by a PDB with minAvailable: 2.

Show answer

Drains use the Eviction API, which honours PDBs. Under the default unhealthyPodEvictionPolicy of IfHealthyBudget, a pod that is Running but not Ready can be evicted only while the application still meets its budget. A crash-looping app never meets its budget, so none of its pods can be evicted, and the drain waits forever. The Kubernetes docs call out exactly this: drains blocked by misbehaving apps guarded by a PDB.

The fix is unhealthyPodEvictionPolicy: AlwaysAllow, stable since v1.31. With it, Running but not-Ready pods count as already disrupted and can be evicted regardless of the budget. Healthy pods are still protected. For most stateless services this is the right default, and a platform team can enforce it with an admission policy.

Keep IfHealthyBudget for systems where a not-ready pod may be doing useful recovery, such as a quorum member catching up. Separately, lint for budgets that allow zero disruptions (maxUnavailable: 0, or minAvailable equal to the replica count): the docs state that a drain of such a node never completes, by design.

Trade-offs

AlwaysAllow unblocks maintenance but gives a struggling app less chance to recover on its own; IfHealthyBudget protects recovery at the cost of stuck drains.

Common mistake

Deleting the PDB or force-deleting pods to finish the drain, which removes protection from the healthy replicas too.

Follow-up

How would you report every PDB in the cluster that currently allows zero disruptions?

PDB that will not block drains on unhealthy pods · yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout
spec:
  minAvailable: 2
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app: checkout
Question 141HardWorkloads

You reverted a bad StatefulSet image, but the rollout still does not move. Why, and how do you recover?

#

A config error in a new image keeps db-2 from ever becoming Ready. You revert the pod template, yet db-2 stays broken and the controller does nothing.

Show answer

With the default OrderedReady pod management and RollingUpdate, the StatefulSet controller updates one ordinal at a time and waits for it to be Running and Ready. If the new template never becomes Ready, the rollout stops and waits.

The Kubernetes docs describe this as a known broken state (kubernetes/kubernetes#67250): reverting the template is not enough, because the controller keeps waiting for the broken pod to become Ready before it will replace it with the reverted version. You must revert the template and then delete the pods that were already created with the bad configuration. The controller then recreates them from the reverted template.

To avoid needing this in production, canary StatefulSet changes with updateStrategy.rollingUpdate.partition: only ordinals at or above the partition get the new template, so you can update the highest ordinal, verify it, and lower the partition step by step. rollingUpdate.maxUnavailable (beta and on by default in v1.37) can speed up large sets, but the docs warn that pods may then become ready out of order, which some databases do not tolerate.

Trade-offs

Ordered rollouts are safe for quorum systems but can stall on one bad pod; partitions add a manual step but make every change a canary.

Common mistake

Waiting for the controller to self-heal after the revert, extending the outage of that replica.

Follow-up

How would you automate the partition-based canary in a GitOps pipeline?

Revert the template, then delete the pod built from the bad one · shell
kubectl rollout undo statefulset/db
kubectl delete pod db-2
kubectl rollout status statefulset/db
Question 142MediumStorage

After scaling a StatefulSet from 10 to 3, the storage bill did not drop, and scaling back up brought back weeks-old data. What is happening?

#

A cache cluster was scaled down after a sale. A month later it was scaled up again, and the new replicas served stale entries.

Show answer

StatefulSets create one PersistentVolumeClaim per ordinal from volumeClaimTemplates. By default those claims are retained when the set is scaled down or deleted, so seven volumes kept billing. When you scaled back up, cache-3 to cache-9 were recreated with the same names and reattached their old claims, complete with old data.

spec.persistentVolumeClaimRetentionPolicy (stable since v1.32) controls this with two fields: whenScaled and whenDeleted, each Retain (the default) or Delete. For a cache or any replica that rebuilds its state from peers, whenScaled: Delete is right. For a database, keep whenDeleted: Retain so an accidental kubectl delete sts does not destroy data, and only delete on scale-down if new replicas resync from scratch anyway.

Remember the second layer: deleting a PVC removes the underlying disk only if the PersistentVolume's reclaim policy is Delete. Check both before promising the finance team a saving.

Trade-offs

Deleting on scale-down saves money and avoids stale state, but a quick scale-down and scale-up now pays the full cost of rebuilding each replica.

Common mistake

Setting whenDeleted: Delete on a database StatefulSet, so one mistaken delete wipes every volume.

Follow-up

How would you find and clean up claims already orphaned by past scale-downs?

Delete claims on scale-down, keep them on accidental delete · yaml
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Delete
Question 143MediumWorkloads

You added a preStop sleep to stop 502s during deploys, and now some pods are killed mid-request. Why?

#

The team set a 25-second preStop sleep on a service that takes up to 15 seconds to drain long requests after SIGTERM. terminationGracePeriodSeconds was left at its default.

Show answer

Endpoint removal and pod shutdown start at the same time. The control plane begins taking the terminating pod out of EndpointSlices while the kubelet starts shutting it down, and proxies and load balancers pick up the change a little later. A preStop sleep keeps the app serving during that gap, which is why it fixes the 502s.

But the grace period countdown starts before the preStop hook runs, and it covers both the hook and the app's own shutdown. The default grace period is 30 seconds. A 25-second sleep plus up to 15 seconds of draining needs 40, so the kubelet sends SIGKILL partway through the drain (after one 2-second extension if the hook is still running). Fix it by setting terminationGracePeriodSeconds to at least sleep plus worst-case drain plus a margin, here around 45 to 50 seconds, and keep the sleep only as long as your load balancer needs to stop sending traffic.

Use the built-in sleep lifecycle action (stable since v1.34) rather than exec: sleep. The kubelet runs it, so it works in distroless images that have no sleep binary.

Trade-offs

A longer sleep and grace period make deploys safer but slower, and they slow node drains and scale-down too.

Common mistake

Treating preStop time as extra time on top of the grace period, rather than time spent from it.

Follow-up

How would you measure how long your ingress or cloud load balancer actually takes to stop sending traffic to a terminating pod?

Sleep action with a grace period that fits sleep plus drain · yaml
spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      image: registry.example.com/api:2.3.1
      lifecycle:
        preStop:
          sleep:
            seconds: 15
Question 144HardReliability

A node running a database StatefulSet pod lost power. The pod has been Terminating for 20 minutes and its volume will not attach elsewhere. What do you do?

#

The hypervisor host died. The Node object is NotReady, db-1 is stuck Terminating, and its replacement cannot start because the volume is still attached to the dead node.

Show answer

When a node shuts down without the kubelet noticing, nothing can confirm that its pods have stopped. A StatefulSet will not create a second db-1 while the first might still be running, and the VolumeAttachment stays on the dead node. That is deliberate: two pods with the same identity writing to the same volume would corrupt data.

The supported recovery is the non-graceful node shutdown flow, stable since v1.28. First confirm the node is really powered off, not rebooting. Then add the node.kubernetes.io/out-of-service taint with NoExecute. Pods without a matching toleration are force-deleted, and their volumes are detached immediately, so the StatefulSet can start db-1 on a healthy node. Remove the taint yourself once the pods have moved. Without the taint, Kubernetes force-detaches a volume only after a pod deletion has been stuck for 6 minutes while the node is unhealthy.

For planned shutdowns, configure graceful node shutdown on the kubelet. Both shutdownGracePeriod and shutdownGracePeriodCriticalPods default to zero, which leaves the feature inactive, and it depends on systemd inhibitor locks.

Trade-offs

The taint gives fast, supported recovery, but it trusts you completely: tainting a node that is still alive can create two writers.

Common mistake

Force-deleting the pod with --grace-period=0 --force and assuming the volume is safe, without fencing the node first.

Follow-up

How would you automate fencing so an on-call engineer never has to judge whether a node is truly off?

Only after confirming the node is powered off · shell
kubectl taint nodes node-7 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute
# after pods have moved
kubectl taint nodes node-7 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute-
Question 145HardBatch

A 1,000-shard Indexed Job fails as a whole because a few shards hit bad input, and retries burn GPU hours on an error that will never succeed. How do you redesign the Job?

#

Each index processes one shard. Bad input makes the worker exit with code 42. One noisy shard exhausts the shared backoffLimit and kills the whole run.

Show answer

A single backoffLimit is shared by every pod in the Job, so a handful of bad shards can fail an otherwise healthy run. Two stable Job features fix this.

backoffLimitPerIndex (stable since v1.33) counts failures per index. A failing index no longer stops the others; it is recorded in .status.failedIndexes, and maxFailedIndexes ends the Job early only if too many shards fail. podFailurePolicy (stable since v1.31) decides what a failure means. Exit code 42 is a known permanent error, so FailIndex marks that index failed at once instead of retrying. A DisruptionTarget condition means the pod was preempted or evicted, which is not the code's fault, so Ignore retries without counting it.

Both require restartPolicy: Never, and per-index limits need completionMode: Indexed. After the run, rerun only the indexes in failedIndexes with fixed input. The result is that a bad shard costs one attempt, not a whole GPU run.

Trade-offs

Rich failure rules save compute but couple the Job spec to the worker's exit-code contract, which must be documented and tested.

Common mistake

Raising backoffLimit to a huge number so the Job finishes, which multiplies the cost of permanent errors.

Follow-up

How would you surface failedIndexes to the team that owns the bad input data?

Per-index retries with a permanent-error rule · yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: embed-shards
spec:
  completions: 1000
  parallelism: 50
  completionMode: Indexed
  backoffLimitPerIndex: 2
  maxFailedIndexes: 20
  podFailurePolicy:
    rules:
      - action: FailIndex
        onExitCodes:
          containerName: worker
          operator: In
          values: [42]
      - action: Ignore
        onPodConditions:
          - type: DisruptionTarget
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: worker
          image: registry.example.com/embedder:3.1.0
Question 146MediumBatch

During node drains, a training Job briefly runs two copies of the same worker and trips the GPU quota. Why, and what setting stops it?

#

Workers have a 120-second grace period to checkpoint. When a node drains, a replacement pod appears while the old one is still checkpointing, and the namespace GPU quota rejects other work.

Show answer

By default, the Job controller creates a replacement as soon as it sees a pod with a deletion timestamp. That is the TerminatingOrFailed policy. The terminating pod still holds its GPU for the whole grace period, so for two minutes you have two workers, double the quota use, and possibly two writers to the same checkpoint path.

Set podReplacementPolicy: Failed (stable since v1.34). The controller then waits until the old pod is fully terminal, phase Failed, before creating the new one. .status.terminating shows how many pods are still shutting down. If the Job uses podFailurePolicy, Failed is already the default and the only allowed value.

The cost is slower recovery: the replacement starts only after the grace period ends. For training with checkpoints that is almost always the right trade, because a duplicate worker is worse than a short delay.

Trade-offs

Waiting for full termination avoids duplicate workers and quota spikes but adds the grace period to every replacement.

Common mistake

Raising the GPU quota to absorb the duplicates instead of fixing the replacement timing.

Follow-up

How would this interact with a gang-scheduled training job where all workers must run together?

Wait for the old pod to finish before replacing it · yaml
apiVersion: batch/v1
kind: Job
spec:
  podReplacementPolicy: Failed
Question 147MediumSecurity

User namespaces for Pods are GA. Would you set hostUsers: false across the cluster, and what blocks it?

#

A platform hosts untrusted customer build jobs next to internal services. Security wants container root to stop being host root.

Show answer

With hostUsers: false, a pod gets its own user namespace: root inside the container maps to an unprivileged user on the host, and capabilities such as CAP_SYS_ADMIN apply only inside that namespace. The docs note several HIGH and CRITICAL vulnerabilities that were not exploitable with user namespaces active. The feature is stable since v1.36.

The blockers are mostly on the node. Filesystems must support idmap mounts, which in practice means Linux 6.3 or later because tmpfs gained support then. The runtime must support it: containerd 2.0 or later or CRI-O 1.25 or later, with runc 1.2+ or crun 1.9+. Pods using it cannot set hostNetwork, hostPID or hostIPC, cannot use raw block volumeDevices, and cannot mount NFS volumes, because the NFS client does not support idmap mounts yet.

So I would not flip it everywhere at once. Start with the untrusted build jobs, which gain the most and rarely need host namespaces. Upgrade those node pools first, run the jobs in staging, then enforce it for that namespace with an admission policy. Pod Security Standards relax runAsNonRoot and runAsUser checks for these pods, because root inside them is not host root.

Trade-offs

User namespaces sharply reduce breakout impact but rule out host namespaces, raw block devices and NFS, and need recent kernels and runtimes.

Common mistake

Enabling it on node pools with an older kernel or containerd 1.x and debugging the resulting pod start failures as an application bug.

Follow-up

Which workloads in your cluster legitimately need hostNetwork, and how would you keep them on separate nodes?

Opt a pod into its own user namespace · yaml
apiVersion: v1
kind: Pod
metadata:
  name: build-runner
spec:
  hostUsers: false
  containers:
    - name: runner
      image: registry.example.com/build-runner:5.0.2
Question 148HardSecurity

In a shared cluster, a team with no registry credentials runs another team's private image successfully. How is that possible, and how do you close it?

#

Team B deploys a pod referencing Team A's private image with imagePullPolicy: IfNotPresent and no imagePullSecrets. It starts on a node where Team A's pod already pulled that image.

Show answer

With IfNotPresent, the kubelet uses an image that is already on the node. Historically it did not re-check that the new pod had credentials for it, so any pod scheduled to that node could run any cached private image. The Kubernetes images docs say plainly that without extra controls, all pods potentially have access to all images.

There are two fixes. The newer one is kubelet credential verification, beta and enabled by default since v1.35 (feature gate KubeletEnsureSecretPulledImages). The kubelet then checks pull credentials for images that need them even when the image is cached. The kubelet setting imagePullCredentialsVerificationPolicy controls it: the default NeverVerifyPreloadedImages skips only images pulled outside the kubelet, and AlwaysVerify checks everything. The older fix is the AlwaysPullImages admission plugin, which forces a registry check on every pod start, at the cost of making pod starts depend on the registry.

In a multi-tenant cluster I would confirm the kubelet setting on every node pool (managed platforms and older versions may differ) and, for strong isolation, still give sensitive tenants their own node pools.

Trade-offs

Verification closes the gap with little cost on new versions; AlwaysPullImages works on old clusters but adds registry load and a dependency on every pod start.

Common mistake

Treating imagePullSecrets as an access control boundary on shared nodes without checking how cached images are handled.

Follow-up

How would you test this gap safely in staging on each node pool?

Question 149HardSecurity

An audit finds Secrets stored unencrypted in etcd on a self-managed cluster. How do you move to KMS v2 encryption without an outage or leaving old data readable?

#

The control plane runs on VMs you manage. etcd backups are copied to object storage, and every Secret in them is readable plaintext.

Show answer

Use the KMS v2 provider, stable since v1.29. It uses envelope encryption: the API server encrypts each object with a data encryption key derived per encryption, and the key encryption key stays in your external KMS. KMS v1 is deprecated and disabled by default, so do not start there.

Roll it out in steps. First, add an EncryptionConfiguration with the kms provider first in the list (the first provider is used for writes) and identity: {} last, so existing plaintext data can still be read. Restart each API server with it. Second, rewrite every Secret so it is stored encrypted; the docs give kubectl get secrets --all-namespaces -o json | kubectl replace -f -, done per namespace on big clusters and retried on conflicts. Third, once everything is rewritten, remove the identity provider, so the API server stops accepting plaintext data. Finally, treat old etcd backups as plaintext secrets: rotate the credentials they contain and expire those backups.

Plan for the KMS as a dependency. The API server waits 3 seconds by default for the plugin, and the docs recommend rotating the KEK at least every 90 days; rotation does not need an API server restart.

Trade-offs

External KMS gives key separation and audit, but adds a component the API server needs for every encrypted read and write.

Common mistake

Enabling encryption and assuming old Secrets are now encrypted, without rewriting them or removing identity.

Follow-up

What does your recovery plan look like if the KMS key is deleted or unreachable during an etcd restore?

Migration-phase EncryptionConfiguration · yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - kms:
          apiVersion: v2
          name: cloud-kms
          endpoint: unix:///var/run/kms/socket.sock
      - identity: {}
Question 150HardControl plane

Every write to a self-managed cluster suddenly fails and etcd reports a NOSPACE alarm. What happened, and how do you recover?

#

Deploys, leases and status updates all fail. Reads still work. etcd metrics show the database at its size quota after weeks of heavy CRD and Event churn.

Show answer

etcd enforces a backend size quota. The default is 2 GiB (set with --quota-backend-bytes), and the etcd docs suggest 8 GiB as a maximum for normal environments. When a member exceeds the quota, etcd raises a cluster-wide NOSPACE alarm and accepts only reads and deletes. For Kubernetes that means nothing can be written, including leader-election leases, so controllers stall too.

Recovery has three steps. Compact old revisions up to the current one. Defragment to give the freed space back; defrag is per member and blocks that member's reads and writes while it runs, so do members one at a time. Then disarm the alarm, which does not clear on its own.

Then fix the cause. Make sure the API server's periodic compaction is working, find which resource types grew (Events, large custom resources, frequently rewritten objects from a misbehaving controller), and fix the writer. Alert on database size against the quota so you see it at 70%, not at 100%. Only raise the quota after you understand the growth.

Trade-offs

Raising the quota buys time but makes snapshots, restores and defrags slower; fixing the write source is the real fix.

Common mistake

Running defrag against all members at once in production, so every member blocks at the same time.

Follow-up

How would you find the single controller responsible for most writes to etcd?

Compact, defragment and disarm (etcd maintenance docs) · shell
rev=$(etcdctl endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9].*')
etcdctl compact $rev
etcdctl --endpoints=https://etcd-1:2379 defrag   # repeat per member, one at a time
etcdctl alarm disarm
Question 151MediumScheduling

A team gives its batch jobs a high PriorityClass, and production API pods start getting preempted. How do you design priority classes and guardrails?

#

A data team copied a PriorityClass with a value higher than the one used for customer-facing services. During a busy hour the scheduler evicted API pods to place batch pods.

Show answer

Priority does two things: higher-priority pods are scheduled first, and, by default, they can preempt lower-priority pods when there is no room. Preemption respects PodDisruptionBudgets only on a best-effort basis: if no victims fit the budgets, the scheduler preempts anyway.

Design a small, fixed set of classes owned by the platform team, for example critical platform, production serving, default, and batch. Leave the two built-in classes, system-cluster-critical and system-node-critical, to real system components. For work that should jump the queue but never evict anyone, use preemptionPolicy: Never; those pods wait at the front of the queue instead of killing others.

Then enforce it. The docs warn that in a cluster with untrusted users, anyone can create pods at the highest priority. Mark the high classes as limited resources in the ResourceQuota admission configuration (limitedResources with a PriorityClass scope); then only namespaces that hold a ResourceQuota with a matching scopeSelector can create pods at those priorities. Set one globalDefault class for pods with no class. Give batch pods short grace periods so preemption completes quickly when it does happen.

Trade-offs

Preemption protects critical work during capacity crunches, but every extra class with preemption is another way to evict production by accident.

Common mistake

Letting each team create its own PriorityClass values, which turns priority into an arms race.

Follow-up

How would you alert when preemption starts evicting pods from production namespaces?

Queue-jumping class that never evicts · yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-urgent
value: 100000
preemptionPolicy: Never
globalDefault: false
description: "Scheduled ahead of normal batch, never preempts running pods."
Question 152MediumTroubleshooting

A namespace has been Terminating for two days. Someone suggests deleting its finalizers through the API. What do you do instead?

#

A team removed an operator with Helm, then deleted its namespace. The namespace never goes away, and the cleanup script someone found online patches out its finalizers.

Show answer

A namespace with a deletion timestamp waits until its controller has deleted everything inside it. Read kubectl get namespace <ns> -o yaml and look at the status conditions. They usually point to one of two causes.

The first is discovery failure: the namespace controller must list every resource type to clean it up, so a single unavailable aggregated API (an APIService reporting not available, for example a broken metrics adapter) can stall the whole deletion. Fix or remove that APIService. The second is content with finalizers. Here, the operator was removed before its custom resources, so nothing is left to process their finalizers. Reinstall the operator long enough to delete its resources cleanly, or, if you understand exactly what the finalizer was protecting, clean that up by hand and then remove it.

The Kubernetes docs warn against removing finalizers just to unblock deletion, because they exist for a reason. Stripping them often leaves cloud load balancers, disks or DNS records that nobody owns. Prevent repeats by deleting custom resources before their operator in your teardown order.

Trade-offs

Fixing the cause takes longer than patching out finalizers, but it is the only path that does not leave orphaned infrastructure.

Common mistake

Removing the namespace finalizer, which hides the namespace while its resources and external objects remain.

Follow-up

How would you order teardown of an operator, its CRDs and its custom resources in a pipeline?

Find out why deletion is waiting · shell
kubectl get namespace payments-old -o jsonpath='{.status.conditions}'
kubectl get apiservices | grep -v True
Question 153MediumGitOps

After moving to server-side apply, deploys keep resetting replicas and fighting the HPA. How do you hand the replicas field to the autoscaler?

#

The Deployment manifest still says replicas: 3. Every pipeline run scales the Deployment back to 3, then the HPA scales it up again.

Show answer

Server-side apply records a field manager for every field. While your manifest sets spec.replicas, the pipeline's manager owns that field and reapplies its value on every run, so it fights the HPA.

The fix is to stop managing replicas in the manifest, but the order matters. The docs warn that if you remove replicas before the HPA has written to it and taken ownership, the API server sets it to the Deployment default of 1, which is an outage for a busy service. The safe basic path is to leave replicas in the manifest until the HPA writes the field and you get a conflict on it, then remove it. The docs also show a handover: apply a replicas-only manifest under a temporary field manager, then remove the field from your main manifest; the temporary manager disappears automatically once the HPA sets a new value.

In GitOps tools, also tell the tool to ignore differences on spec.replicas, otherwise it will report drift forever.

Trade-offs

Field ownership makes shared objects safe, but it turns removing a field into a migration with its own ordering.

Common mistake

Deleting replicas from the manifest in one commit and watching the Deployment drop to one pod.

Follow-up

How would you migrate a fleet from client-side apply to server-side apply without conflicts on fields someone changed with kubectl scale?

Question 154MediumSecurity

A security review finds long-lived service account token Secrets used by CI. What has Kubernetes changed about them, and how do you migrate?

#

Old clusters still have Secrets of type kubernetes.io/service-account-token. A CI system outside the cluster uses one to deploy, and nobody knows when it was created.

Show answer

Legacy service account tokens are Secrets that never expire and are not bound to any pod, so a leaked one works until someone deletes it. Kubernetes now cleans them up: since v1.29, auto-generated legacy tokens that go unused for a period (one year by default) are labelled kubernetes.io/legacy-token-invalid-since and rejected, and if still unused for another period they are deleted. The cleaner is stable since v1.30, and the period is set on kube-controller-manager.

The replacement inside the cluster is the bound token from the TokenRequest API. The kubelet mounts it through a projected volume; it expires after a lifetime (one hour by default) or when the pod is deleted, it is bound to that pod, and the kubelet refreshes it before expiry. For CI outside the cluster, use short-lived credentials: your cloud provider's identity federation for managed clusters, or a token minted per run with kubectl create token and a short duration.

Migrate by finding every token Secret, checking kubernetes.io/legacy-token-last-used, moving each user to a short-lived path, and then deleting the Secret.

Trade-offs

Short-lived tokens limit leak impact but need a working issuance path in every pipeline, which is real engineering work.

Common mistake

Removing the invalid-since label to get a broken pipeline working again and calling the issue closed.

Follow-up

How would you give an external CI system cluster access without any stored long-lived credential?

Find legacy token Secrets and when they were last used · shell
kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token \
  -L kubernetes.io/legacy-token-last-used
Question 155MediumAutoscaling

The HPA can now scale to zero. Would you drop KEDA for your queue workers?

#

Forty queue consumers use KEDA to scale to zero overnight. A new cluster runs Kubernetes v1.37, and someone proposes removing KEDA.

Show answer

HPA scale to zero is beta and enabled by default in v1.37. You set minReplicas: 0, and the HPA reports a ScaledToZero condition. It works only with Object and External metrics, and the API server rejects minReplicas: 0 without one. CPU and memory cannot drive it, because those are measured on running pods, and with zero pods there is nothing to measure.

That is the key point for the KEDA question. The HPA still needs something serving the external metric, such as queue depth, through the external metrics API. KEDA is one such adapter, plus a large catalogue of scalers and authentication for event sources. So native scale to zero removes a reason to use KEDA only if you already have another external metrics adapter you trust.

My recommendation: for new simple cases with an existing adapter, native HPA is fewer moving parts. Keep KEDA where you depend on its scalers. Either way, test the cold path: the time from the first message to the first pod being Ready is your new latency floor, so measure it against the queue's age SLO.

Trade-offs

Native scale to zero means one less controller, but you still own an external metrics pipeline; KEDA bundles that pipeline with many ready-made scalers.

Common mistake

Setting minReplicas: 0 on an HPA that scales on CPU and expecting it to work.

Follow-up

How would you keep cold-start latency acceptable for a worker whose image takes a minute to pull?

HPA that scales to zero on an external queue metric · yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: invoice-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: invoice-worker
  minReplicas: 0
  maxReplicas: 20
  metrics:
    - type: External
      external:
        metric:
          name: queue_messages_ready
        target:
          type: AverageValue
          averageValue: "30"
Question 156MediumAutoscaling

An HPA flaps between 12 and 15 replicas all day. Which knobs do you change, and in what order?

#

The service scales on CPU. Every few minutes replicas go up, new pods start, average CPU drops, and replicas go down again.

Show answer

First check the inputs. If CPU requests are far from real usage, utilization percentages are noisy. Slow-starting pods often burn CPU during startup and then look idle, which feeds the loop. The HPA already sets aside pods that are not yet Ready during CPU scale-ups and ignores CPU samples for a startup period, but a good readiness probe matters for that to help.

Then tune behavior. The HPA skips any change when the ratio of current to desired metric is within a tolerance, 10% by default. Since v1.37, per-direction tolerance is stable, so you can, for example, scale up on a 5% change and require a larger one to scale down. Scale-down already has a 300-second stabilization window that takes the highest recommendation in that window; scale-up has none by default. Add scale-down policies, such as at most 10% of pods per minute, with selectPolicy: Min if you want the gentlest one.

Change one knob at a time and watch replica count, latency and cost for a full day, because flapping often shows only at certain traffic levels.

Trade-offs

Slower scale-down reduces churn and cold starts but keeps extra pods running longer, which costs money.

Common mistake

Raising minReplicas to hide the flapping instead of fixing requests and behaviour.

Follow-up

How would you choose between CPU and a request-rate metric for this service?

Calmer scale-down, quick scale-up · yaml
behavior:
  scaleUp:
    tolerance: 0.05
  scaleDown:
    stabilizationWindowSeconds: 600
    policies:
      - type: Percent
        value: 10
        periodSeconds: 60
Question 157MediumUpgrades

After a Kubernetes upgrade, some nodes stay NotReady because the kubelet will not start. They run an older OS image. What happened?

#

Most node pools upgraded cleanly. One legacy pool built from an old OS image fails, and the kubelet log mentions cgroup v1.

Show answer

Since v1.35 the kubelet setting failCgroupV1 defaults to true, so the kubelet refuses to start on a cgroup v1 node. kubeadm's preflight check also turns this into an error for kubelet v1.35 or later. The Kubernetes blog says the cgroup v1 fallback is still available as a temporary opt-out but is scheduled for removal in v1.38.

Confirm with stat -fc %T /sys/fs/cgroup/ on the node: cgroup v2 returns cgroup2fs. The real fix is a node image with cgroup v2 enabled and kernel 5.8 or later, with the kubelet and container runtime both using the same cgroup driver (systemd is recommended for kubeadm). containerd 2.0 or later can discover the driver automatically.

Before moving workloads, check software that reads cgroup files directly. The migration guidance points to cAdvisor v0.43.0 or later and to minimum Java, Node.js and automaxprocs versions; old runtimes can misread their CPU and memory limits under cgroup v2. Setting failCgroupV1: false is acceptable only as a short bridge with a dated plan to rebuild the pool.

Trade-offs

The opt-out restores nodes quickly but keeps you on a path that a near-future release removes.

Common mistake

Upgrading the control plane and all pools at once without checking node images for cgroup v2 first.

Follow-up

How would you add a cgroup v2 check to your node image pipeline so this never reaches an upgrade?

Check the node cgroup version · shell
stat -fc %T /sys/fs/cgroup/
# cgroup2fs means cgroup v2
Question 158HardResources

A latency-sensitive service shows p99 spikes while its average CPU is 40% of its limit. Would you remove the CPU limit?

#

A multithreaded Go API has a 2-CPU limit. Dashboards show low average CPU, but p99 latency jumps during bursts and throttling metrics are high.

Show answer

CPU limits are enforced by throttling. On cgroup v2 the limit becomes a quota per period in cpu.max, and the kernel default period is 100 ms. A process with many threads can use its whole quota in the first part of a period and is then paused until the next one. Over a minute the average looks fine, but individual requests wait tens of milliseconds, which is your p99. The cgroup's cpu.stat (and container metrics built on it) shows how often this happens.

CPU requests work differently. They act as a weight under contention, so a container with an accurate request still gets its fair share when the node is busy, without a hard ceiling when it is not.

For a trusted latency-critical service, I would set accurate requests, remove the CPU limit (or set it well above burst needs), and keep memory limits, which protect the node. Make sure the runtime sizes its thread pool to the request rather than the node, for example GOMAXPROCS in Go. Keep CPU limits for untrusted or noisy tenants, and for strict isolation use Guaranteed pods with the static CPU manager.

Trade-offs

No CPU limit removes throttling but lets a bug burn spare node CPU; the request still guarantees others their share under contention.

Common mistake

Raising the limit a little at a time while watching average CPU, which never shows the throttling.

Follow-up

How would you set GOMAXPROCS or JVM CPU counts correctly once limits are removed?

Question 159MediumSecurity

Service externalIPs is deprecated. Why was it a security problem, and how do you find and replace it?

#

An on-prem cluster exposes a few services with externalIPs. The upgrade notes for v1.36 show deprecation warnings.

Show answer

spec.externalIPs lets a Service claim any IP address, and kube-proxy then captures traffic to it on the nodes. Kubernetes does not allocate or check those IPs; the docs leave that to the cluster administrator. So anyone who can create or edit a Service can claim an address another system uses and intercept its traffic. This is CVE-2020-8554, and KEP-5707 calls the field a security risk and an architectural problem.

The field is deprecated since v1.36. The KEP's tentative plan is for kube-proxy to stop programming externalIPs by default around v1.40 and to remove the code around v1.43. Until then, block new use with admission control: the DenyServiceExternalIPs admission plugin or your own policy.

To migrate, list every Service that sets the field, then move each to a LoadBalancer Service backed by a load balancer controller (on bare metal, a controller that announces addresses from an administrator-owned pool) or to a Gateway API implementation. Both put address assignment under the platform team instead of any namespace owner.

Trade-offs

A load balancer controller adds a component to run, but it makes address ownership explicit and auditable.

Common mistake

Waiting for the removal release and discovering the dependency when kube-proxy stops forwarding the traffic.

Follow-up

How would you restrict which namespaces may create LoadBalancer Services once externalIPs is gone?

List Services that still use externalIPs · shell
kubectl get svc -A -o json | jq -r '.items[] | select(.spec.externalIPs != null) | "\(.metadata.namespace)/\(.metadata.name) \(.spec.externalIPs)"'
Question 160HardScheduling

What are QoS classes (Guaranteed, Burstable, BestEffort) and eviction order?

#

A node is under memory pressure. A Guaranteed pod survives while a higher-priority Burstable pod is evicted.

Show answer

QoS is assigned at pod creation and does not change later. Guaranteed means every container has CPU and memory requests and limits, with each limit equal to its request. Burstable means at least one container has a CPU or memory request or limit but the pod is not Guaranteed. BestEffort has no CPU or memory requests or limits at all. Setting a limit without a request makes Kubernetes assign a matching request.\n\nFor node-pressure eviction the kubelet orders candidates by whether usage exceeds requests, then by pod Priority, then by the size of the excess. The documentation explicitly says the kubelet does not use QoS class to determine eviction order. That is why a high-priority Burstable pod using only its requested memory can survive while a lower-priority pod over its requests is evicted. Guaranteed pods are not evicted because of another pod consumption, though they can be evicted if system daemons exceed reserved resources.\n\nFor kernel OOM kills the classes do matter: Guaranteed pods get oom_score_adj -997, BestEffort gets 1000, and Burstable gets a value derived from its memory request as a fraction of machine capacity. The kernel kills the highest scoring task, and the container may restart according to its restart policy.

Trade-offs

Guaranteed settings give the most predictable behaviour but reserve resources even when the workload is idle.

Common mistake

Saying BestEffort is always evicted first regardless of priority, or that Guaranteed pods can never be evicted.

Follow-up

Why does setting a memory limit without a request still produce a Burstable pod with an effective request?

Question 161EasyPods

What is the downward API?

#

An application needs its pod name and its CPU limit for log lines and JVM heap sizing, without calling the API server.

Show answer

The downward API exposes pod and container metadata to a container without a Kubernetes client. It works through environment variables or a downwardAPI volume.\n\nfieldRef can expose metadata.name, metadata.namespace, metadata.uid, specific labels and annotations through both mechanisms. Environment-only fields include spec.nodeName, spec.serviceAccountName, status.hostIP and status.podIP. The volume can also project all labels or annotations as read-only text files. resourceFieldRef exposes container-level requests and limits for CPU, memory, hugepages and ephemeral storage, with an optional divisor so a value is expressed in millicores or bytes.\n\nThe important operational difference is freshness: since v1.35, in-place CPU and memory resizes are reflected in the downward API volume, but environment variables are not updated until the container restarts. If a limit is absent, kubelet reports the node allocatable maximum.

Trade-offs

Files update in place, which is good for config that is read repeatedly; env vars are simpler but frozen at container start.

Common mistake

Assuming every field works in both forms, or that an env var changes when a resource request is resized.

Follow-up

Why would you project labels as a volume instead of reading them from the API server at startup?

Question 162HardNetworking

How does externalTrafficPolicy Local versus Cluster affect source IP and balancing?

#

An application needs real client IPs for geo rules, but users report that requests sometimes hang.

Show answer

With externalTrafficPolicy: Cluster (the default), a node receiving external traffic may forward it to a pod on another node. That extra hop source-NATs the packet, so the pod sees the node IP rather than the client. Traffic is spread across every ready endpoint in the cluster.\n\nWith Local, kube-proxy only forwards to endpoints on the receiving node, so the original client IP is preserved when the provider supports it. The cost is that a node with no local endpoints drops the packet. For LoadBalancer Services, those nodes fail the healthCheckNodePort check and are removed from load balancer rotation.\n\ninternalTrafficPolicy is a separate control for pod-originated traffic: Local keeps that traffic on node-local endpoints, and pods on a node with no endpoints see no backends. Cluster considers all endpoints. Not every cloud provider preserves source IP through its load balancer; some expose it in X-Forwarded-For or proxy protocol instead.

Trade-offs

Local avoids a hop and keeps client IPs, but unhealthy distribution across nodes can leave capacity unused.

Common mistake

Assuming Local still balances across all ready pods, or that internalTrafficPolicy governs external clients.

Follow-up

What happens to a request that lands on a node with zero local endpoints under Local?

Question 163HardAutoscaling

What is Karpenter and how does it differ from the Cluster Autoscaler?

#

A batch workload needs GPUs in several instance sizes, and node groups keep running out of capacity.

Show answer

Karpenter reacts to pods that the scheduler marks Unschedulable and provisions nodes for them directly, without requiring node groups. A NodePool defines constraints such as instance families and expiry, while an EC2NodeClass holds AWS-specific settings: subnets, security groups, AMIs, instance profile, kubelet configuration and tags. Multiple NodePools can coexist.\n\nIt batches pending pods, bin-packs by CPU, memory, GPUs, daemonsets and VPC CNI address limits, then asks EC2 Fleet for a set of suitable instance types using price capacity optimised allocation. Spot selection balances price and interruption risk rather than always choosing the cheapest. Consolidation removes empty nodes, replaces several nodes with fewer, or swaps in a cheaper node, while using the Kubernetes eviction API and respecting PodDisruptionBudgets. Drift remediation is enabled by default from v0.33, and Karpenter v1 is the first stable API.\n\nCluster Autoscaler changes the size of existing node groups; Karpenter chooses fresh instance types and sizes. Neither scales pods: HPA, VPA or Keda do that, and the two node scalers can run side by side.

Trade-offs

Direct provisioning is more flexible and faster, but requires careful NodePool limits to avoid sprawl or unexpected spend.

Common mistake

Calling Karpenter a pod autoscaler, or assuming consolidation ignores PodDisruptionBudgets.

Follow-up

How would you stop Karpenter from replacing every node during a working-day traffic spike?

Question 164MediumNetworking

What are EndpointSlices and why do they exist?

#

A Service has thousands of backends, and a single pod restart causes a large update to every node.

Show answer

The old Endpoints API stored every backend IP for a Service in one object. Any single pod change forced that entire object to be sent to every kube-proxy, and etcd 1.5 MB object limit capped a resource at roughly 5,000 pod IPs.\n\nEndpointSlices shard the same information into smaller objects, holding addresses, ports, readiness and topology. By default a slice holds up to 100 endpoints, configurable with the controller manager flag --max-endpoints-per-slice, so a single pod change updates one small slice instead of the whole set. kube-proxy has watched EndpointSlices by default since Kubernetes 1.19, and the design supports Services well beyond 100,000 endpoints.\n\nTopology fields and addressType also enable dual-stack IPv4 and IPv6 and topology-aware routing. The Endpoints API remains stable and has not been removed, though very large Endpoints objects can be truncated.

Trade-offs

Slicing reduces update churn but adds more objects to watch and inspect during debugging.

Common mistake

Saying Endpoints was removed, or that slicing eliminates all endpoint watch traffic.

Follow-up

How would you find which slice contains a particular pod IP?