Reconciliation and eventual convergence

The mechanism

A controller repeatedly compares observed state with desired state and takes actions to reduce the difference. It does not receive a guarantee that one API write immediately creates the intended runtime condition. Delays, retries and external failures are normal parts of the loop.

Controller logic should tolerate repeated observations and already-completed actions. If the desired replica count remains two, repeated reconciliation should not keep adding Pods. The controller works through API objects and ownership relationships so it can distinguish resources it manages from unrelated ones.

For applications, convergence needs a meaningful definition. Two running processes may not mean two ready replicas, and two ready replicas may not prove correct end-to-end behavior. Use progressively stronger evidence rather than one green status label.

Worked example

This is controller pseudocode, not Kubernetes controller-library code. It illustrates the loop without claiming that a real ReplicaSet implementation uses this exact algorithm. Production controllers must handle concurrency, stale observations and failures.

def proposed_replica_actions(desired, observed):
    if desired < 0 or observed < 0:
        raise ValueError("Replica counts must be nonnegative")
    return {"create": max(0, desired - observed),
            "excess": max(0, observed - desired)}

assert proposed_replica_actions(2, 1)["create"] == 1
assert proposed_replica_actions(2, 2)["create"] == 0

Practice: predict, inspect, explain

Offline exercise. Walk through observations 0, 1, 1 and 2 for a desired count of two. Explain why a real controller needs to account for already-requested work before acting on a stale observation. Then add a Pod that exists but is not ready.

Expected observation: creation intent, object existence and readiness are different states. Draw a timeline with delayed API observations and a failed startup. Identify the condition that tells an operator to investigate rather than assuming another reconciliation cycle will solve the application problem.

Troubleshooting and trade-offs

If a controller continually recreates objects, inspect ownership, selectors and competing managers. If the desired state is impossible because of quota or scheduling constraints, retries alone cannot make it possible. Do not repair a managed child object without understanding whether its controller will revert the change. Change the owning desired configuration when that is the intended solution.

Interview practice

Completion check

Trace a delayed reconciliation sequence and identify where repeated actions could become unsafe.

Sources and version notes

Baseline checked 6 October 2026: the official release page lists Kubernetes 1.37.1. Verify your cluster and distribution prerequisites. All manifests are offline teaching examples; no cluster mutations or cloud resources are executed by this handbook.

• Official documentation: Controller
• Official documentation: Deployment
• Official documentation: Working with objects