Day 32 Theory — Reliability: Retry Budgets, Circuit Breakers, Queues and Backpressure

Why retries are no longer enough

Earlier we learned how an individual HTTP call can retry a transient failure. That is useful at small scale.

Now imagine 1,000 Service Desk requests are active when the model provider starts failing.

If every request immediately retries several times, the system may transform:

1,000 user requests

into:

3,000–5,000 dependency calls

exactly when the dependency is least able to handle them.

This is called failure amplification.

Reliability engineering is therefore not “add more retries.” It is about controlling how the whole system behaves when capacity or dependencies degrade.

Reliability is an end-to-end property

A user request may cross several components:

Client
  -> API
  -> Agent runtime
  -> Model provider
  -> Vector store
  -> Tool API
  -> Database

Each dependency can fail independently.

If every layer has its own aggressive retry policy, the combined behavior may be much worse than any single component intended.

Ask:

Which layer owns the retry, and how much additional load is the system allowed to create?

Retry budgets

A retry budget limits retry amplification.

Instead of treating retries as free, the service gives them bounded capacity.

Possible policies include:

• maximum attempts per operation,
• maximum percentage of traffic that may be retries,
• shared token/budget across a dependency,
• deadline-aware retries that stop when useful time is exhausted.

The exact policy depends on the service. There is no universal retry count.

The mental model is:

original traffic  ---------> dependency
      |
      +-- limited retry budget
               |
               +-----------> dependency

not:
original failure -> unlimited retry storm

A retry budget answers:

How much extra load are we willing to create in exchange for recovering transient failures?

Retries need time budgets too

Suppose a user-facing request has only a few seconds of useful latency budget remaining.

Starting another long retry after most of that budget is consumed may guarantee a timeout without improving the user experience.

A robust runtime considers:

• caller deadline,
• per-attempt timeout,
• backoff delay,
• remaining useful time.

Retries should stop when success would arrive too late to matter.

Circuit breakers

A circuit breaker prevents a service from repeatedly calling a dependency that appears unhealthy.

The classic mental model has three states:

CLOSED
calls flow normally
   |
   | failure policy opens circuit
   v
OPEN
calls fail fast / use fallback
   |
   | recovery interval / probe policy
   v
HALF-OPEN
allow limited probes
   |
   +-- healthy --> CLOSED
   |
   +-- failing --> OPEN

Why fail fast?

If the model provider is clearly unavailable, waiting several seconds on every request may consume:

• worker capacity,
• connection pools,
• memory,
• user latency budget.

Failing quickly can preserve capacity for work that can still succeed.

What a circuit breaker does not do

It does not repair the dependency.

It only changes caller behavior while the dependency is unhealthy.

You still need monitoring, incident response and recovery.

Choosing circuit conditions

Do not copy a magic rule such as “open after five failures.”

Consider signals such as:

• error rate over a meaningful window,
• consecutive failures where appropriate,
• timeout rate,
• latency degradation,
• dependency-specific failure categories.

A policy should avoid opening because of one isolated request while also avoiding long periods of destructive retry traffic.

Queues separate arrival rate from processing rate

Sometimes work does not need to finish inside the original HTTP request.

A queue allows producers to submit work while workers process it at controlled concurrency.

Requests ---> Queue ---> Worker pool ---> Dependency
                 |
                 +-- absorbs temporary burst

This can smooth bursts, but a queue does not create infinite capacity.

If work arrives at 1,000 jobs/minute and workers can permanently process only 500 jobs/minute, the backlog grows forever.

That is overload hidden behind storage.

The metrics that make a queue meaningful

Queue depth alone is not enough.

Also watch:

• oldest message age — how long users are actually waiting,
• arrival rate,
• completion rate,
• worker concurrency,
• failure/retry rate,
• dead-letter volume where used.

A queue of 10,000 items may be fine for one workload and catastrophic for another. Age and service objective provide context.

Backpressure

Backpressure means downstream capacity influences how much upstream work is accepted or allowed to proceed.

Without backpressure:

users -> API -> unbounded work -> exhausted workers -> dependency collapse

With backpressure:

users
  |
  v
admission / limits
  |
  +-- capacity available -> process
  |
  +-- capacity exhausted -> queue bounded work / reject / shed

Mechanisms can include:

• bounded queues,
• concurrency semaphores,
• rate limiting,
• admission control,
• load shedding.

The important idea is that the system refuses to promise work it cannot safely carry.

Load shedding is sometimes the reliable choice

Returning a controlled 503 or “try again later” can be healthier than accepting every request and timing all of them out slowly.

A degraded system that serves 80% of requests correctly may be more useful than one that accepts 100% and completes almost none.

The product must decide which work is most important.

Graceful degradation

Dependencies have different criticality.

For the Service Desk:

Knowledge recommendation unavailable
    -> maybe answer with a limited fallback

Authorization unavailable
    -> do NOT execute privileged action

This illustrates a key rule:

Availability must never override a safety invariant.

A fallback that weakens authorization is not graceful degradation; it is a security bug.

Bulkheads and isolation

Another reliability idea is to isolate capacity so one workload cannot consume everything.

For example:

password-reset workers: bounded pool
knowledge-search workers: bounded pool
security-escalation workers: reserved capacity

If one path becomes overloaded, the entire system does not necessarily collapse with it.

This is similar to watertight compartments in a ship: failure is contained rather than allowed to flood every section.

Retry + circuit breaker + queue + backpressure are complementary

These mechanisms solve different problems.

Mechanism	Primary purpose
Retry	Recover a transient failure
Retry budget	Bound retry amplification
Circuit breaker	Stop repeatedly calling an unhealthy dependency
Queue	Decouple temporary arrival bursts from worker execution
Backpressure	Prevent accepting more work than the system can safely carry
Load shedding	Preserve useful service during overload

Using one does not eliminate the need for the others.

Service Desk failure walkthrough

Imagine the certificate API slows dramatically.

A weak design:

request
 -> timeout
 -> retry
 -> retry
 -> retry
 -> worker remains occupied
 -> more users arrive
 -> all workers fill
 -> entire API becomes slow

A controlled design might be:

request
 -> bounded call
 -> retry allowed only within budget
 -> breaker observes sustained failure
 -> circuit opens
 -> new certificate requests fail fast / enter approved queue
 -> unrelated ticket paths keep working
 -> limited probes detect recovery
 -> circuit closes

The second design does not guarantee zero failures. It prevents one failure from turning into a system-wide collapse.

What to debug during an incident

Ask in order:

1. Which dependency is saturated or failing?
2. How much traffic is original versus retry traffic?
3. Are callers respecting deadlines?
4. Is the circuit state behaving as expected?
5. Is queue age growing?
6. Is concurrency bounded?
7. Are we shedding the correct class of work?
8. Are safety-critical paths failing closed?

This is much more useful than simply increasing replica count without understanding the bottleneck.

Service Desk connection

Today our Service Desk learns how to remain useful when parts of the system are unhealthy.

The architectural change is:

Before
------
individual calls with retry logic

After
-----
system-level reliability policy:
retry budgets
+ circuit breakers
+ bounded concurrency/queues
+ backpressure
+ deliberate degradation

The principle is:

Reliable systems bound failure amplification. Retry only within budget, stop hammering unhealthy dependencies, accept only work you can carry, and degrade availability before you degrade correctness or security.