Day 03 Theory — HTTP Clients, API Failures, Rate Limits and Backoff

Why this matters for AI systems

An AI application rarely lives alone. It calls model providers, vector databases, SaaS APIs, internal microservices, identity systems, and observability backends. The moment our Service Desk depends on a remote service, the network becomes part of our program.

Networks fail differently from local functions. A remote call can be slow, rejected, rate-limited, partially completed, or completed even though the client never receives the response.

Today is about learning to treat HTTP failure as a normal engineering condition rather than an unexpected exception.

The HTTP request lifecycle in simple terms

A client usually does more than “send JSON.” At a high level:

Application
   │
   ├─ resolve destination
   ├─ establish connection / TLS
   ├─ send request headers + body
   ├─ wait for server work
   └─ receive status + headers + body

A failure can happen at any of these stages.

That is why we distinguish transport failures from HTTP responses.

A DNS error or connection timeout may mean the server never received the request. A 500 means the server did receive enough of the request to return an HTTP response. Those situations should not automatically be handled in the same way.

Status codes are signals, not retry instructions

Useful groups:

• 2xx: the request was accepted successfully at the HTTP level.
• 4xx: the request or caller usually needs to change something.
• 5xx: the server failed while processing the request.

But never reduce this to “retry every 5xx and never retry a 4xx.” Real APIs define their own semantics.

For example, 429 Too Many Requests is a client-side status code but often explicitly invites a later retry. Some 409 Conflict responses may be retryable in a particular workflow. Provider documentation is authoritative.

Retryability depends on the operation

Before retrying, ask two separate questions:

1. Is the failure transient?
2. Is repeating this operation safe?

A GET request that reads ticket details is usually easier to retry than a POST that creates an incident or charges a card.

This leads to the concept of idempotency.

An idempotent operation can be repeated without creating an additional unintended effect. Some APIs provide idempotency keys so clients can safely repeat a create request after an uncertain failure.

The dangerous timeout scenario

Consider:

Client ── create ticket ──> Server
Client <──── timeout ────── ?

The client does not know whether the server failed or whether the server created the ticket and only the response was lost.

Blindly retrying could create two tickets.

This is one of the most important production lessons in the course: a timeout does not prove that nothing happened.

Why exponential backoff exists

If a service is overloaded and every failed client retries immediately, retries add even more traffic.

Exponential backoff increases the delay after repeated failures:

attempt 1 -> short delay
attempt 2 -> longer delay
attempt 3 -> longer again

A simplified form is:

delay = base * (2 ** attempt)

Real systems also cap the maximum delay and the maximum number of attempts.

Why jitter matters

Imagine thousands of clients all failing at the same moment. If every client waits exactly two seconds, they all retry together two seconds later. This synchronized retry wave is sometimes called a thundering herd.

Jitter adds randomness so retry traffic is spread across time.

delay = exponential_delay + random.uniform(0, jitter_window)

The exact algorithm should match the provider's recommendations when available.

Rate limits are capacity contracts

Rate limiting protects a service from excessive use. Limits may be based on requests, tokens, concurrent operations, users, organizations, or time windows.

A robust client should observe provider headers and documented retry guidance instead of guessing.

The correct response to a rate limit may include:

• wait and retry later,
• reduce concurrency,
• queue work,
• use a different capacity tier,
• return a graceful failure to the user.

Connection pooling

Creating a brand-new network connection for every request adds overhead. HTTP clients such as httpx.AsyncClient can reuse connections.

That is why long-lived clients are often preferable to constructing a new client inside every tiny helper call.

Connection pooling improves efficiency, but it also creates a resource lifecycle that must be closed correctly.

Error handling should preserve meaning

Do not turn every failure into:

except Exception:
    return None

That destroys useful information.

The rest of the program may need to know whether the cause was authentication, a rate limit, invalid input, an upstream outage, or a timeout.

Good error boundaries translate low-level failures into meaningful application-level failures without erasing the original context needed for debugging.

Service Desk connection

Our Service Desk is becoming dependent on remote systems. Today we design its HTTP layer so future LLM calls, ticket integrations, and knowledge services do not each invent their own retry behavior.

The central principle is:

Retries are a policy decision based on failure type, operation semantics, idempotency, and provider guidance—not a generic exception handler.

After today you should be able to explain why an API call failed, whether it is safe to retry, and how your retry policy avoids turning a small outage into a larger one.