Module 1: Engineering Foundations · 3.5h
01 · UNDERSTAND
Day 08 Theory — Streaming with SSE, Cancellation and Partial Failure
Why streaming changes the user experience
An LLM may take time to produce a full answer. If an API waits for the entire result before sending anything, the user sees silence and may assume the application is stuck.
Streaming lets the server send useful progress while work is still happening.
Today we use Server-Sent Events (SSE) to understand streaming as a protocol and reliability problem—not just as a visual typing effect.
Ordinary HTTP response versus streaming response
Normal request:
Client ── request ──> Server
Client <── full response after work completes ── Server
Streaming:
Client ── request ──> Server
Client <── event 1 ── Server
Client <── event 2 ── Server
Client <── event 3 ── Server
Client <── complete ─ Server
The connection remains open while multiple events are delivered.
What SSE provides
SSE uses an HTTP response with text/event-stream. The server emits UTF-8 event records over a long-lived response.
A simple record can contain:
event: token
data: hello
The blank line terminates the event.
Browsers provide an EventSource API for common SSE use cases, although authenticated application designs sometimes use other clients depending on header and transport needs.
SSE is one-way
SSE is designed primarily for server-to-client streaming over HTTP.
If the application needs continuous bidirectional messages over one connection, WebSockets may be a better fit.
Choose a transport based on communication semantics, infrastructure support, and failure handling—not because one technology sounds more modern.
Streaming introduces a protocol of events
Do not stream arbitrary text if the client needs structured progress.
Define event types such as:
start
progress
token
tool_started
tool_finished
warning
error
complete
Each event should have a documented payload shape.
This becomes increasingly important later when agents perform tools and multi-step workflows.
Client disconnects are normal
A user can close the browser, navigate away, lose connectivity, or cancel a request.
The server should detect that the client is gone and stop expensive work when appropriate.
Continuing a costly LLM generation for a response nobody can receive may waste capacity and money.
But cancellation semantics depend on the operation. If a side effect was already committed, “cancelled by client” does not undo it automatically.
Partial failure
Streaming creates states that a single-response API can hide.
Suppose we have already emitted several tokens and then a provider fails. We cannot pretend no response occurred.
The stream needs an explicit terminal failure event or connection behavior the client understands.
The UI should be able to distinguish:
- completed successfully,
- cancelled by the user,
- interrupted by a recoverable error,
- failed after partial output.
Backpressure begins here
If a producer generates events faster than a consumer or network can handle them, data can accumulate in memory.
Today's examples are small, but the principle will return later in the production streaming and reliability modules.
A robust streaming architecture controls buffers, queue sizes, and producer speed.
Proxies and buffering
Streaming behavior can be affected by reverse proxies, CDNs, gateways, and server configuration. A proxy that buffers the response can destroy the perceived benefit of streaming even though the application code emits events correctly.
Production testing must include the actual network path, not only localhost.
Service Desk connection
Today the Service Desk can begin showing progress instead of forcing users to wait for a final answer. Later the same event model can represent retrieval, tool calls, approvals, and agent steps.
The key principle is:
A stream is a long-lived protocol with lifecycle, cancellation, failure and buffering semantics. Treat it as an API contract, not a UI animation.
02 · APPLY
Lesson Overview
This is the applied companion for Day 08. Read DAY_08_THEORY.md first for the beginner-first teaching of Streaming with SSE, Cancellation & Partial Failure. Then use the real service-desk-day-08/ project to trace, run, debug, and explain the concept.
Service Desk Alignment
Day 08 adds Streaming with SSE, Cancellation & Partial Failure to the running Service Desk. Start with app.py, stream.py, services.py, models.py, then follow imports and tests to identify the actual runtime path.
Why This Topic Matters
The theory chapter explains why Streaming with SSE, Cancellation & Partial Failure is needed. Here the goal is evidence: identify where the capability is implemented, what observable behavior changes, and how the repository proves both success and failure behavior.
System Architecture
graph TD
A[Client EventSource] --> B[GET /api/v1/stream]
B --> C[Async Token Generator]
C -->|Chunk 1| A
C -->|Chunk 2| A
A -.->|Client Abort| D[is_disconnected() == True]
D --> E[Terminate Generator & Release Resources]
Worked Code Example: SSE Streaming
import asyncio
from fastapi import FastAPI, Request
from sse_starlette.sse import EventSourceResponse
app = FastAPI()
async def token_generator(request: Request):
tokens = ["Resolving", " Service", " Desk", " Ticket", " #1042...", " Done!"]
try:
for token in tokens:
if await request.is_disconnected():
print("Client disconnected! Cleaning up token generator...")
break
yield {"event": "token", "data": token}
await asyncio.sleep(0.2)
except asyncio.CancelledError:
print("Streaming task cancelled by client disconnect.")
@app.get("/api/v1/stream")
async def stream_resolution(request: Request):
return EventSourceResponse(token_generator(request))
Code Walkthrough & Mechanics
Read app.py, stream.py, services.py, models.py with these theory sections beside you:
- Why streaming changes the user experience — locate its implementation and evidence.
- Ordinary HTTP response versus streaming response — locate its implementation and evidence.
- What SSE provides — locate its implementation and evidence.
- SSE is one-way — locate its implementation and evidence.
- Streaming introduces a protocol of events — locate its implementation and evidence.
For each concept identify the real function/class/protocol boundary, its input/state, its observable result, and the assertion in tests/test_streaming.py that proves the behavior. If a concept has no implementation or evidence, record that as a gap rather than inventing one.
Common Mistakes & Debugging Guidance
Use the theory—not generic timeout or .env advice—to decide what can fail today.
Failure lens: revisit SSE is one-way, Streaming introduces a protocol of events.
- Reproduce the smallest case that violates one of those expectations.
- Trace the real Day 08 modules until you find the first incorrect state/output/decision.
- Use
tests/test_streaming.pyas executable evidence. - Add a regression test if the failure is not already represented.
- Fix the smallest responsible boundary and rerun the relevant test before the full suite.
Your debugging explanation must name the topic-specific invariant that failed, not merely say “an exception occurred.”
Practical Lab Instructions
- Summarize these theory ideas before opening the implementation:
- Why streaming changes the user experience
- Ordinary HTTP response versus streaming response
- What SSE provides
- SSE is one-way
- Inspect the most relevant real Day 08 modules first:
service_desk/app.pyservice_desk/stream.pyservice_desk/services.pyservice_desk/models.py
- Inspect the automated evidence:
tests/test_streaming.py
- Establish the baseline:
cd service-desk-day-08 PYTHONPATH=. pytest tests/test_streaming.py -q - Trace one theory concept through the actual nested modules and tests.
- Run one success case and record input → mechanism → observable result.
- Exercise one topic-specific failure/boundary case and name the invariant that protects the system.
- Re-run the relevant tests and explain theory → implementation → evidence.
Done when: another student can reproduce your trace without relying on an invented sample.
Key Takeaways
- Be able to explain Why streaming changes the user experience and point to its implementation/evidence in Day 08.
- Be able to explain Ordinary HTTP response versus streaming response and point to its implementation/evidence in Day 08.
- Be able to explain What SSE provides and point to its implementation/evidence in Day 08.
Knowledge Check & Scenario Questions
- Concept: Using Why streaming changes the user experience, explain the engineering problem Day 08 is solving without naming a framework as the answer.
- Mechanism: How does Ordinary HTTP response versus streaming response appear in the real project? Start from
service_desk/app.pyand name the observable state/output/event that changes. - Failure: For What SSE provides, describe one incorrect implementation or boundary condition and the evidence you would expect in
tests/test_streaming.py. - Design review: Which assumption in today's design would you verify before reusing this implementation in a different production system?
Official References
- FastAPI Custom Responses & Streaming: https://fastapi.tiangolo.com/advanced/custom-response/#streamingresponse
- MDN Server-Sent Events Standard: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events
03 · EXPLAIN
Interview checkpoint
Explain one design decision from this lesson, the alternative you rejected, and the failure mode or evidence that justified your choice.