Module 1: Engineering Foundations · 4h

01 · UNDERSTAND

Day 04 Theory — How LLM Applications Work and Structured Output

From software function to model call

Until now, most of our Service Desk behavior has been deterministic: given the same validated input and the same program state, our code follows rules we wrote.

An LLM introduces a different kind of component. We provide instructions and context; the model generates a probabilistic continuation. That is powerful, but it changes how we design boundaries.

The model should not become the authority for everything. Our software still owns validation, permissions, side effects, and business invariants.

What actually goes into an LLM request

A model call is not just “the user's question.” It often contains several layers:

System/developer instructions
Conversation history
Retrieved context
Tool descriptions
User request
Output constraints

All of this becomes input tokens processed by the model.

The model then generates output tokens one at a time according to a probability distribution conditioned on the context it can see.

Training versus inference

Training changes model parameters by learning patterns from data.

Inference uses already-trained parameters to produce a prediction or generation for a new input.

When our Service Desk sends an API request to an LLM provider, we are normally performing inference. We are not retraining the model with each prompt.

This distinction matters because sending private data in a prompt is not the same thing as “teaching the model permanently,” although provider data-retention and training policies still matter and must be checked separately.

Tokens and the context window

Models process tokens rather than human concepts such as “one paragraph” or “one page.”

The context window limits how much input and generated output can participate in a single request. Instructions, conversation history, tool schemas, and retrieved documents all compete for that space.

A larger context window is useful, but it does not remove the need for context engineering. More text can increase cost, latency, distraction, and conflicting instructions.

Probabilistic output does not mean uncontrolled software

A common design mistake is to treat an LLM like a normal function that guarantees a semantically correct object.

The model may:

misunderstand the request,
choose an incorrect label,
invent a value,
omit an important fact,
follow malicious context,
return a structurally valid but semantically wrong result.

This is why production AI applications surround model calls with deterministic software boundaries.

What structured output solves

Suppose the Service Desk asks a model to classify a ticket.

Free-form output might look like:

“This sounds like a high-priority authentication issue.”

Useful for a human, but inconvenient for software.

We would prefer:

{
  "category": "AUTHENTICATION",
  "priority": "HIGH",
  "summary": "User cannot access SSO"
}

Provider-native structured output constrains generation to a schema when the provider supports that feature. This is stronger than merely writing “please return JSON” in a prompt.

Schema validity is not semantic truth

This distinction is essential.

If the schema says priority must be one of LOW, MEDIUM, or HIGH, the provider may successfully return:

{"priority": "LOW"}

The response is structurally valid. It may still be the wrong priority.

So we need two layers:

Structural validation — does the result conform to the expected shape and types?
Semantic evaluation — is the content actually correct for the task?

Pydantic is excellent for the first layer. Evaluation and business rules help with the second.

Validation, authorization and correctness are different

A Pydantic model can prove that a field has the expected type. It does not prove the caller is allowed to perform an action.

For example:

class ResetPasswordRequest(BaseModel):
    user_id: str

This validates that user_id is a string. It does not prove that the current caller may reset that user's password.

Keep these concepts separate:

validation: is the data shaped correctly?
authorization: is this actor permitted?
correctness: is the chosen action or answer right?

Provider abstraction without pretending providers are identical

OpenAI, Anthropic, Gemini and other providers expose similar capabilities, but their SDKs, model families, structured-output features, token accounting, safety behavior and failure modes are not identical.

An abstraction can give our application a stable internal interface, but it should not erase provider-specific capabilities that matter.

A safe mental model

Untrusted input
    ↓
Application validation
    ↓
Prompt/context construction
    ↓
LLM inference
    ↓
Structured output constraint
    ↓
Application validation + policy checks
    ↓
Use result

The model is one component inside a larger software system.

Service Desk connection

Today our Service Desk crosses the boundary from deterministic application code into model-assisted behavior. The model can help classify and summarize tickets, but the surrounding application remains responsible for schemas, permissions, fallbacks, observability, and evaluation.

The principle to remember is:

Structured output can make model responses machine-readable. It cannot make them automatically true, authorized, or safe.

02 · APPLY

Lesson Overview

This is the applied companion for Day 04. Read DAY_04_THEORY.md first for the beginner-first teaching of How LLM Applications Work + Structured Output. Then use the real service-desk-day-04/ project to trace, run, debug, and explain the concept.

Service Desk Alignment

Day 04 adds How LLM Applications Work + Structured Output to the running Service Desk. Start with models.py, services.py, classifier.py, then follow imports and tests to identify the actual runtime path.

Why This Topic Matters

The theory chapter explains why How LLM Applications Work + Structured Output is needed. Here the goal is evidence: identify where the capability is implemented, what observable behavior changes, and how the repository proves both success and failure behavior.

System Architecture

graph TD
    A[Raw Model Output JSON] --> B[Pydantic V2 Type Validator]
    B -->|Field Valid| C[Instantiated TicketTriageModel]
    B -->|ValidationError| D[Catch Error & Trigger Re-prompt / Fallback]

Worked Code Example

from pydantic import BaseModel, Field, field_validator
from typing import List

class TicketTriageModel(BaseModel):
    category: str = Field(..., description="Triage category: ACCESS, BILLING, HARDWARE, or OTHER")
    priority: str = Field(..., description="Priority tier: LOW, MEDIUM, HIGH, URGENT")
    affected_systems: List[str] = Field(default_factory=list)
    requires_escalation: bool = False

    @field_validator("priority")
    @classmethod
    def check_priority(cls, value: str) -> str:
        allowed = {"LOW", "MEDIUM", "HIGH", "URGENT"}
        if value.upper() not in allowed:
            raise ValueError(f"Priority must be in {allowed}")
        return value.upper()

# Generate strict OpenAPI / JSON Schema for OpenAI Structured Outputs
json_schema = TicketTriageModel.model_json_schema()
print("Generated JSON Schema for Model Function Calling:", json_schema["properties"].keys())

Detailed Code Explanation

Read models.py, services.py, classifier.py with these theory sections beside you:

From software function to model call — locate its implementation and evidence.
What actually goes into an LLM request — locate its implementation and evidence.
Training versus inference — locate its implementation and evidence.
Tokens and the context window — locate its implementation and evidence.
Probabilistic output does not mean uncontrolled software — locate its implementation and evidence.

For each concept identify the real function/class/protocol boundary, its input/state, its observable result, and the assertion in tests/test_structured_classifier.py that proves the behavior. If a concept has no implementation or evidence, record that as a gap rather than inventing one.

Common Mistakes & Debugging Guidance

Mistake 1: Using regular regex parsing on raw LLM strings instead of strict Pydantic V2 schema validation.
Mistake 2: Omitting Field descriptions, depriving the model of semantic guidance during JSON generation.
Mistake 3: Failing to handle ValidationError exceptions when the model produces partial or corrupted JSON.

Practical Lab Instructions

Summarize these theory ideas before opening the implementation:
- From software function to model call
- What actually goes into an LLM request
- Training versus inference
- Tokens and the context window
Inspect the most relevant real Day 04 modules first:
- service_desk/models.py
- service_desk/services.py
- service_desk/classifier.py
Inspect the automated evidence:
- tests/test_structured_classifier.py

Establish the baseline:

cd service-desk-day-04
PYTHONPATH=. pytest tests/test_structured_classifier.py -q

Trace one theory concept through the actual nested modules and tests.
Run one success case and record input → mechanism → observable result.
Exercise one topic-specific failure/boundary case and name the invariant that protects the system.
Re-run the relevant tests and explain theory → implementation → evidence.

Done when: another student can reproduce your trace without relying on an invented sample.

Key Takeaways

Pydantic V2 schemas enforce deterministic type boundaries between stochastic models and application code.
Use @field_validator to implement custom business rule checks (e.g. enum validation) on LLM outputs.
Always catch ValidationError at the application edge and implement structured repair fallbacks.

Knowledge Check & Scenario Questions

Knowledge Check: What role do Pydantic Field(description=...) annotations play in structured output generation?
- Answer: They provide JSON Schema descriptions that guide the LLM's function calling argument generation.
Scenario Question: How does Pydantic V2 handle missing required payload fields during model validation?
- Answer: It immediately raises a ValidationError detailing the missing field paths and expected types.

Official References

Pydantic V2 Official Documentation: https://docs.pydantic.dev/latest/
OpenAI Structured Outputs Guide: https://platform.openai.com/docs/guides/structured-outputs

03 · EXPLAIN

Interview checkpoint

Explain one design decision from this lesson, the alternative you rejected, and the failure mode or evidence that justified your choice.