Module 6: Evaluation & Production Ops · 2.25h
01 · UNDERSTAND
Day 31 Theory — Identity, Authorization, Multi-Tenancy and PII
Why identity must enter the agent architecture
An agent with tools can read and change real systems. The runtime needs to know who is asking, what they are allowed to do, and which tenant or organization the data belongs to.
The model cannot be the authority for these decisions.
Authentication versus authorization
Authentication answers:
Who is this caller?
Authorization answers:
May this caller perform this action on this resource?
A valid login does not imply permission to reset another user's password or read another tenant's tickets.
Carry trusted identity separately from prompt text
Do not ask the model to extract authorization identity from sentences such as:
“I am an administrator.”
Identity should come from a trusted authentication layer and be passed to the runtime as structured context.
Multi-tenant isolation
A multi-tenant system must prevent one tenant's data from appearing in another tenant's:
- database queries,
- vector retrieval,
- checkpoints,
- memory,
- logs,
- caches.
Tenant filtering should be enforced at deterministic data-access boundaries.
PII minimization
Personally identifiable information should be included in model context only when the task genuinely requires it.
Useful controls include:
- redaction/tokenization,
- purpose limitation,
- field-level access,
- retention limits,
- secure logging.
Do not log entire prompts by default if they contain sensitive data.
Authorization at tool execution
A model can propose any tool call it has learned to request. The runtime checks permissions immediately before the tool executes.
This protects the system even if the model is confused or manipulated.
Service Desk connection
Today the Service Desk stops treating every request as if it came from one trusted demo user.
The principle is:
Identity is trusted application context; authorization is enforced at resource/action boundaries; tenant and PII isolation must survive retrieval, memory, tools and observability.
02 · APPLY
Lesson Overview
This is the applied companion for Day 31. Read DAY_31_THEORY.md first for the beginner-first teaching of Identity, Authorization, Multi-Tenancy and PII. Then use the real service-desk-day-31/ project to trace, run, debug, and explain the concept.
Service Desk Alignment
Day 31 adds Identity, Authorization, Multi-Tenancy and PII to the running Service Desk. Start with security/authz.py, rag/tenant_rag.py, security/context.py, tools/ticket_tools.py, privacy/redactor.py, agent/mock_model.py, then follow imports and tests to identify the actual runtime path.
Why This Topic Matters
The theory chapter explains why Identity, Authorization, Multi-Tenancy and PII is needed. Here the goal is evidence: identify where the capability is implemented, what observable behavior changes, and how the repository proves both success and failure behavior.
System Architecture
This is a repository surface map, not a claim that modules call each other in the displayed order. The modules are ranked by relevance to today's theory.
graph LR
T[Day 31: Identity, Authorization, Multi-Tenancy and PII]
T --> M1[security/authz.py]
T --> M2[rag/tenant_rag.py]
T --> M3[security/context.py]
T --> M4[tools/ticket_tools.py]
T --> M5[privacy/redactor.py]
T --> M6[agent/mock_model.py]
T --> M7[agent/secure_agent.py]
T --> M8[privacy/telemetry_scrubber.py]
Follow imports and tests to discover the actual runtime flow.
Repository Implementation Map
Use the real Day 31 repository, not a fabricated sample, to connect theory to implementation.
Theory concepts to locate:
- Why identity must enter the agent architecture
- Authentication versus authorization
- Carry trusted identity separately from prompt text
- Multi-tenant isolation
Most relevant implementation modules first:
service_desk/security/authz.pyservice_desk/rag/tenant_rag.pyservice_desk/security/context.pyservice_desk/tools/ticket_tools.pyservice_desk/privacy/redactor.pyservice_desk/agent/mock_model.pyservice_desk/agent/secure_agent.pyservice_desk/privacy/telemetry_scrubber.pyservice_desk/models.py
Follow imports/calls from the relevant module and confirm behavior in tests. Record input → mechanism → observable output/state → failure evidence.
Code Walkthrough & Mechanics
Read security/authz.py, rag/tenant_rag.py, security/context.py, tools/ticket_tools.py, privacy/redactor.py, agent/mock_model.py, agent/secure_agent.py, privacy/telemetry_scrubber.py, models.py with these theory sections beside you:
- Why identity must enter the agent architecture — locate its implementation and evidence.
- Authentication versus authorization — locate its implementation and evidence.
- Carry trusted identity separately from prompt text — locate its implementation and evidence.
- Multi-tenant isolation — locate its implementation and evidence.
- PII minimization — locate its implementation and evidence.
For each concept identify the real function/class/protocol boundary, its input/state, its observable result, and the assertion in tests/test_identity_pii.py that proves the behavior. If a concept has no implementation or evidence, record that as a gap rather than inventing one.
Common Mistakes & Debugging Guidance
Use the theory—not generic timeout or .env advice—to decide what can fail today.
Failure lens: revisit Multi-tenant isolation, PII minimization.
- Reproduce the smallest case that violates one of those expectations.
- Trace the real Day 31 modules until you find the first incorrect state/output/decision.
- Use
tests/test_identity_pii.pyas executable evidence. - Add a regression test if the failure is not already represented.
- Fix the smallest responsible boundary and rerun the relevant test before the full suite.
Your debugging explanation must name the topic-specific invariant that failed, not merely say “an exception occurred.”
Practical Lab Instructions
- Summarize these theory ideas before opening the implementation:
- Why identity must enter the agent architecture
- Authentication versus authorization
- Carry trusted identity separately from prompt text
- Multi-tenant isolation
- Inspect the most relevant real Day 31 modules first:
service_desk/security/authz.pyservice_desk/rag/tenant_rag.pyservice_desk/security/context.pyservice_desk/tools/ticket_tools.pyservice_desk/privacy/redactor.pyservice_desk/agent/mock_model.pyservice_desk/agent/secure_agent.pyservice_desk/privacy/telemetry_scrubber.pyservice_desk/models.py
- Inspect the automated evidence:
tests/test_identity_pii.py
- Establish the baseline:
cd service-desk-day-31 PYTHONPATH=. pytest tests/test_identity_pii.py -q - Trace one theory concept through the actual nested modules and tests.
- Run one success case and record input → mechanism → observable result.
- Exercise one topic-specific failure/boundary case and name the invariant that protects the system.
- Re-run the relevant tests and explain theory → implementation → evidence.
Done when: another student can reproduce your trace without relying on an invented sample.
Key Takeaways
- Be able to explain Why identity must enter the agent architecture and point to its implementation/evidence in Day 31.
- Be able to explain Authentication versus authorization and point to its implementation/evidence in Day 31.
- Be able to explain Carry trusted identity separately from prompt text and point to its implementation/evidence in Day 31.
Knowledge Check & Scenario Questions
- Concept: Using Why identity must enter the agent architecture, explain the engineering problem Day 31 is solving without naming a framework as the answer.
- Mechanism: How does Authentication versus authorization appear in the real project? Start from
service_desk/security/authz.pyand name the observable state/output/event that changes. - Failure: For Carry trusted identity separately from prompt text, describe one incorrect implementation or boundary condition and the evidence you would expect in
tests/test_identity_pii.py. - Design review: Which assumption in today's design would you verify before reusing this implementation in a different production system?
Official References
- OAuth 2.0 Scopes & Multi-Tenancy Security: https://oauth.net/2/grant-types/
- NIST PII Protection Standards: https://csrc.nist.gov/publications/detail/sp/800-122/final
03 · EXPLAIN
Interview checkpoint
Explain one design decision from this lesson, the alternative you rejected, and the failure mode or evidence that justified your choice.