Design an internal assistant that answers questions from company documents for 20,000 employees, where every answer must respect the document permissions people already have.
#About 6 million documents across a wiki, a file share and a ticket system, roughly 200,000 changed per day. Peak 50 questions per second. Target: first token within 3 seconds at p95. Some spaces (HR, legal, M&A) are tightly restricted, and leadership asks what happens when someone loses access.
Show answer
Requirements to pin down first: which sources and how fresh (minutes or a day), the revocation target (for example, access removed in the source is enforced within 15 minutes), answer quality per department, and whether restricted spaces are in scope at launch.
Architecture: - Connectors per source pull content and ACLs together: the users and groups who can read each document. ACL-only changes are their own event, because a permission change does not change the text. - Ingestion parses, chunks and embeds. Each chunk stores the document ID, version and the list of principal IDs allowed to read it. - At query time the service resolves the user's identity and group memberships from the identity provider (cached for a few minutes), and passes them as a filter inside the vector and keyword queries. Filtering inside the query keeps recall; filtering the top 10 afterwards can leave nothing for a user with narrow access. - For restricted spaces, re-check the final cited documents against the source system before showing them. That closes the gap between an ACL change and the next index update. - The model sees only chunks the user may read, plus instructions to cite them. Answers and caches are keyed by user or by permission set, never shared across users.
Scale arithmetic: if documents average 10 chunks, that is 60 million vectors; at 1,024 float32 dimensions (4 KB each) about 246 GB of raw vectors before index overhead, so plan for sharding or quantization.
Failure modes and mitigations: - Group sync lag: someone removed from a group still retrieves its documents. Track sync lag as a metric with an alert. - Deleted documents still answer. Treat deletes as events with their own deadline, including caches and evaluation sets. - Injected text inside a shared document. Treat retrieved text as data, and keep tools out of this assistant at launch.
Observability: one trace per question with retrieved chunk IDs, the number of principals in the filter, stage latencies and the cited sources; metrics for connector freshness, ACL sync lag, empty-retrieval rate and citation coverage.
Security testing: plant canary documents restricted to a test group and, nightly, query as users outside it. Any hit is a release-blocking incident. OWASP lists permission-aware vector stores and strict partitioning as the first mitigation for vector and embedding weaknesses.
Rollout: one department with a labelled question set, then wider, adding restricted spaces last.
Trade-offs
Filtering inside the query with synced ACLs is fast but can lag behind the source; checking every result against the source is exact but adds latency and load, so reserve it for restricted spaces and final citations.
Common mistake
Indexing everything with a service account and asking the model not to reveal documents the user cannot see.
Follow-up
A user is added to a group and still cannot find a document for an hour. Where would you look first?