vocero-s2s/docs/RAG_SCOPING_EN.md

Multi-Tenant RAG / User & Role Scoping

Technical Documentation — Vocero S2S v0.2.11+

Date: 01/09/2026 Scope: RAG Server-Side module (speech_to_speech.RAG) + Realtime WebSocket

---

1. Goal

This change introduces a scope isolation mechanism into the Speech-to-Speech pipeline RAG, enabling different clients (logged-in users, roles) to receive only the Knowledge Base chunks they are authorised to access, without sharing private information across different sessions.

Before this release the RAG was single and global: every WS session could see the entire shared KB. Now the retrieval can be filtered by:

• user_id (single logical user)
• roles (role list: agent, supervisor, backoffice…)
• allowed_source_prefixes (custom prefixes on chunk source fields)
• Arbitrary metadata (metadata_filter with exact match + *_in suffix for OR-list matches)

Filters are applied BEFORE the cosine-similarity ranking (via NumPy boolean mask), so there is no accuracy loss or measurable performance penalty.

---

2. Comparison: old system vs new system

Criterion	Before (≤ v0.2.11 vanilla)	After (v0.2.11 + RAG scoping)
Retrieval scope	_GLOBAL_RAG singleton → everybody sees everything	Still a singleton, but search() applies a per-session scope mask
Data isolation	❌ None: different clients share private chunks (PII)	✅ Each WS session declares user_id/roles/prefixes; only authorised chunks become ranking candidates
Memory usage	1 index, 1 embedding model	1 index, 1 embedding model (no increase — filter is O(n) with a numpy mask)
KB population	source is free-form, metadata is optional	source and metadata must follow a convention for filters (see §4)
How to declare scope	❌ Did not exist	✅ WebSocket query-string at connection time
Test filters via HTTP	❌ /v1/rag/search only accepted query/top_k/threshold	✅ /v1/rag/search now also accepts allowed_source_prefixes and metadata_filter
Hot-patch scope mid-session	—	⚠️ Possible only by mutating runtime_config directly (not exposed via session.update — see §9.1)
LLM hook	_inject_rag_context called search(query, top_k, threshold)	Same hook, but additionally passes allowed_prefixes and meta_filter derived from runtime_config
Retrieval logging	Log contained count + sources_summary	Log now also includes scope_prefixes and scope_meta for audit & debugging

---

3. Codebase changes

File	Layer	Description
src/speech_to_speech/RAG/retriever.py	Search core	Extended search(...) signature with allowed_source_prefixes and metadata_filter. Applies a NumPy boolean mask before ranking. Supports *_in OR-list semantics on metadata.
src/speech_to_speech/RAG/router.py	HTTP API	SearchRequest schema + POST /v1/rag/search endpoint now forward the new filters.
src/speech_to_speech/api/openai_realtime/runtime_config.py	Session state	Added 3 fields on RuntimeConfig: rag_user_id, rag_roles (CSV coerced → list), rag_allowed_source_prefixes (CSV coerced → list).
src/speech_to_speech/api/openai_realtime/websocket_router.py	WS claim	Inside realtime_endpoint, right after register(), scope parameters are read from the WS query-string and written into the session's ConnState.runtime_config.
src/speech_to_speech/LLM/base_openai_compatible_language_model.py	LLM hook	_inject_rag_context(...) now receives runtime_config, builds allowed_prefixes (shared + roles + user + custom) and metadata_filter (user_id, roles_in), then forwards them to search(). The inject log now also reports the actual scope.

No files were removed; backward compatibility is preserved (all new parameters default to None, so clients that send no scope continue to see the entire KB exactly like before — "legacy" behaviour).

---

4. Conventions: how to tag chunks per user/role

4.1 source field convention (MANDATORY if you use prefix-based scoping)

The leading part of source controls visibility through the allowed_source_prefixes filter. Use these standard prefixes:

users/<id>/<rest_of_path>              # ← only user <id> sees these chunks
roles/<role_name>/<rest_of_path>       # ← only users with <role_name> see them
shared/<rest_of_path>                  # ← everyone sees them (same as public/)
public/<rest_of_path>                  # ← everyone sees them
<anything_else>/<rest_of_path>         # ← visible ONLY if explicitly passed into
                                       #   rag_allowed_source_prefixes at WS connect

Valid examples:

users/123/user_preferences
users/123/anamnesis_2026
roles/agent/procedure/open_ticket
roles/supervisor/reports/monthly_kpis
shared/general_intro
public/helpdesk_hours
kb_customer_alpha/contracts/clause_23   # ← needs prefixes=kb_customer_alpha/ in WS

4.2 metadata convention (recommended — used by metadata_filter)

Key	Type	Semantics
user_id	str/int	Owner user identifier. The WS auto-filter uses EXACT match.
roles	list[str]	Allowed roles. The WS auto-filter uses roles_in: [...] (OR-match: any match passes).
scopes	list[str]	Optional (not used by the automatic filter — for custom business logic).

Valid JSON examples:

{"user_id": "123", "note": "sensitive data"}
{"roles": ["agent", "supervisor"], "version": 3}

4.3 The two filters don't exclude — they AND together

allowed_source_prefixes and metadata_filter are combined with logical AND: a chunk passes only if both coincide.

💡 Practical tip: stick to only one mechanism to keep things simple. The default WS plumbing uses both for defence in depth — if you mess up the prefix, the metadata filter still prevents leakage, and vice versa.

---

5. How to use it

5.1 Upload scoped chunks

Use /v1/rag/add/document (auto-chunked) or /v1/rag/add/chunks (pre-chunked):

# ---------- Single user 123 ----------
curl -X POST http://127.0.0.1:12345/v1/rag/add/document \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "User 123 preferences: coffee without sugar, nut allergy.",
    "source": "users/123/user_preferences",
    "metadata": {"user_id": "123"}
  }'

# ---------- Visible to roles agent + supervisor ----------
curl -X POST http://127.0.0.1:12345/v1/rag/add/document \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Procedure: to open a ticket, email support@example.com with the customer ID.",
    "source": "roles/agent/procedure/open_ticket",
    "metadata": {"roles": ["agent", "supervisor"]}
  }'

# ---------- Visible to EVERYBODY ----------
curl -X POST http://127.0.0.1:12345/v1/rag/add/document \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Helpdesk hours: Mon-Fri 9-18.",
    "source": "shared/helpdesk_hours",
    "metadata": {"scopes": ["public"]}
  }'

# ---------- Custom scope (e.g. a whole customer tenant) ----------
curl -X POST http://127.0.0.1:12345/v1/rag/add/document \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Alpha contract clause 23: 5% penalty for delay > 10 days.",
    "source": "kb_customer_alpha/contracts/clause_23",
    "metadata": {"customer": "alpha"}
  }'

5.2 Connect the WebSocket declaring the scope

Scope is populated at WS connection time via query-string (simpler and more standard than stuffing custom fields inside session.update).

WS param	Aliases	Example	Explanation
user_id	rag_user_id	user_id=123	User id. Produces prefix users/123/ + filter metadata.user_id=="123".
roles	rag_roles	roles=agent,supervisor	CSV role list. Produces prefixes roles/agent/, roles/supervisor/ + filter metadata.roles ∩ [agent,supervisor] ≠ ∅.
prefixes	allowed_source_prefixes / rag_allowed_source_prefixes	prefixes=shared/,kb_customer_alpha/	CSV list of EXTRA custom prefixes to add on top of user/role/shared.

Example WS URL — User 123, role agent, also needs kb_customer_alpha/

ws://host:12345/v1/realtime?user_id=123&roles=agent&prefixes=kb_customer_alpha/

What the system AUTOMATICALLY includes from this URL:

allowed_source_prefixes = [
  "shared/",                      # always (builtin public)
  "public/",                      # always (builtin public)
  "roles/agent/",                 # from roles=agent
  "users/123/",                   # from user_id=123
  "kb_customer_alpha/"            # from prefixes=
]

metadata_filter = {
  "roles_in": ["agent"],         # from roles=agent
  "user_id":  "123"              # from user_id=123
}

Both filters are passed to search(). Result: retrieval can only pick from authorised chunks.

5.3 Test via HTTP /v1/rag/search

To debug without opening a WS, hit the HTTP endpoint by explicitly sending the filters:

# Search as if we were "user 123 with role agent"
curl -sS -X POST http://127.0.0.1:12345/v1/rag/search \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "how do I open a ticket",
    "top_k": 5,
    "threshold": 0.05,
    "allowed_source_prefixes": ["shared/", "users/123/", "roles/agent/"],
    "metadata_filter": {
      "user_id": "123",
      "roles_in": ["agent", "supervisor"]
    }
  }' | python3 -m json.tool

Response (structure is unchanged — only the candidate chunk pool changes):

{
  "query": "how do I open a ticket",
  "count": 1,
  "results": [
    {
      "text": "Procedure: to open a ticket, email support@example.com...",
      "source": "roles/agent/procedure/open_ticket",
      "chunk_index": 0,
      "score": 0.71,
      "metadata": {"roles": ["agent", "supervisor"]}
    }
  ]
}

5.4 Cleanup

Delete every chunk for one user:

curl -X POST http://127.0.0.1:12345/v1/rag/remove \
  -H 'Content-Type: application/json' \
  -d '{"source_prefix": "users/123/", "exact": false, "persist": true}'

Delete every chunk for one role:

curl -X POST http://127.0.0.1:12345/v1/rag/remove \
  -H 'Content-Type: application/json' \
  -d '{"source_prefix": "roles/backoffice/", "exact": false, "persist": true}'

Reload the index after editing files inside kb/:

curl -X POST http://127.0.0.1:12345/v1/rag/reload \
  -H 'Content-Type: application/json' \
  -d '{"force_rebuild": true}'

---

6. Logging & Debugging

What to look for in the PM2 pipeline logs:

Event	Level	You'll see
WS connection with scope	INFO	RAG scope loaded from WS — user_id=X roles=Y prefixes=Z
Scope parsing failed	WARNING	error applying RAG scope from query-string: ...
No chunk matched the scope	DEBUG	no relevant chunks for turn=... scope_prefixes=... scope_meta=...
Retrieval OK (prompt injection)	INFO	injected N chunks for turn=... (scope_prefixes=..., scope_meta=...)

Handy one-liner to watch active scopes live:

pm2 logs start_pipeline_rag --lines 200 | grep -i "RAG scope"

---

7. Full end-to-end example

Scenario: An external CRM opens 2 WS sessions: one for user 123 (private email stored in the RAG) and one for user 456.

# 1) Seed the KB
curl -X POST http://127.0.0.1:12345/v1/rag/upsert/document -H 'Content-Type: application/json' \
  -d '{"text":"Account 123: mario.rossi@secretmail.example","source":"users/123/contacts","metadata":{"user_id":"123"}}'

curl -X POST http://127.0.0.1:12345/v1/rag/upsert/document -H 'Content-Type: application/json' \
  -d '{"text":"Account 456: lucia.bianchi@other.example","source":"users/456/contacts","metadata":{"user_id":"456"}}'

curl -X POST http://127.0.0.1:12345/v1/rag/upsert/document -H 'Content-Type: application/json' \
  -d '{"text":"Secretary office hours 9-18","source":"shared/hours"}'

# 2) User 123 connects
ws://127.0.0.1:12345/v1/realtime?user_id=123

User 123 asks by voice: "What is my email address?" → the RAG returns theirs, NOT user 456's (filtered out).

# 3) User 456 connects
ws://127.0.0.1:12345/v1/realtime?user_id=456

Same question → the RAG returns user 456's email, never sees user 123's.

Both users see shared/hours because it is force-included by default.

---

8. Backward compatibility

• With no WS params: continues to behave exactly like before — no filter is applied, the entire KB is visible. Good for legacy scenarios.
• Existing /v1/rag/* APIs: signatures unchanged; new fields are optional (Optional[...]) so pre-existing scripts do not break.
• On-disk persisted files: no changes to _dynamic.jsonl, _chunks.jsonl, _index.npz — format is byte-identical.

---

9. Current limits & possible evolutions

9.1 Scope cannot be changed mid-session

Today scope can only be fixed at ws.accept() via the query-string. Once the WS is connected you cannot update user_id/roles through session.update — you'd have to disconnect and reconnect.

Why: session.update is typed as RealtimeSessionCreateRequest from the openai-python package, and we don't want to break SDK compatibility by attaching unofficial custom fields to it.

How to unlock it (future option): add a custom event session.scope_update (non-standard) inside RealtimeService.parse_client_event that mutates ConnState.runtime_config.rag_*.

9.2 Soft (logical) isolation, not hard

This is "Solution A": one shared index with a Python/NumPy vectorised filter. Sufficient for 90% of scenarios. It does not protect you from bugs that skip the filter (e.g. a future patch that forgets to thread it through).

If you need process-level / index-level isolation (healthcare or finance data), you need "Solution B": a registry of multiple RAGRetriever instances, one per tenant, with separate NPZ indexes.

9.3 Multiple roles, but no permission hierarchy

Today the roles_in filter is a flat OR. There is no concept of hierarchy (e.g. supervisor inherits every agent permission).

Workaround: when writing metadata.roles list every role that should be allowed to read that chunk, not only the "base" one. E.g.:

{"roles": ["agent", "supervisor", "admin"]}   # instead of just ["agent"]

9.4 Embeddings cache

The mask is recomputed on every search(). It is overwhelmingly fast up to n < 500.000 chunks; beyond that you could pre-compute one bitmask per discrete scope type and reuse it across sessions of the same kind.

---

10. Code references

• Core filter: RAGRetriever.search()
• HTTP request schema: SearchRequest
• RuntimeConfig scope fields: RuntimeConfig.rag_*
• WS query-string handler: realtime_endpoint scope parsing
• LLM hook with injection: _inject_rag_context