vocero-s2s/docs/RAG_SCOPING_EN.md

# Multi-Tenant RAG / User & Role Scoping
## Technical Documentation — Vocero S2S v0.2.11+

> Date: 01/09/2026
> Scope: RAG Server-Side module (`speech_to_speech.RAG`) + Realtime WebSocket

---

## 1. Goal

This change introduces a **scope isolation mechanism** into the Speech-to-Speech
pipeline RAG, enabling different clients (logged-in users, roles) to receive
**only the Knowledge Base chunks they are authorised to access**, without
sharing private information across different sessions.

Before this release the RAG was **single and global**: every WS session could
see the entire shared KB. Now the retrieval can be filtered by:
- **`user_id`** (single logical user)
- **`roles`** (role list: agent, supervisor, backoffice…)
- **`allowed_source_prefixes`** (custom prefixes on chunk `source` fields)
- **Arbitrary metadata** (`metadata_filter` with exact match + `*_in` suffix
  for OR-list matches)

Filters are applied **BEFORE** the cosine-similarity ranking (via NumPy boolean
mask), so there is no accuracy loss or measurable performance penalty.

---

## 2. Comparison: old system vs new system

| Criterion | Before (≤ v0.2.11 vanilla) | After (v0.2.11 + RAG scoping) |
|---|---|---|
| **Retrieval scope** | `_GLOBAL_RAG` singleton → everybody sees everything | Still a singleton, but `search()` applies a per-session scope mask |
| **Data isolation** | ❌ None: different clients share private chunks (PII) | ✅ Each WS session declares `user_id`/`roles`/`prefixes`; only authorised chunks become ranking candidates |
| **Memory usage** | 1 index, 1 embedding model | 1 index, 1 embedding model (no increase — filter is O(n) with a numpy mask) |
| **KB population** | `source` is free-form, `metadata` is optional | `source` and `metadata` must follow a convention for filters (see §4) |
| **How to declare scope** | ❌ Did not exist | ✅ **WebSocket query-string** at connection time |
| **Test filters via HTTP** | ❌ `/v1/rag/search` only accepted query/top_k/threshold | ✅ `/v1/rag/search` now also accepts `allowed_source_prefixes` and `metadata_filter` |
| **Hot-patch scope mid-session** | — | ⚠️ Possible only by mutating `runtime_config` directly (not exposed via `session.update` — see §9.1) |
| **LLM hook** | `_inject_rag_context` called `search(query, top_k, threshold)` | Same hook, but additionally passes `allowed_prefixes` and `meta_filter` derived from `runtime_config` |
| **Retrieval logging** | Log contained `count` + `sources_summary` | Log now also includes `scope_prefixes` and `scope_meta` for audit & debugging |

---

## 3. Codebase changes

| File | Layer | Description |
|---|---|---|
| `src/speech_to_speech/RAG/retriever.py` | Search core | Extended `search(...)` signature with `allowed_source_prefixes` and `metadata_filter`. Applies a NumPy boolean mask before ranking. Supports `*_in` OR-list semantics on metadata. |
| `src/speech_to_speech/RAG/router.py` | HTTP API | `SearchRequest` schema + `POST /v1/rag/search` endpoint now forward the new filters. |
| `src/speech_to_speech/api/openai_realtime/runtime_config.py` | Session state | Added 3 fields on `RuntimeConfig`: `rag_user_id`, `rag_roles` (CSV coerced → list), `rag_allowed_source_prefixes` (CSV coerced → list). |
| `src/speech_to_speech/api/openai_realtime/websocket_router.py` | WS claim | Inside `realtime_endpoint`, right after `register()`, scope parameters are read from the WS query-string and written into the session's `ConnState.runtime_config`. |
| `src/speech_to_speech/LLM/base_openai_compatible_language_model.py` | LLM hook | `_inject_rag_context(...)` now receives `runtime_config`, builds `allowed_prefixes` (shared + roles + user + custom) and `metadata_filter` (`user_id`, `roles_in`), then forwards them to `search()`. The inject log now also reports the actual scope. |

No files were removed; backward compatibility is preserved (all new parameters
default to `None`, so clients that send no scope continue to see the **entire**
KB exactly like before — "legacy" behaviour).

---

## 4. Conventions: how to tag chunks per user/role

### 4.1 `source` field convention (MANDATORY if you use prefix-based scoping)

The leading part of `source` controls visibility through the
`allowed_source_prefixes` filter. Use these standard prefixes:

```
users/<id>/<rest_of_path>              # ← only user <id> sees these chunks
roles/<role_name>/<rest_of_path>       # ← only users with <role_name> see them
shared/<rest_of_path>                  # ← everyone sees them (same as public/)
public/<rest_of_path>                  # ← everyone sees them
<anything_else>/<rest_of_path>         # ← visible ONLY if explicitly passed into
                                       #   rag_allowed_source_prefixes at WS connect
```

**Valid examples:**
```
users/123/user_preferences
users/123/anamnesis_2026
roles/agent/procedure/open_ticket
roles/supervisor/reports/monthly_kpis
shared/general_intro
public/helpdesk_hours
kb_customer_alpha/contracts/clause_23   # ← needs prefixes=kb_customer_alpha/ in WS
```

### 4.2 `metadata` convention (recommended — used by `metadata_filter`)

| Key | Type | Semantics |
|---|---|---|
| `user_id` | `str`/`int` | Owner user identifier. The WS auto-filter uses EXACT match. |
| `roles` | `list[str]` | Allowed roles. The WS auto-filter uses `roles_in: [...]` (OR-match: any match passes). |
| `scopes` | `list[str]` | Optional (not used by the automatic filter — for custom business logic). |

**Valid JSON examples:**
```json
{"user_id": "123", "note": "sensitive data"}
{"roles": ["agent", "supervisor"], "version": 3}
```

### 4.3 The two filters don't exclude — they AND together

`allowed_source_prefixes` and `metadata_filter` are combined with logical AND:
a chunk passes only if **both** coincide.

> 💡 Practical tip: stick to **only one mechanism** to keep things simple.
> The default WS plumbing uses **both** for defence in depth — if you mess up
> the prefix, the metadata filter still prevents leakage, and vice versa.

---

## 5. How to use it

### 5.1 Upload scoped chunks

Use `/v1/rag/add/document` (auto-chunked) or `/v1/rag/add/chunks`
(pre-chunked):

```bash
# ---------- Single user 123 ----------
curl -X POST http://127.0.0.1:12345/v1/rag/add/document \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "User 123 preferences: coffee without sugar, nut allergy.",
    "source": "users/123/user_preferences",
    "metadata": {"user_id": "123"}
  }'

# ---------- Visible to roles agent + supervisor ----------
curl -X POST http://127.0.0.1:12345/v1/rag/add/document \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Procedure: to open a ticket, email support@example.com with the customer ID.",
    "source": "roles/agent/procedure/open_ticket",
    "metadata": {"roles": ["agent", "supervisor"]}
  }'

# ---------- Visible to EVERYBODY ----------
curl -X POST http://127.0.0.1:12345/v1/rag/add/document \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Helpdesk hours: Mon-Fri 9-18.",
    "source": "shared/helpdesk_hours",
    "metadata": {"scopes": ["public"]}
  }'

# ---------- Custom scope (e.g. a whole customer tenant) ----------
curl -X POST http://127.0.0.1:12345/v1/rag/add/document \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Alpha contract clause 23: 5% penalty for delay > 10 days.",
    "source": "kb_customer_alpha/contracts/clause_23",
    "metadata": {"customer": "alpha"}
  }'
```

### 5.2 Connect the WebSocket declaring the scope

Scope is populated **at WS connection time** via **query-string** (simpler and
more standard than stuffing custom fields inside `session.update`).

| WS param | Aliases | Example | Explanation |
|---|---|---|---|
| `user_id` | `rag_user_id` | `user_id=123` | User id. Produces prefix `users/123/` + filter `metadata.user_id=="123"`. |
| `roles` | `rag_roles` | `roles=agent,supervisor` | CSV role list. Produces prefixes `roles/agent/`, `roles/supervisor/` + filter `metadata.roles ∩ [agent,supervisor] ≠ ∅`. |
| `prefixes` | `allowed_source_prefixes` / `rag_allowed_source_prefixes` | `prefixes=shared/,kb_customer_alpha/` | CSV list of EXTRA custom prefixes to add on top of user/role/shared. |

**Example WS URL — User 123, role agent, also needs kb_customer_alpha/**

```
ws://host:12345/v1/realtime?user_id=123&roles=agent&prefixes=kb_customer_alpha/
```

What the system AUTOMATICALLY includes from this URL:

```
allowed_source_prefixes = [
  "shared/",                      # always (builtin public)
  "public/",                      # always (builtin public)
  "roles/agent/",                 # from roles=agent
  "users/123/",                   # from user_id=123
  "kb_customer_alpha/"            # from prefixes=
]

metadata_filter = {
  "roles_in": ["agent"],         # from roles=agent
  "user_id":  "123"              # from user_id=123
}
```

Both filters are passed to `search()`. Result: retrieval can only pick from
authorised chunks.

### 5.3 Test via HTTP /v1/rag/search

To debug without opening a WS, hit the HTTP endpoint by explicitly sending the
filters:

```bash
# Search as if we were "user 123 with role agent"
curl -sS -X POST http://127.0.0.1:12345/v1/rag/search \
  -H 'Content-Type: application/json' \
  -d '{
    "query": "how do I open a ticket",
    "top_k": 5,
    "threshold": 0.05,
    "allowed_source_prefixes": ["shared/", "users/123/", "roles/agent/"],
    "metadata_filter": {
      "user_id": "123",
      "roles_in": ["agent", "supervisor"]
    }
  }' | python3 -m json.tool
```

Response (structure is unchanged — only the candidate chunk pool changes):

```json
{
  "query": "how do I open a ticket",
  "count": 1,
  "results": [
    {
      "text": "Procedure: to open a ticket, email support@example.com...",
      "source": "roles/agent/procedure/open_ticket",
      "chunk_index": 0,
      "score": 0.71,
      "metadata": {"roles": ["agent", "supervisor"]}
    }
  ]
}
```

### 5.4 Cleanup

Delete every chunk for one user:
```bash
curl -X POST http://127.0.0.1:12345/v1/rag/remove \
  -H 'Content-Type: application/json' \
  -d '{"source_prefix": "users/123/", "exact": false, "persist": true}'
```

Delete every chunk for one role:
```bash
curl -X POST http://127.0.0.1:12345/v1/rag/remove \
  -H 'Content-Type: application/json' \
  -d '{"source_prefix": "roles/backoffice/", "exact": false, "persist": true}'
```

Reload the index after editing files inside `kb/`:
```bash
curl -X POST http://127.0.0.1:12345/v1/rag/reload \
  -H 'Content-Type: application/json' \
  -d '{"force_rebuild": true}'
```

---

## 6. Logging & Debugging

What to look for in the PM2 pipeline logs:

| Event | Level | You'll see |
|---|---|---|
| WS connection with scope | INFO | `RAG scope loaded from WS — user_id=X roles=Y prefixes=Z` |
| Scope parsing failed | WARNING | `error applying RAG scope from query-string: ...` |
| No chunk matched the scope | DEBUG | `no relevant chunks for turn=... scope_prefixes=... scope_meta=...` |
| Retrieval OK (prompt injection) | INFO | `injected N chunks for turn=... (scope_prefixes=..., scope_meta=...)` |

Handy one-liner to watch active scopes live:
```bash
pm2 logs start_pipeline_rag --lines 200 | grep -i "RAG scope"
```

---

## 7. Full end-to-end example

**Scenario:** An external CRM opens 2 WS sessions: one for user 123 (private
email stored in the RAG) and one for user 456.

```bash
# 1) Seed the KB
curl -X POST http://127.0.0.1:12345/v1/rag/upsert/document -H 'Content-Type: application/json' \
  -d '{"text":"Account 123: mario.rossi@secretmail.example","source":"users/123/contacts","metadata":{"user_id":"123"}}'

curl -X POST http://127.0.0.1:12345/v1/rag/upsert/document -H 'Content-Type: application/json' \
  -d '{"text":"Account 456: lucia.bianchi@other.example","source":"users/456/contacts","metadata":{"user_id":"456"}}'

curl -X POST http://127.0.0.1:12345/v1/rag/upsert/document -H 'Content-Type: application/json' \
  -d '{"text":"Secretary office hours 9-18","source":"shared/hours"}'
```

```
# 2) User 123 connects
ws://127.0.0.1:12345/v1/realtime?user_id=123
```

User 123 asks by voice: *"What is my email address?"* → the RAG returns theirs,
**NOT** user 456's (filtered out).

```
# 3) User 456 connects
ws://127.0.0.1:12345/v1/realtime?user_id=456
```

Same question → the RAG returns user 456's email, never sees user 123's.

Both users see `shared/hours` because it is force-included by default.

---

## 8. Backward compatibility

- **With no WS params**: continues to behave exactly like before — no filter is
  applied, the entire KB is visible. Good for legacy scenarios.
- **Existing `/v1/rag/*` APIs**: signatures unchanged; new fields are
  optional (`Optional[...]`) so pre-existing scripts **do not break**.
- **On-disk persisted files**: no changes to `_dynamic.jsonl`,
  `_chunks.jsonl`, `_index.npz` — format is byte-identical.

---

## 9. Current limits & possible evolutions

### 9.1 Scope cannot be changed mid-session

Today scope can only be fixed at `ws.accept()` via the query-string. Once the
WS is connected you cannot update `user_id`/`roles` through `session.update`
— you'd have to disconnect and reconnect.

**Why:** `session.update` is typed as `RealtimeSessionCreateRequest` from the
`openai-python` package, and we don't want to break SDK compatibility by
attaching unofficial custom fields to it.

**How to unlock it (future option):** add a custom event
`session.scope_update` (non-standard) inside
`RealtimeService.parse_client_event` that mutates
`ConnState.runtime_config.rag_*`.

### 9.2 Soft (logical) isolation, not hard

This is "Solution A": one shared index with a Python/NumPy vectorised filter.
Sufficient for 90% of scenarios. It **does not** protect you from bugs that
skip the filter (e.g. a future patch that forgets to thread it through).

If you need process-level / index-level isolation (healthcare or finance data),
you need "Solution B": a registry of multiple `RAGRetriever` instances, one
per tenant, with separate NPZ indexes.

### 9.3 Multiple roles, but no permission hierarchy

Today the `roles_in` filter is a flat OR. There is no concept of hierarchy
(e.g. `supervisor` inherits every `agent` permission).

**Workaround:** when writing `metadata.roles` list every role that should be
allowed to read that chunk, not only the "base" one. E.g.:

```json
{"roles": ["agent", "supervisor", "admin"]}   # instead of just ["agent"]
```

### 9.4 Embeddings cache

The mask is recomputed on every `search()`. It is overwhelmingly fast up to
`n < 500.000 chunks`; beyond that you could pre-compute one bitmask per
discrete scope type and reuse it across sessions of the same kind.

---

## 10. Code references

- Core filter: [RAGRetriever.search()](file:///home/azurian/speech-to-speech/src/speech_to_speech/RAG/retriever.py#L380-L489)
- HTTP request schema: [SearchRequest](file:///home/azurian/speech-to-speech/src/speech_to_speech/RAG/router.py#L71-L82)
- RuntimeConfig scope fields: [RuntimeConfig.rag_*](file:///home/azurian/speech-to-speech/src/speech_to_speech/api/openai_realtime/runtime_config.py#L40-L78)
- WS query-string handler: [realtime_endpoint scope parsing](file:///home/azurian/speech-to-speech/src/speech_to_speech/api/openai_realtime/websocket_router.py#L626-L667)
- LLM hook with injection: [_inject_rag_context](file:///home/azurian/speech-to-speech/src/speech_to_speech/LLM/base_openai_compatible_language_model.py#L292-L389)