Day 36: Hybrid Search Production

user query
  -> embedding query
  -> vector search top_k
  -> build context
  -> LLM answer

"HTTP 429"
"SLA P1 enterprise"
"VAT invoice"
"CVE-2024-..."
"điều 12.3"
"BAAI/bge-m3 normalize_embeddings"

query
  -> BM25 / sparse search
  -> dense vector search
  -> merge candidates
  -> optional reranker
  -> context builder

query text -> query embedding
chunk text -> chunk embedding
similarity(query embedding, chunk embedding) -> top_k chunks

query: "HTTP 429"
doc: "Vượt giới hạn API trả về HTTP 429."

query: "lấy lại tiền"
doc: "Khách hàng có thể yêu cầu hoàn tiền."

score cao nếu query term xuất hiện trong document,
term hiếm có trọng số cao hơn term phổ biến,
document quá dài bị normalize,
term lặp nhiều có lợi nhưng bị saturation.

score(D, Q) = sum(IDF(q_i) * ((tf(q_i, D) * (k1 + 1)) / (tf(q_i, D) + k1 * (1 - b + b * |D| / avgdl))))

query: "lấy lại tiền"
neural sparse expansion có thể tăng weight cho token liên quan tới "refund", "hoàn tiền"

                         +-> BM25 top 50 -----+
query -> normalize/filter|                    +-> merge/dedupe -> rerank -> context
                         +-> dense top 50 ----+

1. nhận query + auth context
2. normalize query an toàn
3. build mandatory filters: tenant, ACL, index_version, deleted=false
4. chạy BM25 top_n và dense top_n song song
5. merge bằng RRF, không merge raw score trực tiếp
6. deduplicate theo chunk_id hoặc canonical_chunk_id
7. optional rerank top 20-100
8. build context có citation, source, page, section
9. log ranks/scores/latency/debug info đã redact PII
10. evaluate offline và monitor online

rrf_score(doc) = sum(1 / (k + rank_i(doc)))

"C++"       -> "c"       # sai
"S3"        -> "s"       # sai
"HTTP 429"  -> ""        # sai
"P1/P2"     -> "p p"     # sai
"node.js"   -> "node js" # có thể sai nếu corpus giữ "node.js"

if query contains many codes/acronyms/numbers:
    increase BM25 top_k or weight
elif query is long natural language:
    increase dense top_k
else:
    use balanced hybrid

max_chunks_per_document = 2
prefer_latest_document_version = true
require_citation_fields = true
drop_chunks_below_min_score_after_rerank = true

semantic
keyword
mixed
acronym
exact_code
no_diacritic
english_mix
short_query
long_question
negative_or_ambiguous

| Config | Hit@5 | Recall@10 | MRR@10 | p95 ms | Ghi chú |
|---|---:|---:|---:|---:|---|
| BM25-only | | | | | |
| Dense-only | | | | | |
| Hybrid RRF | | | | | |
| Hybrid RRF + reranker | | | | | |

Dense-only average tốt nhưng exact_code rất tệ.
BM25-only keyword tốt nhưng semantic rất tệ.
Hybrid average tốt và ít category bị rơi mạnh hơn.

latency_total = normalize + bm25 + dense + merge + rerank + context

latency_total ~= normalize + max(bm25, dense) + merge + rerank + context

tenant_id = current_user.tenant_id
AND index_version = active_index_version
AND deleted_at IS NULL
AND acl_roles intersects current_user.roles

tenant_id
user_permission_hash
query_normalized
index_version
retrieval_config_version

{
  "query_hash": "sha256:...",
  "tenant_id": "company_a",
  "bm25_top_k": 50,
  "dense_top_k": 50,
  "rrf_k": 60,
  "bm25_latency_ms": 42,
  "dense_latency_ms": 65,
  "merge_latency_ms": 2,
  "final_chunk_ids": ["..."],
  "index_version": "rag-index-2026-05-10"
}

query + auth context
  -> safe normalization
  -> mandatory filters
  -> BM25 top_n    \
                   -> RRF -> dedupe -> optional rerank -> context
  -> dense top_n   /

retrieval:
  active_index_version: "rag-index-2026-05-10-bge-m3-v2"
  bm25:
    enabled: true
    top_k: 50
    analyzer: "vi_en_code_safe_v1"
  dense:
    enabled: true
    top_k: 50
    embedding_model: "BAAI/bge-m3"
    metric: "cosine"
  fusion:
    strategy: "rrf"
    rrf_k: 60
    final_top_k: 20
    max_chunks_per_document: 2
  reranker:
    enabled: false
    top_k: 30
  security:
    require_tenant_filter: true
    require_acl_filter: true
  cache:
    enabled: true
    ttl_seconds: 300

pip install rank-bm25 sentence-transformers numpy

from __future__ import annotations

import re
import time
import unicodedata
from dataclasses import dataclass
from typing import Iterable, Mapping, Protocol, Sequence

import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer


@dataclass(frozen=True)
class AuthContext:
    tenant_id: str
    roles: frozenset[str]
    permission_hash: str


@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    document_id: str
    text: str
    tenant_id: str
    acl_roles: frozenset[str]
    source_uri: str
    page: int | None
    index_version: str
    deleted: bool = False


@dataclass(frozen=True)
class SearchHit:
    chunk: Chunk
    score: float
    rank: int
    source: str


@dataclass(frozen=True)
class HybridHit:
    chunk: Chunk
    rrf_score: float
    final_rank: int
    bm25_rank: int | None
    dense_rank: int | None


@dataclass(frozen=True)
class RetrievalConfig:
    active_index_version: str
    bm25_top_k: int = 50
    dense_top_k: int = 50
    final_top_k: int = 10
    rrf_k: int = 60
    max_chunks_per_document: int = 2


class Retriever(Protocol):
    def search(self, query: str, auth: AuthContext, limit: int) -> list[SearchHit]:
        ...


CODE_SAFE_TOKEN_PATTERN = re.compile(
    r"[A-Za-z]+[+#]{1,2}|[A-Za-z0-9]+(?:[._:/-][A-Za-z0-9]+)+|\w+",
    re.UNICODE,
)


def normalize_query(query: str) -> str:
    normalized = unicodedata.normalize("NFKC", query)
    normalized = re.sub(r"\s+", " ", normalized).strip()
    return normalized


def fold_vietnamese_accents(token: str) -> str:
    token = token.replace("Đ", "D").replace("đ", "d")
    decomposed = unicodedata.normalize("NFD", token)
    return "".join(ch for ch in decomposed if unicodedata.category(ch) != "Mn")


def tokenize_code_safe(text: str) -> list[str]:
    text = normalize_query(text).lower()
    tokens = CODE_SAFE_TOKEN_PATTERN.findall(text)
    expanded: list[str] = []
    for token in tokens:
        expanded.append(token)
        folded = fold_vietnamese_accents(token)
        if folded != token:
            expanded.append(folded)
    return expanded


def is_visible(chunk: Chunk, auth: AuthContext, active_index_version: str) -> bool:
    return (
        chunk.tenant_id == auth.tenant_id
        and chunk.index_version == active_index_version
        and not chunk.deleted
        and bool(chunk.acl_roles.intersection(auth.roles))
    )


class InMemoryBM25Retriever:
    def __init__(self, chunks: Sequence[Chunk], active_index_version: str) -> None:
        self._chunks = list(chunks)
        self._active_index_version = active_index_version
        tokenized_corpus = [tokenize_code_safe(chunk.text) for chunk in self._chunks]
        self._bm25 = BM25Okapi(tokenized_corpus)

    def search(self, query: str, auth: AuthContext, limit: int) -> list[SearchHit]:
        tokens = tokenize_code_safe(query)
        if not tokens:
            return []

        scores = np.asarray(self._bm25.get_scores(tokens), dtype=np.float32)
        order = np.argsort(-scores)
        hits: list[SearchHit] = []

        for idx in order:
            chunk = self._chunks[int(idx)]
            if scores[idx] <= 0:
                continue
            if not is_visible(chunk, auth, self._active_index_version):
                continue
            hits.append(
                SearchHit(
                    chunk=chunk,
                    score=float(scores[idx]),
                    rank=len(hits) + 1,
                    source="bm25",
                )
            )
            if len(hits) >= limit:
                break

        return hits


class InMemoryDenseRetriever:
    def __init__(
        self,
        chunks: Sequence[Chunk],
        model_name: str,
        active_index_version: str,
    ) -> None:
        self._chunks = list(chunks)
        self._model = SentenceTransformer(model_name)
        self._active_index_version = active_index_version
        self._doc_embeddings = self._model.encode(
            [chunk.text for chunk in self._chunks],
            convert_to_numpy=True,
            normalize_embeddings=True,
        )

    def search(self, query: str, auth: AuthContext, limit: int) -> list[SearchHit]:
        query_embedding = self._model.encode(
            [normalize_query(query)],
            convert_to_numpy=True,
            normalize_embeddings=True,
        )[0]
        scores = np.asarray(self._doc_embeddings @ query_embedding, dtype=np.float32)
        order = np.argsort(-scores)
        hits: list[SearchHit] = []

        for idx in order:
            chunk = self._chunks[int(idx)]
            if not is_visible(chunk, auth, self._active_index_version):
                continue
            hits.append(
                SearchHit(
                    chunk=chunk,
                    score=float(scores[idx]),
                    rank=len(hits) + 1,
                    source="dense",
                )
            )
            if len(hits) >= limit:
                break

        return hits


def rrf_fuse(rankings: Sequence[Sequence[SearchHit]], rrf_k: int) -> list[HybridHit]:
    by_chunk: dict[str, Chunk] = {}
    rrf_scores: dict[str, float] = {}
    ranks: dict[str, dict[str, int]] = {}

    for ranking in rankings:
        for hit in ranking:
            chunk_id = hit.chunk.chunk_id
            by_chunk[chunk_id] = hit.chunk
            rrf_scores[chunk_id] = rrf_scores.get(chunk_id, 0.0) + 1.0 / (rrf_k + hit.rank)
            ranks.setdefault(chunk_id, {})[hit.source] = hit.rank

    ordered = sorted(rrf_scores.items(), key=lambda item: item[1], reverse=True)
    return [
        HybridHit(
            chunk=by_chunk[chunk_id],
            rrf_score=score,
            final_rank=i + 1,
            bm25_rank=ranks.get(chunk_id, {}).get("bm25"),
            dense_rank=ranks.get(chunk_id, {}).get("dense"),
        )
        for i, (chunk_id, score) in enumerate(ordered)
    ]


def limit_document_diversity(
    hits: Iterable[HybridHit],
    final_top_k: int,
    max_chunks_per_document: int,
) -> list[HybridHit]:
    per_document_count: dict[str, int] = {}
    selected: list[HybridHit] = []

    for hit in hits:
        count = per_document_count.get(hit.chunk.document_id, 0)
        if count >= max_chunks_per_document:
            continue
        per_document_count[hit.chunk.document_id] = count + 1
        selected.append(
            HybridHit(
                chunk=hit.chunk,
                rrf_score=hit.rrf_score,
                final_rank=len(selected) + 1,
                bm25_rank=hit.bm25_rank,
                dense_rank=hit.dense_rank,
            )
        )
        if len(selected) >= final_top_k:
            break

    return selected


class HybridSearchService:
    def __init__(
        self,
        bm25: Retriever,
        dense: Retriever,
        config: RetrievalConfig,
    ) -> None:
        self._bm25 = bm25
        self._dense = dense
        self._config = config

    def search(self, query: str, auth: AuthContext) -> tuple[list[HybridHit], Mapping[str, float]]:
        normalized_query = normalize_query(query)

        started = time.perf_counter()
        bm25_hits = self._bm25.search(normalized_query, auth, self._config.bm25_top_k)
        bm25_ms = (time.perf_counter() - started) * 1000

        started = time.perf_counter()
        dense_hits = self._dense.search(normalized_query, auth, self._config.dense_top_k)
        dense_ms = (time.perf_counter() - started) * 1000

        started = time.perf_counter()
        fused = rrf_fuse([bm25_hits, dense_hits], self._config.rrf_k)
        final_hits = limit_document_diversity(
            fused,
            final_top_k=self._config.final_top_k,
            max_chunks_per_document=self._config.max_chunks_per_document,
        )
        merge_ms = (time.perf_counter() - started) * 1000

        metrics = {
            "bm25_ms": bm25_ms,
            "dense_ms": dense_ms,
            "merge_ms": merge_ms,
            "bm25_candidates": float(len(bm25_hits)),
            "dense_candidates": float(len(dense_hits)),
            "final_hits": float(len(final_hits)),
        }
        return final_hits, metrics

CHUNKS = [
    Chunk(
        chunk_id="a:refund:001",
        document_id="refund_policy",
        text="Khách hàng có thể yêu cầu hoàn tiền trong 7 ngày cho gói Pro.",
        tenant_id="company_a",
        acl_roles=frozenset({"employee", "support"}),
        source_uri="s3://company-a/policy/refund.pdf",
        page=1,
        index_version="dev-index-v1",
    ),
    Chunk(
        chunk_id="a:invoice:001",
        document_id="invoice_vat",
        text="Để xuất hóa đơn VAT, cần cung cấp tên công ty và mã số thuế.",
        tenant_id="company_a",
        acl_roles=frozenset({"employee", "finance"}),
        source_uri="s3://company-a/finance/vat.pdf",
        page=2,
        index_version="dev-index-v1",
    ),
    Chunk(
        chunk_id="a:sla:001",
        document_id="sla_enterprise",
        text="Gói Enterprise có SLA uptime 99.9% và hỗ trợ P1 trong 2 giờ.",
        tenant_id="company_a",
        acl_roles=frozenset({"support"}),
        source_uri="s3://company-a/support/sla.pdf",
        page=4,
        index_version="dev-index-v1",
    ),
    Chunk(
        chunk_id="a:security:001",
        document_id="security_2fa",
        text="Tài khoản admin bắt buộc bật xác thực hai lớp 2FA.",
        tenant_id="company_a",
        acl_roles=frozenset({"admin"}),
        source_uri="s3://company-a/security/2fa.pdf",
        page=3,
        index_version="dev-index-v1",
    ),
    Chunk(
        chunk_id="a:api:001",
        document_id="api_rate_limit",
        text="API giới hạn 600 request mỗi phút. Vượt giới hạn trả về HTTP 429.",
        tenant_id="company_a",
        acl_roles=frozenset({"developer", "support"}),
        source_uri="s3://company-a/dev/api-rate-limit.md",
        page=None,
        index_version="dev-index-v1",
    ),
    Chunk(
        chunk_id="b:refund:001",
        document_id="refund_policy",
        text="Khách hàng công ty B có thể hoàn tiền trong 14 ngày.",
        tenant_id="company_b",
        acl_roles=frozenset({"employee", "support"}),
        source_uri="s3://company-b/policy/refund.pdf",
        page=1,
        index_version="dev-index-v1",
    ),
]

config = RetrievalConfig(active_index_version="dev-index-v1", final_top_k=5)
auth = AuthContext(
    tenant_id="company_a",
    roles=frozenset({"employee", "support", "developer"}),
    permission_hash="demo-permission-hash",
)

bm25 = InMemoryBM25Retriever(CHUNKS, active_index_version=config.active_index_version)
dense = InMemoryDenseRetriever(
    CHUNKS,
    model_name="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
    active_index_version=config.active_index_version,
)
service = HybridSearchService(bm25=bm25, dense=dense, config=config)

for query in ["tôi muốn lấy lại tiền", "HTTP 429", "SLA enterprise P1", "xuat VAT"]:
    hits, metrics = service.search(query, auth)
    print(query, metrics)
    for hit in hits:
        print(
            hit.final_rank,
            hit.chunk.document_id,
            hit.rrf_score,
            {"bm25_rank": hit.bm25_rank, "dense_rank": hit.dense_rank},
        )

def hit_at_k(results: Sequence[str], relevant: set[str], k: int) -> float:
    return float(any(doc_id in relevant for doc_id in results[:k]))


def recall_at_k(results: Sequence[str], relevant: set[str], k: int) -> float:
    if not relevant:
        return 0.0
    return len(set(results[:k]).intersection(relevant)) / len(relevant)


def mrr_at_k(results: Sequence[str], relevant: set[str], k: int) -> float:
    for idx, doc_id in enumerate(results[:k], start=1):
        if doc_id in relevant:
            return 1.0 / idx
    return 0.0


def evaluate(
    predictions: Mapping[str, Sequence[str]],
    qrels: Mapping[str, set[str]],
    k: int,
) -> dict[str, float]:
    query_ids = list(qrels)
    return {
        f"hit@{k}": sum(hit_at_k(predictions[qid], qrels[qid], k) for qid in query_ids) / len(query_ids),
        f"recall@{k}": sum(recall_at_k(predictions[qid], qrels[qid], k) for qid in query_ids) / len(query_ids),
        f"mrr@{k}": sum(mrr_at_k(predictions[qid], qrels[qid], k) for qid in query_ids) / len(query_ids),
    }

## Retrieval Benchmark

Corpus:
- Chunks:
- Documents:
- Languages:
- Index version:
- Embedding model:
- BM25 analyzer:

Query set:
- Total queries:
- semantic:
- keyword:
- mixed:
- exact_code:
- no_diacritic:
- english_mix:

| Config | Hit@5 | Recall@10 | MRR@10 | p95 ms | p99 ms | Notes |
|---|---:|---:|---:|---:|---:|---|
| BM25-only | | | | | | |
| Dense-only | | | | | | |
| Hybrid RRF | | | | | | |
| Hybrid RRF + reranker | | | | | | |

Findings:
1.
2.
3.

Decision:
- Roll out:
- Need more eval:
- Config:
- Risks:

python -m venv .venv
source .venv/bin/activate
pip install rank-bm25 sentence-transformers numpy pandas pytest

DOCS = [
    {
        "id": "refund_policy",
        "tenant_id": "company_a",
        "roles": ["employee", "support"],
        "text": "Khách hàng có thể yêu cầu hoàn tiền trong 7 ngày cho gói Pro.",
    },
    {
        "id": "invoice_vat",
        "tenant_id": "company_a",
        "roles": ["employee", "finance"],
        "text": "Để xuất hóa đơn VAT, cần cung cấp tên công ty và mã số thuế.",
    },
    {
        "id": "sla_enterprise",
        "tenant_id": "company_a",
        "roles": ["support"],
        "text": "Gói Enterprise có SLA uptime 99.9% và hỗ trợ P1 trong 2 giờ.",
    },
    {
        "id": "password_reset",
        "tenant_id": "company_a",
        "roles": ["employee", "support"],
        "text": "Người dùng có thể reset mật khẩu bằng email đăng ký.",
    },
    {
        "id": "security_2fa",
        "tenant_id": "company_a",
        "roles": ["admin"],
        "text": "Tài khoản admin bắt buộc bật xác thực hai lớp 2FA.",
    },
    {
        "id": "api_rate_limit",
        "tenant_id": "company_a",
        "roles": ["developer", "support"],
        "text": "API giới hạn 600 request mỗi phút. Vượt giới hạn trả về HTTP 429.",
    },
    {
        "id": "refund_policy_b",
        "tenant_id": "company_b",
        "roles": ["employee", "support"],
        "text": "Khách hàng công ty B có thể yêu cầu hoàn tiền trong 14 ngày.",
    },
]

QUERIES = [
    {
        "id": "q_semantic_refund",
        "query": "tôi muốn lấy lại tiền gói Pro",
        "category": "semantic",
        "relevant": {"refund_policy"},
    },
    {
        "id": "q_keyword_vat",
        "query": "xuat VAT cho cong ty",
        "category": "keyword_no_diacritic",
        "relevant": {"invoice_vat"},
    },
    {
        "id": "q_keyword_sla",
        "query": "SLA enterprise P1",
        "category": "keyword",
        "relevant": {"sla_enterprise"},
    },
    {
        "id": "q_code_429",
        "query": "lỗi 429 là gì",
        "category": "exact_code",
        "relevant": {"api_rate_limit"},
    },
    {
        "id": "q_semantic_password",
        "query": "quên password thì làm sao vào lại tài khoản",
        "category": "mixed",
        "relevant": {"password_reset"},
    },
]

import re
import unicodedata


TOKEN_PATTERN = re.compile(
    r"[A-Za-z]+[+#]{1,2}|[A-Za-z0-9]+(?:[._:/-][A-Za-z0-9]+)+|\w+",
    re.UNICODE,
)


def normalize_text(text: str) -> str:
    text = unicodedata.normalize("NFKC", text)
    text = re.sub(r"\s+", " ", text).strip()
    return text


def fold_vietnamese_accents(token: str) -> str:
    token = token.replace("Đ", "D").replace("đ", "d")
    decomposed = unicodedata.normalize("NFD", token)
    return "".join(ch for ch in decomposed if unicodedata.category(ch) != "Mn")


def tokenize(text: str) -> list[str]:
    tokens = TOKEN_PATTERN.findall(normalize_text(text).lower())
    expanded: list[str] = []
    for token in tokens:
        expanded.append(token)
        folded = fold_vietnamese_accents(token)
        if folded != token:
            expanded.append(folded)
    return expanded

assert "429" in tokenize("HTTP 429")
assert "2fa" in tokenize("Bật 2FA")
assert "bat" in tokenize("Bật 2FA")
assert tokenize("C++") != ["c"]

import numpy as np
from rank_bm25 import BM25Okapi


class BM25Search:
    def __init__(self, docs: list[dict]) -> None:
        self.docs = docs
        self.index = BM25Okapi([tokenize(doc["text"]) for doc in docs])

    def search(self, query: str, tenant_id: str, roles: set[str], k: int) -> list[tuple[str, float]]:
        scores = np.asarray(self.index.get_scores(tokenize(query)), dtype=np.float32)
        order = np.argsort(-scores)
        results: list[tuple[str, float]] = []

        for idx in order:
            doc = self.docs[int(idx)]
            if scores[idx] <= 0:
                continue
            if doc["tenant_id"] != tenant_id:
                continue
            if not roles.intersection(doc["roles"]):
                continue
            results.append((doc["id"], float(scores[idx])))
            if len(results) >= k:
                break

        return results

from sentence_transformers import SentenceTransformer


class DenseSearch:
    def __init__(self, docs: list[dict], model_name: str) -> None:
        self.docs = docs
        self.model = SentenceTransformer(model_name)
        self.doc_embeddings = self.model.encode(
            [doc["text"] for doc in docs],
            convert_to_numpy=True,
            normalize_embeddings=True,
        )

    def search(self, query: str, tenant_id: str, roles: set[str], k: int) -> list[tuple[str, float]]:
        query_embedding = self.model.encode(
            [normalize_text(query)],
            convert_to_numpy=True,
            normalize_embeddings=True,
        )[0]
        scores = np.asarray(self.doc_embeddings @ query_embedding, dtype=np.float32)
        order = np.argsort(-scores)
        results: list[tuple[str, float]] = []

        for idx in order:
            doc = self.docs[int(idx)]
            if doc["tenant_id"] != tenant_id:
                continue
            if not roles.intersection(doc["roles"]):
                continue
            results.append((doc["id"], float(scores[idx])))
            if len(results) >= k:
                break

        return results

MODEL_NAME = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"

def rrf_merge(rankings: list[list[tuple[str, float]]], rrf_k: int = 60) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, (doc_id, _) in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (rrf_k + rank)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

final_score = bm25_score + cosine_score

def hit_at_k(results: list[str], relevant: set[str], k: int) -> float:
    return float(any(doc_id in relevant for doc_id in results[:k]))


def recall_at_k(results: list[str], relevant: set[str], k: int) -> float:
    return len(set(results[:k]).intersection(relevant)) / len(relevant)


def mrr_at_k(results: list[str], relevant: set[str], k: int) -> float:
    for rank, doc_id in enumerate(results[:k], start=1):
        if doc_id in relevant:
            return 1.0 / rank
    return 0.0

def evaluate_system(name: str, predictions: dict[str, list[str]], k: int = 5) -> dict[str, float]:
    by_id = {item["id"]: item for item in QUERIES}
    hit = []
    recall = []
    mrr = []
    for query_id, ranked_doc_ids in predictions.items():
        relevant = by_id[query_id]["relevant"]
        hit.append(hit_at_k(ranked_doc_ids, relevant, k))
        recall.append(recall_at_k(ranked_doc_ids, relevant, k))
        mrr.append(mrr_at_k(ranked_doc_ids, relevant, k))
    return {
        "system": name,
        f"hit@{k}": sum(hit) / len(hit),
        f"recall@{k}": sum(recall) / len(recall),
        f"mrr@{k}": sum(mrr) / len(mrr),
    }

bm25 = BM25Search(DOCS)
dense = DenseSearch(DOCS, MODEL_NAME)

tenant_id = "company_a"
roles = {"employee", "support", "developer"}

bm25_predictions: dict[str, list[str]] = {}
dense_predictions: dict[str, list[str]] = {}
hybrid_predictions: dict[str, list[str]] = {}

for item in QUERIES:
    bm25_hits = bm25.search(item["query"], tenant_id, roles, k=5)
    dense_hits = dense.search(item["query"], tenant_id, roles, k=5)
    hybrid_hits = rrf_merge([bm25_hits, dense_hits], rrf_k=60)[:5]

    bm25_predictions[item["id"]] = [doc_id for doc_id, _ in bm25_hits]
    dense_predictions[item["id"]] = [doc_id for doc_id, _ in dense_hits]
    hybrid_predictions[item["id"]] = [doc_id for doc_id, _ in hybrid_hits]

    print("\\nQUERY:", item["query"], "|", item["category"])
    print("expected:", item["relevant"])
    print("bm25:", bm25_predictions[item["id"]])
    print("dense:", dense_predictions[item["id"]])
    print("hybrid:", hybrid_predictions[item["id"]])

print(evaluate_system("bm25", bm25_predictions, k=5))
print(evaluate_system("dense", dense_predictions, k=5))
print(evaluate_system("hybrid", hybrid_predictions, k=5))

def test_company_a_cannot_see_company_b():
    bm25 = BM25Search(DOCS)
    hits = bm25.search("hoàn tiền", tenant_id="company_a", roles={"employee"}, k=20)
    assert hits
    assert "refund_policy_b" not in [doc_id for doc_id, _ in hits]


def test_employee_cannot_see_admin_2fa_doc():
    bm25 = BM25Search(DOCS)
    hits = bm25.search("2FA admin", tenant_id="company_a", roles={"employee"}, k=20)
    assert "security_2fa" not in [doc_id for doc_id, _ in hits]


def test_rrf_keeps_dense_only_candidate():
    bm25_hits = [("doc_a", 10.0)]
    dense_hits = [("doc_b", 0.9)]
    fused = [doc_id for doc_id, _ in rrf_merge([bm25_hits, dense_hits])]
    assert "doc_a" in fused
    assert "doc_b" in fused

# Day 36 Hybrid Retrieval Report

## Dataset

- Documents:
- Queries:
- Query categories:
- Embedding model:
- Analyzer/tokenizer:

## Metrics

| System | Hit@5 | Recall@5 | MRR@5 | Notes |
|---|---:|---:|---:|---|
| BM25-only | | | | |
| Dense-only | | | | |
| Hybrid RRF | | | | |

## Findings

1. BM25 mạnh ở:
2. Dense mạnh ở:
3. Hybrid cải thiện ở:
4. Query normalization làm sai ở:
5. ACL/filter test:

## Production Readiness

Hybrid search dùng được trong production không?

Trả lời:

Điều kiện còn thiếu trước production:

- Golden set tối thiểu 30-50 query có tag `keyword`, `semantic`, `no-diacritic`, `code-heavy` và `permission`.
- Tenant/ACL/deleted/index filters được enforce giống nhau ở BM25 path và dense path.
- Có p95/p99 latency, query timeout, fallback khi một retriever lỗi và regression test cho analyzer/embedding/index version.