Best Practices for Hybrid Vector Search & RAG Retrieval
Key strategies for document chunking, hybrid keyword-dense embedding indexing, Reciprocal Rank Fusion (RRF), and grounded context validation in enterprise knowledge search.
Gourav Singh
Founder & AI Systems Architect
The Limits of Naive Vector Retrieval
Retrieval-Augmented Generation (RAG) is commonly implemented by chunking documents into arbitrary 500-character blocks, computing dense vector embeddings (e.g. OpenAI text-embedding-3 or BGE-M3), and querying a vector index via cosine similarity.
In production enterprise systems, this naive architecture suffers from three fatal weaknesses: loss of document hierarchy, vulnerability to specific keyword/SKU queries, and poor ranking of dense technical tables. Solving this requires a hybrid multi-stage retrieval architecture.
Heading-Aware & Structure-Preserving Chunking
Rather than slicing text by arbitrary character or token boundaries, chunking must be semantic and document-aware. In technical documentation and enterprise manuals, every chunk must inherit its parent section hierarchy (e.g., `Document Title > Chapter 3 > Subsection B`).
Prepending the breadcrumb hierarchy to the chunk content before embedding guarantees that the vector accurately captures both local detail and broader document context.
interface StructuredChunk {
chunkId: string;
hierarchy: string[]; // e.g. ["HR Manual", "Health Benefits", "Dental Coverage"]
content: string;
tokenCount: number;
metadata: {
pageNumber: number;
sourceDocument: string;
sectionHash: string;
};
}
export function buildSemanticContextString(chunk: StructuredChunk): string {
const contextHeader = chunk.hierarchy.join(' > ');
return `[Context: ${contextHeader}]
${chunk.content}`;
}Hybrid Retrieval & Reciprocal Rank Fusion (RRF)
To achieve both semantic comprehension and exact-match precision, NorAI employs hybrid search combining sparse BM25 keyword matching with dense vector similarity.
The individual score distributions from dense vector search and sparse BM25 cannot be directly summed because their scales differ. We normalize and merge the ranked candidate lists using Reciprocal Rank Fusion (RRF), where constant k=60 prevents top-rank skewing:
| Retrieval Model | Recall@5 | Precision@5 | P95 Latency |
|---|---|---|---|
| Dense Embeddings Only | 74.2% | 68.1% | 18ms |
| BM25 Keyword Search Only | 68.9% | 61.4% | 4ms |
| Hybrid RRF (Dense + BM25) | 91.8% | 87.6% | 22ms |
| Hybrid RRF + Cross-Encoder Rerank | 96.4% | 93.2% | 48ms |
Retrieval accuracy benchmark on enterprise legal and technical spec datasets.
export function reciprocalRankFusion(
denseRankings: string[],
sparseRankings: string[],
k: number = 60
): Map<string, number> {
const fusedScores = new Map<string, number>();
const processList = (list: string[]) => {
list.forEach((docId, rank) => {
const currentScore = fusedScores.get(docId) || 0;
const rrfScore = 1 / (k + (rank + 1));
fusedScores.set(docId, currentScore + rrfScore);
});
};
processList(denseRankings);
processList(sparseRankings);
return new Map(
[...fusedScores.entries()].sort((a, b) => b[1] - a[1])
);
}Cross-Encoder Re-Ranking & Context Hygiene
After initial hybrid retrieval extracts the top-25 candidate chunks, a lightweight cross-encoder model (such as BGE-Reranker-Large or Cohere Rerank v3) re-evaluates the query-chunk pair with full cross-attention.
The top-5 highest-scoring chunks are formatted into a clean, markdown-delimited prompt with strict citation requirements: "Cite [Doc ID: X, Page: Y] for every claim made. If the provided context does not contain the answer, explicitly state that the information is unavailable."
Enterprise Knowledge Hub & RAG
Zero-hallucination document intelligence pipelines for proprietary enterprise data.