Best Practices for Hybrid Vector Search & RAG Retrieval

Retrieval-Augmented Generation (RAG) is commonly implemented by chunking documents into arbitrary 500-character blocks, computing dense vector embeddings (e.g. OpenAI text-embedding-3 or BGE-M3), and querying a vector index via cosine similarity.

In production enterprise systems, this naive architecture suffers from three fatal weaknesses: loss of document hierarchy, vulnerability to specific keyword/SKU queries, and poor ranking of dense technical tables. Solving this requires a hybrid multi-stage retrieval architecture.

Rather than slicing text by arbitrary character or token boundaries, chunking must be semantic and document-aware. In technical documentation and enterprise manuals, every chunk must inherit its parent section hierarchy (e.g., `Document Title > Chapter 3 > Subsection B`).

Prepending the breadcrumb hierarchy to the chunk content before embedding gives the vector both the local detail and the surrounding document context, instead of an orphaned sentence.

To achieve both semantic comprehension and exact-match precision, Nor AI employs hybrid search combining sparse BM25 keyword matching with dense vector similarity.

The individual score distributions from dense vector search and sparse BM25 cannot be directly summed because their scales differ. We normalize and merge the ranked candidate lists using Reciprocal Rank Fusion (RRF), where the constant k=60 is the value from the original RRF paper (Cormack et al., 2009) and is what stops top-rank skewing:

After initial hybrid retrieval extracts the top-25 candidate chunks, a lightweight cross-encoder model (such as BGE-Reranker-Large or Cohere Rerank v3) re-evaluates the query-chunk pair with full cross-attention.

The top-5 highest-scoring chunks are formatted into a clean, markdown-delimited prompt with strict citation requirements: "Cite [Doc ID: X, Page: Y] for every claim made. If the provided context does not contain the answer, explicitly state that the information is unavailable."