Reranking in RAG: The Missing Layer Between Retrieval and Generation
Missing part in the RAG pipeline, why the reranking is important, different types of rerankers, is what this article discusses about.
Your retriever returns 50 passages. The passage that directly answers the question is present, but it is ranked 18th. Your application sends only the first five passages to the language model, so the answer is incomplete.
This is not primarily a generation failure. It is a ranking failure.
Dense vector search is designed to search a large corpus quickly. A reranker solves a different problem: given a manageable candidate set, which passages are most relevant to this exact query? Production RAG systems often need both stages because the architecture that retrieves broadly is not necessarily the architecture that ranks precisely.
Key takeaways
Retrieval and reranking optimize different objectives. First-stage retrieval must find good candidates; second-stage ranking must order them well.
A reranker cannot recover a passage that was absent from the candidate set. Measure candidate recall before tuning the reranker.
Reciprocal Rank Fusion combines ranked lists. It is useful before neural reranking, but it is not a substitute for query-document relevance modeling.
Cross-encoders usually offer stronger query-document interaction than single-vector retrieval, at higher latency.
Model selection should be based on your own labeled queries, latency budget, document lengths, languages, privacy constraints, and cost.
Evaluate the complete RAG system. Better ranking metrics matter only if they improve supported answers for real users.
Why Vector Search Alone Often Misses the Best Evidence
Dense retrieval compresses a query and every document into separate vectors. At query time, the system compares the query vector with precomputed document vectors and searches an approximate nearest-neighbor index.
That separation is what makes dense retrieval scalable. The corpus can be embedded before users ask questions, and millions of vectors can be searched quickly.
The tradeoff is information compression. Each query and passage becomes one vector, and relevance is estimated through a similarity function such as dot product or cosine similarity. The model does not jointly inspect the query and passage during retrieval.
That creates common ranking errors:
A passage is topically similar but does not answer the question.
A query contains several constraints, but a candidate satisfies only one.
Exact identifiers, product names, error codes, or legal clauses matter more than semantic similarity.
Negation changes the meaning: “features supported” and “features not supported” can remain close in embedding space.
Two passages discuss the same entity, but only one contains the required date, version, exception, or causal explanation.
Dense retrieval is not “bad at precision” in every workload. Strong embedding models can rank many queries well. The engineering point is narrower: single-vector similarity and exact query-document relevance are different operations, so measure whether the first is sufficient for your data.
The Production Pattern: Retrieve Broadly, Rank Carefully
A robust retrieval pipeline separates candidate generation from final ranking:
Query
|
+--> Dense retrieval ---------+
| |
+--> Sparse/BM25 retrieval ----+--> rank fusion --> candidate set
|
v
neural reranker
|
v
selected evidence
|
v
LLM
The first stage prioritizes candidate recall: relevant evidence should appear somewhere in the set.
The second stage prioritizes ranking quality: the best evidence should move toward the top.
The generation stage should receive only the amount of evidence it can use effectively, with source identifiers preserved for citations.
There is no universal K=100 or top_n=5. Those are reasonable starting points, not production constants. Tune them from measured curves:
Increase candidate count
Kand plot recall@K.Rerank those candidates and plot nDCG@N, MRR, or precision@N.
Measure end-to-end answer quality and unsupported-claim rate.
Record p50 and p95 latency plus cost per query.
Choose the smallest configuration that satisfies quality and service-level targets.
Bi-Encoders and Cross-Encoders Solve Different Problems
Bi-encoder retrieval
A bi-encoder processes the query and document independently:
query --> encoder --> query vector ----+
+--> similarity score
document --> encoder --> document vector --+
This architecture supports precomputed document embeddings and scalable approximate search.
Cross-encoder reranking
A cross-encoder processes the pair jointly:
[query tokens] [separator] [document tokens]
|
transformer
|
relevance score
Because query and document tokens interact inside the model, a cross-encoder can model fine-grained relationships that a single dot product may miss.
The cost is computation. Document representations cannot be reused in the same way, because every query-document pair must be scored. Cross-encoders therefore operate on tens or hundreds of retrieved candidates, not directly over the full corpus.
Late interaction
ColBERT occupies a useful middle ground. It independently encodes queries and passages into multiple token-level vectors, then applies a MaxSim operation at search time:
import numpy as np
def maxsim_score(query_vectors: np.ndarray, document_vectors: np.ndarray) -> float:
"""Compute the ColBERT-style sum of per-query-token maximum similarities."""
similarities = query_vectors @ document_vectors.T
return float(similarities.max(axis=1).sum())
This preserves more token-level information than a single-vector retriever while avoiding full cross-attention for every query-document pair. ColBERTv2 adds residual compression to reduce the storage cost of multi-vector indexing. The related PLAID engine accelerates late-interaction retrieval through centroid-based pruning.
Late interaction is not simply a “faster cross-encoder.” It is a different retrieval architecture with its own index, storage, and serving tradeoffs.
Hybrid Retrieval and RRF Come Before Neural Reranking
Dense and sparse retrieval fail differently:
Dense search handles semantic similarity and paraphrases.
BM25 or another sparse method rewards exact lexical overlap.
Hybrid retrieval combines both candidate sources. Reciprocal Rank Fusion (RRF) is a simple, score-independent way to merge their ranked lists:
RRF(d) = sum(1 / (c + rank_i(d)))
The constant c controls how much weight the highest positions receive. The value 60 is common because it was used in the original RRF work, but it is not mandatory.
RRF is best understood as rank fusion, not semantic reranking. It does not read a query and document together. It rewards documents that appear high across one or more input rankings.
That makes the typical order:
dense results + sparse results
|
v
RRF
|
v
neural reranker
|
v
final context
Executable RRF implementation
from collections import defaultdict
from collections.abc import Hashable, Mapping, Sequence
from dataclasses import dataclass
@dataclass(frozen=True)
class FusedResult:
document_id: Hashable
score: float
ranks: dict[str, int]
def reciprocal_rank_fusion(
rankings: Mapping[str, Sequence[Hashable]],
*,
constant: int = 60,
limit: int | None = None,
) -> list[FusedResult]:
"""Fuse ordered document-ID lists with Reciprocal Rank Fusion."""
if constant < 0:
raise ValueError("constant must be non-negative")
if limit is not None and limit < 1:
raise ValueError("limit must be positive")
scores: dict[Hashable, float] = defaultdict(float)
source_ranks: dict[Hashable, dict[str, int]] = defaultdict(dict)
for source, document_ids in rankings.items():
seen: set[Hashable] = set()
for rank, document_id in enumerate(document_ids, start=1):
if document_id in seen:
continue
seen.add(document_id)
scores[document_id] += 1.0 / (constant + rank)
source_ranks[document_id][source] = rank
fused = [
FusedResult(document_id, score, source_ranks[document_id])
for document_id, score in scores.items()
]
fused.sort(key=lambda item: (-item.score, str(item.document_id)))
return fused[:limit]
if __name__ == "__main__":
rankings = {
"dense": ["asyncio", "python", "snake", "coroutines"],
"bm25": ["coroutines", "asyncio", "python", "tornado"],
}
for result in reciprocal_rank_fusion(rankings, limit=5):
print(result)
Expected ordering starts with documents supported by both retrievers. Do not interpret the resulting RRF score as a calibrated probability of relevance.
Current Reranker Options and How to Choose
Model leaderboards are easy to misuse. Scores depend on the base retriever, candidate depth, dataset, language, document length, prompts or instructions, and evaluation implementation. A single table that mixes vendor-reported and independently reported numbers is not a sound selection method.
Use the following shortlist as a deployment map, then benchmark finalists on your data.
As of June 2026, Cohere documents two Rerank 4 variants: rerank-v4.0-pro for quality-oriented workloads and rerank-v4.0-fast for latency and throughput. Both support multilingual and semi-structured data with a 32K context window.
BAAI’s bge-reranker-v2-m3 is an open multilingual reranker. Its underlying model supports long sequences, but the maintainers note that it was fine-tuned with shorter inputs and recommend evaluating long-document behavior rather than assuming the maximum context is equally effective.
Qwen’s Qwen3-Reranker family is available in 0.6B, 4B, and 8B sizes and accepts task instructions. That flexibility is useful when “relevance” changes by workflow, but instruction wording becomes another parameter to version and test.
A Minimal Cohere Reranking Example
This example uses Cohere’s current v2 client and Rerank 4 model ID. It deliberately excludes retrieval and generation so the reranking contract remains clear.
"""
Requires:
pip install "cohere>=5,<6"
Environment:
export COHERE_API_KEY="..."
"""
import os
import cohere
def rerank_documents(
query: str,
documents: list[str],
*,
top_n: int = 5,
model: str = "rerank-v4.0-fast",
) -> list[dict[str, object]]:
if not documents:
return []
client = cohere.ClientV2(api_key=os.environ["COHERE_API_KEY"])
response = client.rerank(
model=model,
query=query,
documents=documents,
top_n=min(top_n, len(documents)),
)
return [
{
"index": result.index,
"score": result.relevance_score,
"document": documents[result.index],
}
for result in response.results
]
if __name__ == "__main__":
candidates = [
"Password resets are available from the account security page.",
"Enterprise plans support SAML SSO and automated user provisioning.",
"Invoices can be downloaded by workspace owners.",
]
ranked = rerank_documents(
"How do enterprise customers configure single sign-on?",
candidates,
top_n=2,
)
for item in ranked:
print(item)
Treat relevance scores as model outputs for ordering, not universal confidence thresholds. Calibrate any cutoff on labeled examples from your application.
A Local BGE Reranker Example
The official BGE model card recommends the FlagEmbedding package:
"""
Requires:
pip install -U FlagEmbedding
"""
from FlagEmbedding import FlagReranker
def rerank_locally(
query: str,
documents: list[str],
*,
top_n: int = 5,
) -> list[tuple[str, float]]:
model = FlagReranker(
"BAAI/bge-reranker-v2-m3",
use_fp16=True,
)
pairs = [[query, document] for document in documents]
scores = model.compute_score(pairs, normalize=True)
ranked = sorted(
zip(documents, scores, strict=True),
key=lambda item: item[1],
reverse=True,
)
return [(document, float(score)) for document, score in ranked[:top_n]]
if __name__ == "__main__":
candidates = [
"The deployment guide describes blue-green releases.",
"The authentication guide covers SAML SSO configuration.",
"The billing guide explains invoice exports.",
]
print(rerank_locally("How do I configure SAML?", candidates, top_n=2))
For CPU-only environments, set use_fp16=False. Load the model once when the service starts; do not instantiate it per request as this compact example does.
How to Evaluate Whether Reranking Helps
The correct question is not “Which reranker tops a public benchmark?” It is “Does reranking improve retrieval and answers for our users within our latency and cost budget?”
Build a labeled evaluation set
Start with real queries from logs or expected workflows. For each query:
Record one or more relevant passages.
Use graded labels when possible: irrelevant, partially relevant, highly relevant.
Include difficult negatives that share vocabulary but do not answer the question.
Include version-sensitive, multilingual, exact-match, and multi-constraint queries if your users ask them.
Keep a held-out set for final comparisons.
Even 100 carefully judged queries are more useful than a large generic benchmark for early product decisions.
Measure the retrieval stages separately
Candidate recall@K asks whether at least one relevant passage was retrieved:
queries with a relevant passage in top K / total queries
If candidate recall is poor, improve chunking, indexing, query rewriting, metadata filters, or hybrid retrieval. The reranker cannot score a missing passage.
MRR rewards systems that place the first relevant result early. It is useful when one passage is sufficient.
nDCG@N handles graded relevance and multiple relevant results. It is useful when the order of several good passages matters.
Precision@N measures how much of the final context is relevant. It helps detect noisy context sent to the LLM.
Measure end-to-end behavior
Retrieval metrics are proxies. Also measure:
answer correctness;
citation correctness;
percentage of claims supported by retrieved evidence;
refusal behavior when evidence is absent;
p50 and p95 latency;
reranking cost per query;
context tokens sent to the generator;
task completion or user acceptance.
Run an ablation:
A: dense retrieval
B: dense + sparse + RRF
C: dense + sparse + RRF + neural reranker
Use the same queries, generator, prompt, and evaluation rubric. This isolates the value of each retrieval layer.
Latency Engineering
Reranker latency depends on hardware, model size, sequence length, batch size, candidate count, and service topology. Published millisecond numbers are rarely portable to another deployment.
Profile your own pipeline by stage:
query processing
+ dense retrieval
+ sparse retrieval
+ fusion
+ reranking
+ context construction
+ generation
Then optimize the measured bottleneck:
Batch candidate pairs. GPUs benefit from batching, within memory limits.
Cap or chunk long documents deliberately. Silent truncation can remove the answer-bearing region.
Reduce K only after plotting recall@K. Arbitrary reductions can erase the reranker’s opportunity.
Use a fast model as the default and route difficult queries to a stronger model.
Cache carefully. Include normalized query, document version, model ID, instruction, and preprocessing version in the key.
Separate online and offline paths. Some ranking work can happen during indexing or document enrichment.
Use concurrency controls. Unbounded batches can create p95 latency spikes and GPU out-of-memory failures.
A safer cache key
Do not hash only the first 500 characters of a document; different documents can share that prefix. Include the full content digest and every factor that changes the score:
import hashlib
import json
def rerank_cache_key(
*,
query: str,
document: str,
model: str,
instruction: str = "",
) -> str:
payload = {
"query": " ".join(query.split()),
"document_sha256": hashlib.sha256(document.encode()).hexdigest(),
"model": model,
"instruction": instruction,
"schema_version": 1,
}
encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(encoded.encode()).hexdigest()
Semantic caching of “similar” queries is a separate optimization and can reuse an incorrect score. Validate it independently.
Production Service Design
A reranker can run inside an application process at low volume. A separate service becomes useful when you need independent scaling, dedicated accelerators, model versioning, batching across callers, or centralized observability.
Regardless of topology, production systems should define:
maximum candidates per request;
maximum query and document length;
truncation or chunking behavior;
request timeout and cancellation;
concurrency and batch limits;
model and preprocessing versions;
fallback behavior;
score and latency telemetry;
privacy and data-retention controls.
Do not assume the service is healthy because the endpoint returns HTTP 200. Monitor ranking distributions and retrieval outcomes. A tokenizer change, document-format change, or embedding migration can degrade relevance without producing an infrastructure error.
When Reranking Is the Wrong Next Step
Reranking adds value only when ranking quality is the constraint.
Skip or postpone it when:
Candidate recall is poor. Fix retrieval first.
The task is an exact lookup. Structured queries or deterministic filters may be better.
The complete candidate set already fits the downstream operation. Ranking may not change the outcome.
The latency budget cannot absorb the measured cost. Consider better first-stage retrieval, RRF, a smaller model, or an asynchronous workflow.
You have no evaluation set. Without labels, you cannot determine whether complexity improved the system.
Documents are badly chunked. A reranker cannot restore context split across inappropriate boundaries.
Metadata constraints are ignored. Apply access control, tenant, date, product, and version filters before semantic ranking.
Corpus size alone is not a reliable decision rule. A 500-document corpus can still contain ambiguous near-duplicates, while a million-document corpus with exact identifiers may need deterministic search more than neural reranking.
Common Failure Modes
Treating the reranker as a missing-document detector
It only reorders candidates. Track candidate recall separately.
Comparing raw relevance scores across models
Scores are not calibrated to a universal scale. Compare ordering and task metrics.
Sending unauthorized candidates to the reranker
Security filters must apply before reranking. Relevance does not override access control.
Losing source identity
Preserve document IDs, chunk IDs, versions, URLs, and original ranks through every stage. The generator needs stable provenance for citations.
Evaluating with synthetic easy queries
Use real ambiguity: similar policies, old and new product versions, misleading keyword overlap, negation, multilingual phrasing, and incomplete questions.
Assuming more context is always better
Additional low-relevance passages can distract the generator and increase cost. Optimize final context precision, not only recall.
Frequently Asked Questions
Does a strong embedding model remove the need for reranking?
Sometimes. A better embedding model may already meet your ranking and answer-quality targets. Cross-encoder reranking still offers a different interaction mechanism, but its incremental value must be measured against added latency and cost.
How many candidates should I rerank?
Choose K from a candidate recall curve, then test latency and ranking quality. Start with a broad range such as 20, 50, 100, and 200 rather than adopting a fixed value from another system.
Should RRF replace a neural reranker?
Not usually. RRF combines evidence from ranked lists; a neural reranker directly models query-document relevance. Test dense-only, hybrid plus RRF, and hybrid plus RRF plus neural reranking as separate configurations.
Can reranking solve multi-hop retrieval?
It improves ordering within a retrieval step, but it does not by itself discover a chain of evidence across multiple searches. Multi-hop questions may require query decomposition, iterative retrieval, graph retrieval, or an agentic search loop.
Should I use a score threshold?
Only after calibration on your data. A useful threshold depends on the model, query distribution, document type, and preprocessing. Often it is safer to select top N and separately detect insufficient evidence using a labeled validation set.
How should long documents be handled?
Understand the model’s documented context and chunking behavior. For important long documents, compare document-level reranking with section-level reranking and preserve parent-document metadata for reconstruction.
A Practical Rollout Plan
Collect representative queries and relevance judgments.
Establish a dense or hybrid retrieval baseline.
Measure candidate recall@K.
Add RRF if dense and sparse retrieval are complementary.
Benchmark at least one managed and one local reranker when governance permits.
Compare ranking, answer quality, latency, and total cost.
Preserve provenance and enforce authorization before ranking.
Deploy behind a feature flag and log model versions.
Review failed and low-confidence queries regularly.
Retune after corpus, chunking, embedding, or model changes.
Conclusion
Reranking is not a mandatory checkbox for every RAG application. It is a precise response to a measurable problem: the first-stage retriever finds relevant evidence but orders it poorly for the downstream task.
The production pattern is straightforward:
Retrieve broadly enough to achieve high candidate recall.
Combine complementary retrievers when useful.
Rerank a bounded candidate set for exact query-document relevance.
Pass only strong, authorized, traceable evidence to the generator.
Measure the effect on real answers, latency, and cost.
The central engineering lesson is to separate objectives. Retrieval, fusion, reranking, and generation are different stages with different failure modes. A production RAG system improves when each stage is evaluated for the job it actually performs.
Have been trying out RAG, and have been reading about Dense vs Sparse retrieval but my major question is how much of quality gets affected based on the selected embedding model I tried but with bunch of text the changes were minimal so I was not able to conclude it