How to add reranking to your RAG pipeline

Practical · ~10 min read · Updated 21 Jun 2026

The fix for mediocre RAG answers is rarely a bigger LLM — it's usually a better retrieval order. A reranker sits between your vector search and your LLM call, re-scores the candidates, and ensures only the most relevant passages land in the prompt.

The ordering problem in RAG

A typical RAG system embeds your documents and stores the vectors. At query time it fetches the k-nearest vectors to the query embedding and stuffs those chunks into the LLM prompt. The problem: cosine similarity between independent embeddings is a coarse relevance signal. The correct chunk might be in the top 20 results, but sitting at position 14 — outside the 5 you actually send to the model.

This is the "lost in the middle" problem in reverse: the right answer was never at the top in the first place. Reranking fixes it by applying a more expensive, more accurate relevance model to the shortlist the retriever already found.

The retrieve-rerank-generate pattern

The retriever handles scale (millions of documents at millisecond speed). The reranker handles quality (precise ordering of a few dozen candidates). The LLM handles synthesis. Each stage does only what it's good at.

Code walkthrough

Here's a self-contained Python example using a local bge-reranker. In production you'd swap rank_documents for a call to Cohere, Jina, or Voyage if you prefer hosted APIs.

With a local cross-encoder (sentence-transformers)

from sentence_transformers import CrossEncoder

# Load once at startup — reuse across requests
reranker = CrossEncoder("BAAI/bge-reranker-base", max_length=512)

def rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
    """Return top_n candidates reranked by relevance to query."""
    pairs = [(query, doc) for doc in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(scores, candidates), reverse=True)
    return [doc for _, doc in ranked[:top_n]]

# --- In your RAG pipeline ---
raw_chunks = vector_db.search(query, top_k=50)   # retrieve wide
best_chunks = rerank(query, raw_chunks, top_n=5)  # rerank tight
answer = llm.complete(build_prompt(query, best_chunks))  # generate

With the Cohere hosted API

import cohere

co = cohere.Client("YOUR_API_KEY")

def rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
    result = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=candidates,
        top_n=top_n,
    )
    return [candidates[r.index] for r in result.results]

With Jina Reranker API

import requests

def rerank(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
    resp = requests.post(
        "https://api.jina.ai/v1/rerank",
        headers={"Authorization": "Bearer YOUR_KEY"},
        json={
            "model": "jina-reranker-v2-base-multilingual",
            "query": query,
            "documents": candidates,
            "top_n": top_n,
        },
    ).json()
    indices = [r["index"] for r in resp["results"]]
    return [candidates[i] for i in indices]

Inside a framework (LangChain · LlamaIndex · Haystack)

If you already use a RAG framework, reranking is usually a drop-in node that wraps your existing retriever. The pattern is identical — retrieve wide, rerank, keep the top-n — just expressed in the framework's vocabulary.

# LangChain — ContextualCompressionRetriever wraps any base retriever
from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

model = HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-base")
compressor = CrossEncoderReranker(model=model, top_n=5)
retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=vectorstore.as_retriever(search_kwargs={"k": 50}),
)
docs = retriever.invoke(query)   # retrieve 50 → rerank → top 5

# LlamaIndex — a node postprocessor on the query engine
from llama_index.core.postprocessor import SentenceTransformerRerank

reranker = SentenceTransformerRerank(model="BAAI/bge-reranker-base", top_n=5)
query_engine = index.as_query_engine(
    similarity_top_k=50,                 # retrieve wide
    node_postprocessors=[reranker],       # rerank to top 5
)
response = query_engine.query(query)

# Haystack 2.x — a Ranker component in the pipeline
from haystack import Pipeline
from haystack.components.rankers import TransformersSimilarityRanker

ranker = TransformersSimilarityRanker(model="BAAI/bge-reranker-base", top_k=5)
pipe = Pipeline()
pipe.add_component("retriever", retriever)   # returns ~50 docs
pipe.add_component("ranker", ranker)
pipe.connect("retriever.documents", "ranker.documents")

All three accept hosted rerankers too (Cohere, Jina, Voyage) via their respective integration packages — swap the component, keep the pipeline.

Choosing top-k values

You have two k values to tune: how many to retrieve and how many to keep after reranking.

Latency trade-offs

Reranking adds a model call to your pipeline. The cost depends on the approach:

For most RAG applications, 100–300 ms total pipeline latency is fine and the quality gain is worth it. If your SLA is very tight, either host on GPU, use a tiny model, or cap retrieval_k at 20–30 instead of 50.

Measuring whether it actually helped

Don't add a reranker on faith — measure it. Build a small evaluation set of queries paired with the chunk(s) that should be retrieved, then compare retrieval-only vs retrieval+rerank on the same set. Three metrics cover almost every case:

MRR is the intuitive one: if the right chunk lands at position 1 you score 1.0, at position 2 you score 0.5, at position 4 you score 0.25. Averaged over your query set, it's a single number that captures "how near the top is the answer?" — exactly what a reranker is meant to improve.

# Minimal MRR + Recall@k over a labelled eval set.
# Each example: a query, the candidate texts, and the indices that are relevant.
def reciprocal_rank(ranked_ids, relevant_ids):
    for i, doc_id in enumerate(ranked_ids, start=1):
        if doc_id in relevant_ids:
            return 1.0 / i
    return 0.0

def evaluate(examples, rank_fn, k=5):
    mrr = recall = 0.0
    for ex in examples:
        ranked = rank_fn(ex["query"], ex["candidates"])  # returns ids, best first
        mrr += reciprocal_rank(ranked, ex["relevant"])
        recall += 1.0 if set(ranked[:k]) & set(ex["relevant"]) else 0.0
    n = len(examples)
    return {"MRR": mrr / n, f"Recall@{k}": recall / n}

base = evaluate(eval_set, retrieve_only)        # baseline
reranked = evaluate(eval_set, retrieve_then_rerank)  # with the reranker
print(base, reranked)   # expect MRR and nDCG to rise after reranking

Which reranker to pick

The short version:

• Want to self-host, English-only, free: bge-reranker-v2-m3 — strong, widely deployed.
• Want a hosted API, best multilingual quality: Cohere Rerank v3.5.
• Want open weights + hosted API + tiny browser-runnable model: Jina Reranker v2.
• Optimising for retrieval-specific quality: Voyage Rerank 2.
• Want Apache 2.0 open weights, browser-runnable xsmall: mxbai-rerank.

See the full model comparison →

Keep reading