How to self-host a reranker

Production · ~10 min read · Updated 12 Oct 2026

Hosted rerank APIs are convenient, but self-hosting gives you zero per-call cost, full data control, and predictable latency on your own GPU or CPU fleet. Here’s a practical path from a first CrossEncoder call to a production microservice.

When self-hosting wins

• High volume. Reranking 50–100 chunks per query adds up fast on per-1k-doc APIs.
• Data residency. Queries and documents never leave your VPC.
• Custom fine-tuning. You can swap in a domain-tuned checkpoint without waiting on a vendor.

When you need multilingual quality with minimal ops, a hosted API — Cohere, Voyage, or in mainland China Alibaba Cloud Model Studio and SiliconFlow — may still be cheaper all-in. The cost calculator prices your volume; see also choose by scenario.

Pick a model

Start with bge-reranker-v2-m3, which runs on CPU. On GPU, A/B Qwen3-Reranker-4B against it or mxbai-rerank-base-v2 on your labelled set — do not assume leaderboard order transfers. Until October 2026 this table recommended bge-reranker-base, BAAI’s 2023 model for Chinese and English only.

Quick start with sentence-transformers

pip install sentence-transformers torch

from sentence_transformers import CrossEncoder

model = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

query = "How do I add reranking to RAG?"
docs = [
    "Retrieve 50–100 candidates, rerank with a cross-encoder, keep top 5.",
    "London is the capital of the United Kingdom.",
]

pairs = [(query, d) for d in docs]
scores = model.predict(pairs)  # higher = more relevant
ranked = sorted(zip(scores, docs), reverse=True)

Batch your pairs — scoring 50 documents in one predict() call is far faster than 50 separate forwards. max_length=512 truncates anything longer, so chunk passages to fit rather than letting the end of a passage silently drop off.

Serving options

# Minimal FastAPI sketch
from fastapi import FastAPI
from sentence_transformers import CrossEncoder

app = FastAPI()
model = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

@app.post("/rerank")
def rerank(body: dict):
    q, docs = body["query"], body["documents"]
    scores = model.predict([(q, d) for d in docs])
    order = sorted(range(len(docs)), key=lambda i: -scores[i])
    return {"results": [{"index": i, "score": float(scores[i])} for i in order]}

You don’t have to write the service yourself: Hugging Face’s Text Embeddings Inference and Infinity both serve cross-encoders behind a rerank endpoint — the gte-multilingual-reranker-base card shows the Docker commands for each. Check that your model is on the server’s supported list first.

For ONNX export and C++ runtimes, see the model cards on Hugging Face. For browser-only experiments, try our in-browser demo with the xsmall ONNX builds.

Ops checklist

• GPU: worth it once you rerank more than a few dozen passages per query. Measure p50 / p95 on your own hardware before sizing — model cards rarely publish latency, and we don’t quote figures we haven’t measured.
• Memory: in fp16 the weights alone take about 2 bytes per parameter — roughly 1.1 GB for bge-reranker-v2-m3’s 568M and 8 GB for a 4B model — plus activations that grow with batch size and passage length.
• Warm-up: run a dummy predict on startup to avoid cold-start latency spikes.
• Monitoring: track p95 latency and NDCG on a labelled slice — evaluate rerankers.
• Fallback: if the rerank service is down, pass through vector top-k rather than failing the whole RAG request.

Keep reading