Dense Passage Retrieval, annotated

How to read this page

Any dotted word explains itself when you hover it, tab to it or tap it; so does every symbol in every equation. Each idea climbs the same ladder: an everyday picture, a tiny example, a diagram or demo, the math, and why it still matters. The contrastive training lesson trains a small dual encoder with exactly this loss, and the RAG lesson uses a retriever like this one inside a full question-answering pipeline.

Abstract · original

“In this work, we show that retrieval can be practically implemented using dense representations alone, where embeddings are learned from a small number of questions and passages by a simple dual-encoder framework.”Karpukhin et al. (2020), Abstract

Everyday picture

To answer a trivia question from Wikipedia, you first need to find the right page. The classic librarian, BM25, matches the words in your question against the words in each passage. DPR is a librarian who has learned what questions mean: it turns every passage into a point in space once, turns your question into a point, and fetches the nearest passages, even when they share no words with your question.

What the paper claims

• Two BERT encoders, one for questions and one for passages, fine-tuned only on question-passage pairs, with no special pre-training.
• 9 to 19 points better than a strong Lucene BM25 system at getting an answer-containing passage into the top 20.
• New state-of-the-art end-to-end answer accuracy on several open-domain QA benchmarks.

Why it matters today

“Embed the question, find the nearest passages, hand them to a reader” is the retrieval half of every RAG system. DPR is the paper that made it the default (see the RAG companion).

1 Introduction · original

Everyday picture

Ask “Who is the bad guy in Lord of the Rings?” The passage with the answer says “Sala Baker is best known for portraying the villain Sauron”. It never says “bad guy”. A keyword matcher struggles; a system that knows “bad guy” and “villain” mean the same thing finds it. That example is the paper's own.

Tiny example: sparse versus dense

A keyword system represents text as a very long, mostly-zero vector, one slot per vocabulary word (a sparse vector). “bad guy” has 1s in the “bad” and “guy” slots; “villain” has a 1 in the “villain” slot; their overlap is 0. A dense system represents each as, say, 768 numbers learned so that similar meanings point the same way: “bad guy” might be (0.8, 0.1, …) and “villain” (0.7, 0.2, …), with a high dot product. The paper stresses the two are complementary: dense for meaning, sparse for exact rare names.

The claim in numbers

On Natural Questions, DPR puts an answer-containing passage in its top 5 for 65.2% of questions, against 42.9% for BM25. End-to-end answer accuracy rises to 41.5% from the 33.3% of ORQA, the earlier dense retriever that needed an expensive extra pre-training task.

2 Background: retrieve, then read · original

Everyday picture

An open-book exam with a library of 21 million pages. You cannot read them all per question, so a retriever fetches a handful and a reader studies those closely to extract the exact answer. If the retriever misses the right page, the best reader in the world cannot help.

In words: “a retriever takes a question and the whole collection and returns a small set of k passages, far fewer than the collection holds.”

With the numbers: the collection has 21,015,324 passages; k is typically 20 to 100. Even k = 100 hands back one passage for every 210,153 in the collection.

In Python:

# |C|, passages in the collection
C = 21_015_324
# |C_F|, passages handed back
k = 100
# k ≪ |C|: one kept for every this many
C // k  # → 210153

How a retriever is graded: top-k accuracy

Top-k retrieval accuracy is the share of questions for which at least one of the k retrieved passages contains the answer. Tiny example: for 4 questions, if the answer shows up in the top 20 for questions 1, 2 and 4 but not 3, top-20 accuracy is 3/4 = 75%. It is a form of recall@k, and it matters most because the reader can only use what the retriever returns.

3 Dense Passage Retriever · original

3.1 Overview · original

Everyday picture

Two translators who share a secret coordinate system. One reads questions and writes down coordinates; the other reads passages and writes down coordinates. They are trained together so that a question lands right next to the passage that answers it.

Reading it: the dashed box on the left happens once, before any question arrives: every passage goes through the passage encoder and its vector is stored in an index. On the right, a question goes through a different encoder, and its vector is used to find the stored passage vectors with the highest dot product, which go to the reader. The two sides only meet at the dot product, which is what makes it possible to precompute the left side. A cross-attention model, which reads question and passage together, would be more expressive but could not be precomputed.

In words: “the similarity of a question and a passage is the dot product of their two vectors.”

With the numbers (illustrative, 4 dimensions): question (2.0, 0.1, 0.0, 0.2) and passage (1.8, 0.2, 0.1, 0.0) give 3.6 + 0.02 + 0 + 0 = 3.62.

In Python:

# E_Q(q), the question's vector
E_Q = [2.0, 0.1, 0.0, 0.2]
# E_P(p), the passage's vector
E_P = [1.8, 0.2, 0.1, 0.0]
# E_Q(q)ᵀ E_P(p)
sim = sum(q_i * p_i for q_i, p_i in zip(E_Q, E_P))
round(sim, 2)  # → 3.62

Each encoder is BERT-base (uncased) and the vector is BERT's output for its [CLS] token, so d = 768. The paper tried cosine and Euclidean distance too: Euclidean performed comparably to the dot product and both beat cosine, so the simpler dot product was kept.

3.2 Training · original

Everyday picture

A matching game. Show the model a question, its correct passage and several wrong passages, and reward it for giving the correct passage the highest score. That is a multiple-choice question where the options are passages, graded with a softmax and cross-entropy.

In words: “turn the scores of the right passage and the n wrong ones into probabilities with a softmax, and pay minus the log of the right passage's probability.”

With the numbers (illustrative): the right passage scores 3.62 and three wrong ones 0.21, 0.24 and 0.39. Then e^3.62 = 37.34 out of a total 37.34 + 1.23 + 1.27 + 1.48 = 41.32, a share of 0.904 and a loss of −ln 0.904 = 0.10. Add one hard wrong passage scoring 3.26 (e^3.26 = 26.05): the share falls to 37.34 / 67.37 = 0.554 and the loss jumps to 0.59. The hard one is where the learning is.

In Python:

import math
# sim(q_i, p_i+), the right passage
pos = 3.62
# sim(q_i, p_ij−), the wrong ones
neg = [0.21, 0.24, 0.39]
# the softmax's denominator
total = math.exp(pos) + sum(math.exp(s) for s in neg)
round(math.exp(pos), 2), round(total, 2)  # → (37.34, 41.32)
share = math.exp(pos) / total
# the right passage's share, and L
round(share, 3), round(-math.log(share), 2)  # → (0.904, 0.1)
# one hard wrong passage joins
neg.append(3.26)
total = math.exp(pos) + sum(math.exp(s) for s in neg)
share = math.exp(pos) / total
round(total, 2), round(share, 3), round(-math.log(share), 2)  # → (67.37, 0.554, 0.59)

Which wrong passages? Three kinds

• Random: any passage from the collection. Easy to reject; teaches little.
• BM25: top keyword matches for the question that do not contain the answer. These are hard negatives: same words, wrong answer.
• Gold: correct passages of other questions in the training set.

The best model uses the gold passages of the other questions in the same mini-batch, plus one BM25 negative per question.

In-batch negatives: the free lunch

“In this way, we reuse computation and effectively train on B² (q_i, p_j) question/passage pairs in each batch.”Karpukhin et al. (2020), §3.2

Everyday picture

At a speed-dating event with B couples, each person is supposed to pick their own partner out of the whole room. You do not need to invite extra strangers to make it hard: everyone else's partner is already a wrong answer. A batch of B questions with their B passages gives every question B − 1 wrong passages at no extra encoding cost.

In words: “stack the batch's question vectors and passage vectors as rows, and one matrix multiplication scores every question against every passage.”

With the numbers: the paper's batch size B = 128 gives a 128 × 128 score matrix: 16,384 pairs, of which 128 are correct and 127 wrong passages face each question.

In Python:

# questions (and their right passages) in a batch
B = 128
# entries of S = Q Pᵀ, one per question-passage pair
B * B  # → 16384
# wrong passages each question is scored against
B - 1  # → 127

Reading it: rows are questions and columns are passages; each cell is the dot-product score, and darker blue means higher. The green-outlined diagonal holds each question's correct passage; every other cell in a row is a free wrong answer. The right-hand column shows each row's loss (the −log of the correct passage's softmax share). With in-batch negatives only, every loss is small: the wrong passages are about other topics, so they are easy to reject. Switch on the BM25 hard negatives (orange outlines): two extra columns that share keywords with questions 1 and 2 but do not answer them. Those rows' losses jump, because the model now has something difficult to learn. As in the paper, the hard negatives are shared as extra columns for every question in the batch.

Why it matters today

In-batch negatives plus mined hard negatives is still how most text embedding models are trained, and it is the same loss (often called InfoNCE) that CLIP uses for images and captions. The contrastive training lesson builds it and measures the effect of hard negatives.

4 Experimental setup · original

4.1 The library

The English Wikipedia dump of 20 December 2018, cleaned of tables, infoboxes, lists and disambiguation pages, then cut into disjoint blocks of 100 words: 21,015,324 passages. Each passage is prefixed with its article title. This is a simple form of chunking; the paper found fixed-length passages worked better than natural paragraphs, and overlapping passages did not help.

4.2 The questions

Five datasets: Natural Questions (real Google search queries), TriviaQA (trivia questions), WebQuestions (Google Suggest questions answered from Freebase), CuratedTREC (TREC QA and web questions) and SQuAD v1.1 (questions written while looking at a paragraph). Where a dataset gives only an answer, the positive passage is the highest-ranked BM25 passage that contains it.

5 Passage retrieval results · original

Training settings: batch size 128 with one extra BM25 negative per question, up to 40 epochs on large datasets and 100 on small ones, Adam with learning rate 10⁻⁵, linear schedule with warm-up, and dropout 0.1. They also try a hybrid: take the top 2,000 passages from each of BM25 and DPR and rerank the union by a weighted sum.

In words: “add the keyword score to the dense score, with the dense score scaled by a weight chosen on development data.”

With the numbers (illustrative scores): with λ = 1.1, a passage with BM25 = 20 and sim = 60 scores 20 + 66 = 86, beating one with BM25 = 30 and sim = 50 (30 + 55 = 85).

In Python:

# λ
lam = 1.1
# BM25(q, p) + λ · sim(q, p), first passage
round(20 + lam * 60)  # → 86
# second passage
round(30 + lam * 50)  # → 85

Reading it: three candidate passages with illustrative scores. At λ = 0 only BM25 counts, so the keyword-heavy passage wins; as λ grows the dense score takes over and the semantically matching passage rises. The paper's λ = 1.1 was chosen on development data, where the two scales happened to balance. Choosing λ is fiddly because the two scores live on different scales, which is why many systems today merge the two rankings with reciprocal rank fusion instead of adding raw scores (see the retrieval lesson).

5.1 Main results · original

Reading it: for each dataset, the grey bar is BM25's top-20 accuracy and the blue bar is DPR's (trained on that dataset alone). DPR wins everywhere except SQuAD, most of all on Natural Questions: 78.4% against 59.1%. The paper notes the gap is especially large at small k, when only a few passages can be returned. The paper gives two reasons for the SQuAD exception: its questions were written while looking at the passage, so they copy its words (which favours BM25), and it covers only about 500 Wikipedia articles, so the training data is narrow. Selected values from Table 2 of Karpukhin et al. (2020).

5.2 What mattered in training · original

Sample efficiency

With only 1,000 training questions, DPR already beats BM25 on Natural Questions; more training data (up to 59,000 questions) keeps improving it. A pre-trained BERT needs surprisingly few labelled pairs to become a good retriever.

Negatives and batches

Reading it: each row is one training recipe; the bar is top-5 accuracy on the Natural Questions development set, and the right column adds top-20 and top-100. With 7 negatives per question and no in-batch trick, the kind of negative barely matters. Taking the 7 negatives from the same batch (the “in-batch” rows) helps immediately, and growing the batch so each question sees 127 in-batch negatives helps more. The biggest single jump comes from adding just one BM25 hard negative per question: top-5 accuracy goes from 55.8% to 65.8%. Adding a second BM25 negative did not help further. Selected rows of Table 3 of Karpukhin et al. (2020).

Other findings

• Similarity and loss: Euclidean distance performed comparably to the dot product and both beat cosine; a triplet loss gave similar results to the softmax loss.
• Generalisation: trained on Natural Questions only and tested directly on WebQuestions and CuratedTREC, DPR loses 3 to 5 points of top-20 accuracy against models trained on those datasets, but still beats BM25 by a wide margin.

5.3 Where each method wins · original

Everyday picture

The keyword librarian and the meaning librarian fail in opposite ways. Ask for “the body of water between England and Ireland”: the keyword librarian brings a page about British cycling that mentions England and Ireland many times, while the meaning librarian brings the page on the Irish Sea. Ask “who plays Thoros of Myr in Game of Thrones?”: the rare name “Thoros of Myr” is decisive, and the keyword librarian finds it while the meaning librarian drifts to a different actor. Both examples are from the paper's appendix.

Why it matters today

This is the whole argument for hybrid search in enterprise systems full of product codes, error numbers and names. The primer's retrieval lesson reproduces both failure modes on a toy corpus, and fixes them with fusion.

5.4 Run-time efficiency · original

Reading it: the top two cards are query speed; the bottom two are one-off indexing costs. Dense retrieval is fast to query once the index exists, because an approximate nearest-neighbour index (the paper used FAISS's HNSW with 512 neighbours per node) touches only a tiny fraction of the 21 million vectors. It is slow to build: every passage must go through BERT, and the graph must be constructed. A keyword index builds in half an hour. Figures from §5.4 of Karpukhin et al. (2020).

Storage, our arithmetic: 21,015,324 passages × 768 numbers × 4 bytes = about 64.6 GB of raw vectors before any index overhead, one reason compression (see the compression lesson) matters so much in practice.

6 End-to-end question answering · original

Everyday picture

Now the reader: a BERT model that looks at the top k passages, decides which passage is most likely to hold the answer, and highlights the answer span inside it. Unlike the retriever, the reader does read question and passage together (cross-attention), which is affordable because it only sees k passages.

In words: “score every token of passage i as a possible answer start with one learned vector, and turn the scores into probabilities.” An identical formula with another vector scores the end token, and a third vector applied to each passage's [CLS] token picks the passage.

With the numbers (illustrative): if the tokens of “... the villain Sauron in ...” get start scores (0.1, 0.3, 2.5, 0.2), softmax puts about 77% on “Sauron”, which becomes the answer if the end scores agree.

In Python:

import math
# P_i w_start: one start score per token
scores = [0.1, 0.3, 2.5, 0.2]
total = sum(math.exp(s) for s in scores)
# softmax(...)_s
P_start = [math.exp(s) / total for s in scores]
# the token "Sauron"
round(P_start[2], 2)  # → 0.77

Results

On Natural Questions, the DPR-based system reaches 41.5% exact-match accuracy, against 33.3% for ORQA and 32.6% for the same reader on BM25 passages; it beats the concurrently developed REALM (40.4%) without REALM's expensive extra pre-training. Better retrieval turned into better answers on every dataset except SQuAD. Reading k = 50 passages was best on Natural Questions, and k = 10 cost less than one point (40.8%).

7 to 8 Related work and conclusion · original

The conclusion is deliberately modest: dense retrieval can outperform, and potentially replace, sparse retrieval for open-domain QA, and more complex frameworks or similarity functions did not add value. The related-work section notes the concurrent ColBERT, which keeps one vector per token instead of one per passage (its own companion page walks through that alternative), and follow-up work combining DPR with generators, which became retrieval-augmented generation.

What changed since 2020

Glossary

Every term with hover guidance on this page, in one place.