word2vec, annotated

How to read this page

Nothing here assumes you already know the jargon. Any dotted word explains itself when you hover it, tab to it, or tap it. Every symbol inside an equation does the same. The diagrams and demos are live: hover, tap, drag and pick.

Each idea climbs the same ladder: an everyday picture, a tiny example you could check by hand, a diagram or demo, then the math, and finally why it still matters. The word embeddings lesson trains skip-gram with negative sampling from scratch in NumPy, so you can watch the ideas on this page actually run.

Paper 1: Efficient Estimation of Word Representations in Vector Space

Mikolov, Chen, Corrado and Dean (2013). arXiv:1301.3781. The paper that introduced the two word2vec architectures, CBOW and skip-gram.

Abstract · original

“We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set.”Mikolov, Chen, Corrado and Dean (2013), Abstract

Everyday picture

Imagine giving every word a home address on a map, so that words with similar meanings live on the same street. Before 2013, building such a map with a neural network took weeks of computing for a few hundred million words. This paper strips the network down to almost nothing and finds that the map gets better, because the simpler model can afford to read ten times more text.

What the paper claims

• Two new, very cheap architectures for learning word embeddings: CBOW (guess a word from its neighbours) and skip-gram (guess the neighbours from a word).
• High-quality vectors from 1.6 billion words in under a day.
• A new test of 8,869 semantic and 10,675 syntactic analogy questions, on which the vectors set new records.

Why it matters today

“Represent a thing as a list of numbers learned from the company it keeps” is the idea underneath every modern embedding model, search engine and chat model. word2vec is where most practitioners first saw it work at scale.

1 Introduction · original

“Many current NLP systems and techniques treat words as atomic units - there is no notion of similarity between words, as these are represented as indices in a vocabulary.”Mikolov et al. (2013a), §1

Everyday picture

A dictionary index is an atomic representation: “cat” is entry 17 and “dog” is entry 18, and the numbers 17 and 18 say nothing about both being pets. A map is a distributed representation: each word is a point, described by several coordinates at once, and nearby points mean related words.

Tiny example

With a four-word vocabulary (car, cat, dog, sky), the atomic way writes “cat” as the one-hot vector (0, 1, 0, 0) and “dog” as (0, 0, 1, 0). Their dot product is 0·0 + 1·0 + 0·1 + 0·0 = 0: as unrelated as “cat” and “sky”. Now give each word two learned numbers instead, say cat = (0.8, 0.3), dog = (0.7, 0.4) and sky = (−0.2, 0.9). The cosine similarity of cat and dog is (0.56 + 0.12) / (0.854 × 0.806) = 0.99, while cat and sky get (−0.16 + 0.27) / (0.854 × 0.922) = 0.14. The numbers now carry meaning.

The famous observation (§1.1)

Earlier work by the same group had noticed that differences between word vectors capture relationships: vector(“King”) − vector(“Man”) + vector(“Woman”) lands closest to vector(“Queen”). This paper's goal is to build architectures that preserve such linear regularities and to measure them properly. Try the relationship yourself in the word map in section 5.

Why it matters today

The switch from “words are IDs” to “words are points in space” is the single idea that the rest of the similarity lesson and every vector database builds on.

2 Model architectures, and where the time goes · original

Everyday picture

Before comparing cars, agree on how to measure fuel use. The paper's “fuel use” is the number of parameters the model has to touch for each training word. Multiply that by how many words you read and how many times you reread them, and you have the total cost.

In words: “total training work is the number of passes over the data, times the number of words, times the work per word.”

With the numbers: 3 passes over 1 billion words with Q = 15,000 parameters touched per word gives 3 × 10⁹ × 15,000 = 4.5 × 10¹³ parameter touches.

In Python:

# passes, words, parameters touched per word
E, T, Q = 3, 10**9, 15_000
# O = E × T × Q
O = E * T * Q
print(f"{O:.1e}")  # → 4.5e+13

2.1 The feed-forward neural network language model (NNLM)

The older model reads the previous N words, looks each one up in a table of D numbers, feeds all N × D numbers into a hidden layer of H neurons, and finally scores every one of the V words in the vocabulary. Its cost per word is:

In words: “look up N word vectors, push all of them through the hidden layer, then score every word in the vocabulary.”

With the numbers: the paper's typical sizes, N = 10, D = 500, H = 500 and V = 1,000,000, give 5,000 + 2,500,000 + 500,000,000. Scoring the whole vocabulary (H × V) is 99.5% of the work.

In Python:

N, D, H, V = 10, 500, 500, 1_000_000
# look-ups, hidden layer, scoring the vocabulary
N * D, N * D * H, H * V  # → (5000, 2500000, 500000000)
Q = N * D + N * D * H + H * V
# % of the work spent on H × V
round(100 * H * V / Q, 1)  # → 99.5

The standard escape is a hierarchical softmax, which replaces the V scores with about log₂(V) ≈ 20 yes-or-no decisions. The paper uses a Huffman tree, which gives frequent words even shorter paths. Once that is done, the hidden layer (N × D × H) becomes the bottleneck, and that is exactly what word2vec deletes.

2.2 The recurrent language model (RNNLM)

A recurrent network keeps a hidden state of H numbers and updates it word by word, so Q = H × H + H × V, or H × H + H × log₂(V) with the tree. The H × H matrix, applied at every word, now dominates.

Try it: where does the work go?

Reading it: each bar is one architecture's work per training word, Q, on a log scale, so each extra digit in the number adds the same length to the bar. All models use the hierarchical softmax except the top bar, which shows the NNLM scoring the full vocabulary. Drag the vocabulary size and watch the top bar grow with V while the others barely move, because log₂(V) grows so slowly. Drag the hidden-layer size: NNLM and RNNLM grow, but CBOW and skip-gram do not move at all, because they have no hidden layer. That gap, a factor of 100 or more, is what lets word2vec read ten times more text in the same time.

2.3 Parallel training

The models ran on DistBelief, Google's system for training many copies of a model at once, each sending its gradient updates to a central parameter server, with the Adagrad optimizer. Typical runs used 50 to 100 replicas.

Why it matters today

“Count the work per example before you design the model” is still how efficient systems are built. The same arithmetic, applied to attention instead, explains why long contexts are expensive (see the attention lesson).

3 New log-linear models: CBOW and skip-gram · original

“The main observation from the previous section was that most of the complexity is caused by the non-linear hidden layer in the model.”Mikolov et al. (2013a), §3

Everyday picture

Two party games. In CBOW (continuous bag of words) you see the words around a gap and guess the missing word: “the cat ___ on the mat”. In skip-gram you are shown one word and must guess which words are likely nearby: given “sat”, guess “cat”, “on”, “mat”. Words that play the same games end up with similar vectors, because they have to make the same guesses.

Tiny example

Take the sentence “the quick brown fox jumps” and the centre word “brown” with a window of 2. CBOW averages the vectors of the, quick, fox, jumps and tries to predict brown: one prediction. Skip-gram makes four predictions from brown: (brown → the), (brown → quick), (brown → fox), (brown → jumps). “Log-linear” means there is no hidden layer at all: the only operations are a table lookup, an average or a copy, and a softmax (or its hierarchical version).

Reading it: read the left diagram left to right. Four context words are looked up (pink), their vectors are averaged into one (yellow), and that single vector is used to predict the middle word (green). The right diagram is the mirror image: one word is looked up and its vector is used to predict each of the surrounding words separately. Neither side has a hidden layer, so each training step touches only a handful of vectors. Word order inside the window is ignored by CBOW: the averaging throws it away, which is why it is called a bag of words.

How skip-gram picks its training pairs

Nearby words are usually more related than distant ones. Skip-gram handles this with a neat trick: for every centre word it draws a random window size R between 1 and C (the maximum distance, C = 10 in the paper) and uses the R words on each side. A word at distance 1 is always used; a word at distance C is used only when R = C. The paper's cost for skip-gram is:

In words: “for each of up to C context predictions, touch the input vector and walk a path of log₂(V) tree nodes, each with its own D numbers.”

With the numbers: C = 10, D = 500, V = 1,000,000: 10 × (500 + 500 × 19.93) ≈ 104,700 parameters per word, against about 2.5 million for the NNLM.

In Python:

import math
C, D, V = 10, 500, 1_000_000
# nodes on a path through the tree
round(math.log2(V), 2)  # → 19.93
# Q = C × (D + D × log2 V)
Q = C * (D + D * math.log2(V))
round(Q, -2)  # → 104700.0

Reading it: the orange word is the centre word and the blue words are the context used this time; greyed words are outside the window drawn for this step. Press “Draw a new R” a few times: the window shrinks and grows at random. The bars underneath show, for each distance, how often a word at that distance is used over many draws: (C − d + 1) / C. With C = 5, the neighbour at distance 1 is used every time and the word at distance 5 only one time in five. The model therefore spends most of its effort on the closest, most informative neighbours without any explicit weighting.

Why it matters today

Skip-gram is the direct ancestor of contrastive training: “pull a word towards the words it appears with, push it away from random words”. The same shape appears in Sentence-BERT, DPR and CLIP. The word embeddings lesson implements it.

4 Results · original

Everyday picture

How do you grade a map of meaning? Ask it riddles: “big is to biggest as small is to what?” If the map is good, walking from “big” to “biggest” and then taking the same step from “small” should land on “smallest”.

4.1 The test

The authors wrote 8,869 semantic questions (capital cities, currencies, man and woman pairs such as brother and sister) and 10,675 syntactic ones (plurals, past tenses, opposites such as ethical and unethical). A question counts as correct only if the single closest word is exactly the right one: synonyms count as wrong, so 100% is probably impossible.

In words: “take the step from big to bigger, apply it starting from small, and answer with the word whose vector points most nearly the same way as where you land, ignoring the three question words.”

With the numbers: in the illustrative map in section 5, big = (−2, −4) and bigger = (−1, −2.8), so the step is (1, 1.2). From small = (1, −5) the same step lands on (2, −3.8), which is exactly where “smaller” sits.

In Python:

import math
vec = {"big": (-2, -4), "bigger": (-1, -2.8), "small": (1, -5),
       "smaller": (2, -3.8), "cold": (3.5, -5.5), "colder": (4.5, -4.3)}
# bigger − big
step = [b - a for a, b in zip(vec["big"], vec["bigger"])]
[round(s, 1) for s in step]  # → [1, 1.2]
# ... + small
X = [s + c for s, c in zip(step, vec["small"])]
[round(x, 1) for x in X]  # → [2, -3.8]
def cos(a, b):
    return (a[0] * b[0] + a[1] * b[1]) / (math.hypot(*a) * math.hypot(*b))
# arg max over w
max(["smaller", "cold", "colder"], key=lambda w: cos(X, vec[w]))  # → 'smaller'

4.2 More data and more dimensions, together

Training CBOW on growing slices of a 6-billion-word Google News corpus (vocabulary limited to the 30,000 most common words for this test) showed diminishing returns: more dimensions help only if there is also more data, and vice versa.

Reading it: the x-axis is the number of training words (24 million to 783 million, log scale) and the y-axis is accuracy on the analogy questions. The 50-dimension line flattens out around 23%: it cannot store more even when given more text. The 600-dimension line keeps climbing, to 50.4%. At the smallest data size the two differ by only 10 points; at the largest by 27. Size the vectors to the data. Selected values from Table 2 of Mikolov et al. (2013a), CBOW, 3 training epochs.

4.3 Comparing architectures on the same data

All four models were trained on the same 320 million words with 640-dimensional vectors:

Reading it: each model has two bars, semantic accuracy (blue, solid) and syntactic accuracy (orange, striped). The recurrent model is good at grammar but poor at meaning (9% semantic). CBOW is the best at syntax (64%). Skip-gram is slightly behind CBOW on syntax but far ahead on meaning (55% semantic against 24%). If you want capital cities and currencies, predict the context from the word. Key numbers from Table 3 of Mikolov et al. (2013a).

4.4 Scaled up

On the full 6-billion-word corpus with DistBelief, 1000-dimensional skip-gram vectors reached 65.6% overall in 2.5 days on about 125 CPU cores, and CBOW 63.7% in 2 days on about 140 cores, against 50.8% for a 100-dimensional NNLM that needed 14 days on about 180 cores (Table 6). Training on twice the data for one epoch matched or beat three epochs on the same data (Table 5).

4.5 Sentence completion

On the Microsoft Research Sentence Completion Challenge (1,040 sentences with one missing word and five choices), skip-gram alone scored 48.0%, below older methods, but combined with recurrent language models it set a new record of 58.9%, up from 55.4%.

Why it matters today

Analogy tests are rarely used now, because modern embeddings are judged on retrieval and classification benchmarks. But the lesson of section 4.2, that model size and data size must grow together, reappeared a decade later as the scaling laws for language models.

5 Examples of the learned relationships · original

“Thus for example, Paris - France + Italy = Rome.”Mikolov et al. (2013a), §5

Everyday picture

On a street map, “from the station to the museum” is a direction and a distance. If every city was laid out the same way, you could take that same walk from any station and arrive at its museum. In a good word map, “from country to capital” is such a walk.

Try it: vector arithmetic on a word map

Reading it: the solid orange arrow goes from B to A: that is the relationship, for example “from man to king”. The dashed red arrow is the same arrow moved so that it starts at C. Its tip is X = A − B + C, and the red dot is the nearest word to X, with A, B and C themselves excluded as in the paper. Try the presets, then make your own. Relationships that exist in the map (gender, royalty, country to capital, adjective to comparative) work; mixing clusters, such as king − man + Paris, lands in empty space and returns whatever happens to be closest, which is exactly how real embeddings fail too.

What the paper reports

Table 8 lists relationships the 300-dimensional skip-gram vectors got right, from chemistry (copper to Cu gives zinc to Zn) to company products, and some they got wrong (the step from Einstein to scientist, applied to Messi and Mozart, gives midfielder and violinist). By the exact-match rule the examples would score only about 60%. Averaging the relationship over ten example pairs instead of one improved accuracy by about 10 points absolute. The same arithmetic also solves “which word does not belong?” by finding the word farthest from the average of a list.

Why it matters today

That meaning becomes geometry, so that directions in the space encode relationships, is what makes embeddings searchable, clusterable and comparable. The word embeddings lesson reproduces an analogy from a model trained from scratch.

6 to 7 Conclusion and follow-up · original

The conclusion: very simple models trained on much more data beat complex models trained on less. The authors expect CBOW and skip-gram to scale to a trillion words. The follow-up note announces the open-source C code (training on billions of words per hour on one machine) and 1.4 million pre-trained vectors for named entities built from more than 100 billion words. That release, more than the paper, is why “word2vec” became a household name in machine learning.

Paper 2: Distributed Representations of Words and Phrases and their Compositionality

Mikolov, Sutskever, Chen, Corrado and Dean (2013), NIPS. arXiv:1310.4546. The paper that made skip-gram fast and good, with negative sampling, subsampling and phrases.

1 Introduction · original

“For example, the result of a vector calculation vec(“Madrid”) - vec(“Spain”) + vec(“France”) is closer to vec(“Paris”) than to any other word vector.”Mikolov et al. (2013b), §1

Everyday picture

Paper 1 built a fast car. Paper 2 tunes the engine: skip the words that tell you nothing (“the”, “a”), replace an expensive exam with a cheap true-or-false quiz, and treat “New York Times” as one word instead of three.

The three extensions

• Subsampling frequent words: 2 to 10 times faster, and better vectors for rare words.
• Negative sampling: a simplified version of noise contrastive estimation that replaces the (hierarchical) softmax.
• Phrases: find multi-word expressions such as “Boston Globe” (a newspaper, not a globe in Boston) and give each its own vector.

One headline number: an optimised single-machine implementation trains on more than 100 billion words in a day.

2 The skip-gram model · original

Everyday picture

Each word carries two cards: an input card, used when it is the centre word, and an output card, used when it is being guessed as a neighbour. Training nudges the cards so that a word's input card points the same way as the output cards of the words it really appears next to.

The training goal

In words: “for every position in the text, add up how confidently the model predicts each word within c places of it, and average over the text; training makes this as large as possible.”

With the numbers: at the centre word “brown” in “the quick brown fox jumps” with c = 2, the four terms are log p(the | brown) + log p(quick | brown) + log p(fox | brown) + log p(jumps | brown). If each probability is 0.01, the sum is 4 × ln 0.01 = −18.4, and training pushes it towards 0.

In Python:

import math
# within c = 2 of "brown"
context = ["the", "quick", "fox", "jumps"]
# p(w_(t+j) | brown)
p = {w: 0.01 for w in context}
round(sum(math.log(p[w]) for w in context), 1)  # → -18.4

The expensive part: the full softmax

In words: “score the guess by the dot product of the centre word's input vector with the guessed word's output vector, and turn the scores into probabilities with a softmax over the entire vocabulary.”

With the numbers: with a toy vocabulary of three words whose output vectors score 2.0, 1.0 and 0.5 against the input vector, p = e² / (e² + e¹ + e^0.5) = 7.39 / 11.76 = 0.63. With a real vocabulary, W is 10⁵ to 10⁷, so the bottom of that fraction costs up to ten million dot products for every single training pair. The next two sections are two ways around it.

In Python:

import math
# v'_wᵀ v_(w_I) for each of the W = 3 words
scores = [2.0, 1.0, 0.5]
exps = [math.exp(s) for s in scores]
round(exps[0], 2), round(sum(exps), 2)  # → (7.39, 11.76)
# p(w_O | w_I) for the first word
round(exps[0] / sum(exps), 2)  # → 0.63

Reading it: each country (left) is joined by an arrow to its capital (right). The arrows are nearly parallel and nearly the same length: “capital of” is roughly one direction in the space. The paper's point is that nobody told the model what a capital is. It discovered the relationship purely from which words appear near which. The positions here are illustrative; the paper's own figure is a PCA projection of real 1000-dimensional skip-gram vectors.

2.1 Hierarchical softmax · original

Everyday picture

Twenty questions. Instead of scoring all million words at once, play a game of yes-or-no: “is it in the left half of the vocabulary?”, then “the left half of that?”, and so on. After about 20 questions you have narrowed a million words down to one. Each question is a coin flip whose bias the model learns.

Tiny example

Eight words sit at the leaves of a three-level tree. To reach “cat” the path goes left, right, left. If, for the current centre word, the three branch probabilities along that path are 0.6, 0.7 and 0.5, then p(cat) = 0.6 × 0.7 × 0.5 = 0.21, from three sigmoid evaluations instead of eight (or a million) dot products.

Reading it: each inner circle is a learned yes-or-no question; the number beside each branch is the probability of taking it, for one particular centre word (the values are illustrative). Click a leaf: its path lights up and its probability is the product of the numbers along the path. The line below the tree adds up all eight leaf probabilities: it is always exactly 1, whatever you set the root slider to, because every internal question splits its probability between its two children. That is why the tree is a proper probability distribution without ever summing over the whole vocabulary. The second line compares path lengths: a Huffman tree built from word counts puts frequent words near the root.

In words: “walk from the root to the word; at each inner node, compute a sigmoid of that node's vector dotted with the centre word's vector, flipping the sign when the path goes the other way; multiply the results.”

With the numbers: for the path to “cat”, three factors 0.6 × 0.7 × 0.5 = 0.21. With a million words, a balanced tree needs log₂(10⁶) ≈ 20 factors, and a Huffman tree fewer for common words.

In Python:

import math
# σ(± v'ᵀ v_(w_I)) at each node on the path to "cat"
factors = [0.6, 0.7, 0.5]
# Π over the path
round(math.prod(factors), 2)  # → 0.21
# factors on a balanced tree over a million words
round(math.log2(10**6))  # → 20

Why it matters today

Hierarchical softmax is rarely used in modern language models, which can afford the full softmax on GPUs. The idea of replacing one huge choice with a short chain of small ones lives on in tree-based indexes and in the coarse-then-fine search of HNSW.

2.2 Negative sampling · original

“Thus the task is to distinguish the target word w_O from draws from the noise distribution P_n(w) using logistic regression, where there are k negative samples for each data sample.”Mikolov et al. (2013b), §2.2

Everyday picture

Replace the multiple-choice exam with a million options by a quick true-or-false quiz. Show the model the real pair (sat, cat) and ask “did these appear together?”: the answer should be yes. Then show it a few made-up pairs, (sat, galaxy), (sat, invoice), and it should say no. Five or so fake pairs per real one are enough to teach it the difference.

Tiny example

Suppose the real pair scores 2.0 (the dot product of the two vectors) and two fake pairs score 1.5 and −0.5. The quiz rewards a high chance of “yes” on the real pair and of “no” on the fakes, using the sigmoid σ:

• real pair: log σ(2.0) = log 0.881 = −0.127 (good: already confident);
• fake pair 1: log σ(−1.5) = log 0.182 = −1.701 (bad: the fake looks real);
• fake pair 2: log σ(0.5) = log 0.622 = −0.474.

The total, −2.302, is what training pushes up. Most of the push goes to fake pair 1, the most convincing fake.

In words: “make the real pair's score high and each of the k random pairs' scores low, each judged through a sigmoid, and replace every log-probability term in the skip-gram goal with this.”

With the numbers: −0.127 + (−1.701) + (−0.474) = −2.302 for the example above. The cost is now k + 1 dot products (3 here) instead of W (a million).

In Python:

import math
def sigma(x):
    return 1 / (1 + math.exp(-x))
# log σ(v'_(w_O)ᵀ v_(w_I))
real = math.log(sigma(2.0))
# log σ(−v'_(w_i)ᵀ v_(w_I)), k = 2
fakes = [math.log(sigma(-s)) for s in (1.5, -0.5)]
round(real, 3), [round(f, 3) for f in fakes]  # → (-0.127, [-1.701, -0.474])
round(real + sum(fakes), 3)  # → -2.302

Reading it: each bar is one term of the objective, drawn as its size (longer means worse, since every term is a negative log). The number on the right of each row is the push that training gives that pair's score: +(1 − σ(s)) upward for the real pair, −σ(s) downward for each fake. Drag the real score up: its bar shrinks and its push fades, since there is nothing left to learn. Drag a fake's score up: its bar grows and its downward push approaches −1. Training effort flows automatically to the pairs the model currently gets most wrong, the same property that makes hard negatives so valuable in DPR.

Which fake words? The noise distribution

Fake words are drawn at random, but not uniformly. The paper found that drawing each word with probability proportional to its count raised to the power 3/4 works much better than either plain counts or a uniform draw.

In words: “flatten the word frequencies a little by raising them to the power 0.75, then rescale so they add to 1.”

With the numbers: for four words with frequencies 0.9, 0.08, 0.018 and 0.002, the powers are 0.924, 0.150, 0.049 and 0.0095, which sum to Z = 1.133. The rare word's share rises from 0.2% to 0.83%, about four times more often, while the common word falls from 90% to 81.6%.

In Python:

# word frequencies U(w)
U = [0.9, 0.08, 0.018, 0.002]
# U(w)^(3/4)
powered = [u ** 0.75 for u in U]
[round(x, 3) for x in powered[:3]], round(powered[3], 4)  # → ([0.924, 0.15, 0.049], 0.0095)
Z = sum(powered)
round(Z, 3)  # → 1.133
# P_n(w) = U(w)^(3/4) / Z
P_n = [x / Z for x in powered]
# % shares: most common, rarest
round(100 * P_n[0], 1), round(100 * P_n[3], 2)  # → (81.6, 0.83)

Reading it: drag the exponent. At 1 the shares are the raw frequencies, so almost every fake word would be “the”. At 0 every word is equally likely, so obscure words would dominate the fakes. At the paper's 0.75, common words are still drawn most often, but rare words get noticeably more practice. The rightmost column shows how many times more (or less) often each word is drawn than its raw frequency.

The paper also reports that k = 5 to 20 negatives suit small datasets, while 2 to 5 are enough for large ones. Unlike noise contrastive estimation, negative sampling needs only samples from the noise distribution, not its probabilities, and does not try to recover proper probabilities: it only cares about good vectors.

Why it matters today

This is the template for almost every embedding model since: score a true pair against a handful of false ones. DPR reuses the other questions' answers in a batch as the false ones (in-batch negatives), and CLIP does the same with images and captions. The losses lesson and the contrastive training lesson build this family of losses.

2.3 Subsampling of frequent words · original

Everyday picture

If you learn about France by reading which words appear near it, “Paris” teaches you a lot and “the” teaches you nothing, because “the” is next to everything. So randomly skip most occurrences of the most common words: you lose almost no information and save a great deal of time.

In words: “throw away each occurrence of a word with a probability that grows with how common the word is; words rarer than the threshold t are always kept.”

With the numbers: with t = 10⁻⁵, a word making up 5% of the text (f = 0.05) is discarded with probability 1 − √(0.0002) = 1 − 0.014 = 0.986, so only 1.4% of its occurrences survive. A word with f = 10⁻⁴ is discarded with probability 1 − 0.316 = 0.684. A word with f = 10⁻⁵ or rarer is never discarded.

In Python:

import math
t = 1e-5
def P(f):
    # P(w_i) = 1 − √(t / f(w_i))
    return 1 - math.sqrt(t / f)
round(math.sqrt(t / 0.05), 3), round(P(0.05), 3)  # → (0.014, 0.986)
round(P(1e-4), 3)  # → 0.684
P(1e-5)  # → 0.0

Reading it: the x-axis is a word's frequency on a log scale, from one in ten million to one in ten; the y-axis is the chance each occurrence is thrown away. Left of the threshold t the curve sits at 0: rare words are always kept. Right of it, the curve rises steeply towards 1, so the most common words lose almost all their occurrences. Slide t: a larger threshold moves the knee right, so fewer words get thinned. The paper chose the formula by hand; it keeps the frequency ranking intact while aggressively trimming the top.

The effect in the paper's experiments: training 2 to 10 times faster, and more accurate vectors for rare words, because they now make up a larger share of the training pairs.

3 Empirical results · original

Everyday picture

A bake-off: the same one billion words of news, the same 300-dimensional skip-gram, and four ways of handling the output: hierarchical softmax, noise contrastive estimation, and negative sampling with 5 or 15 fakes. Then the same again with subsampling switched on.

Reading it: each row is one training method; the bar is its total accuracy on the analogy test, and the right column shows the training time in minutes. Negative sampling with 15 negatives is the most accurate (61%) and hierarchical softmax the least (47%). Switching on subsampling (the lower block) keeps accuracy about the same or better while cutting training time: NEG-5 drops from 38 to 14 minutes. Key numbers from Table 1 of Mikolov et al. (2013b); vocabulary of 692,000 words (words seen fewer than 5 times were dropped).

4 Learning phrases · original

Everyday picture

“Air Canada” is an airline, not air plus Canada. Some word pairs are really single names. The trick is to spot pairs that appear together far more often than their individual popularity would predict, and glue them into one token before training.

In words: “how often the two words appear side by side, minus a small discount, divided by how often each appears at all.”

With the numbers (illustrative counts): if “new” appears 2,000 times, “york” 900 times and “new york” 800 times, with δ = 5 the score is (800 − 5) / (2,000 × 900) = 4.4 × 10⁻⁴. For “this is”, with “this” 50,000 times, “is” 80,000 times and the pair 4,000 times: (4,000 − 5) / (4 × 10⁹) = 1.0 × 10⁻⁶, over 400 times lower, so “this is” stays two words.

In Python:

# δ, the discount
delta = 5
def score(count_ij, count_i, count_j):
    return (count_ij - delta) / (count_i * count_j)
# "new york"
print(f"{score(800, 2_000, 900):.1e}")  # → 4.4e-04
# "this is"
print(f"{score(4_000, 50_000, 80_000):.1e}")  # → 1.0e-06
round(score(800, 2_000, 900) / score(4_000, 50_000, 80_000))  # → 442

Reading it: each row is a candidate pair with illustrative counts and its score on a log scale. Pairs above the threshold (green) become single tokens such as new_york; pairs below stay separate. Pairs of rare words that happen to meet once or twice are held back by the discount δ, which is subtracted before dividing. The paper runs 2 to 4 passes with a decreasing threshold, so longer phrases such as “New York Times” can form from pieces glued in earlier passes.

4.1 Results

A new test of 3,218 phrase analogies, such as Montreal is to Montreal Canadiens as Toronto is to Toronto Maple Leafs. With 300 dimensions, hierarchical softmax plus subsampling did best (47%), a reversal of the word-level result. The best model, trained on about 33 billion words with 1000 dimensions and the whole sentence as context, reached 72%; the same setup on 6 billion words reached 66%. More data, again, was crucial.

5 Additive compositionality · original

“The product works here as the AND function: words that are assigned high probabilities by both word vectors will have high probability, and the other words will have low probability.”Mikolov et al. (2013b), §5

Everyday picture

Ask two friends for restaurant suggestions: one knows good Italian places, the other knows places near your office. The places both would recommend are the answer. Adding two word vectors does the same: “Russian” + “river” lands near words likely in the context of both, such as “Volga River”.

Tiny example

Why addition acts like AND: a word vector's dot products with output vectors behave like log-probabilities of context words. Adding two vectors adds those logs, which multiplies the probabilities. Say “Russian” gives the context words (Moscow, Volga, Amazon) chances (0.5, 0.4, 0.1) and “river” gives (0.1, 0.45, 0.45). The products are 0.05, 0.18 and 0.045: after rescaling to sum to 1, Volga gets 0.66, far more than either word alone gave it, because only Volga is likely under both.

Examples from the paper's Table 5 (best skip-gram model): Vietnam + capital → Hanoi, German + airlines → Lufthansa, Russian + river → Volga River.

Why it matters today

Averaging word vectors to get a sentence vector, the obvious baseline that Sentence-BERT compares against, rests on this additive property. It works surprisingly well and fails exactly where word order matters: “dog bites man” and “man bites dog” average to the same vector.

6 to 7 Comparison and conclusion · original

Against published vectors from earlier neural models, the big skip-gram phrase model (1000 dimensions, about 30 billion words, trained in about a day) gives visibly better nearest neighbours for rare words: for “Redmond” it returns Redmond Wash., Redmond Washington and Microsoft, where older models returned unrelated names. The older models had been trained for up to two months on two to three orders of magnitude less data. The conclusion names the decisions that matter most: the architecture, the vector size, the subsampling rate and the window size.

What changed since 2013

What survived intact: the distributional idea (meaning from context), dense vectors compared by cosine, and the practice of training on a true pair against sampled false ones.

Glossary

Every term with hover guidance on this page, in one place.