Language Models are Few-Shot Learners (GPT-3), annotated

Zero, one and few shots

“…the model is given a few demonstrations of the task at inference time as conditioning […], but no weight updates are allowed.”Brown et al. (2020), §2, defining few-shot

Everyday picture

Imagine asking a colleague to translate a word. Zero-shot: you just describe the job (“translate English to French: cheese”). One-shot: you describe it and show one worked example. Few-shot: you show several. Fine-tuning is the old way: send them on a course and permanently change what they know.

Reading it: the left column is fine-tuning. Each example triggers a gradient step, thousands of them, and the result is a new copy of the model for that one task. The three columns on the right all feed a single frozen model (the dashed box): the only difference between them is how much text is placed in front of the question. Zero-shot has only a description; one-shot adds one worked example; few-shot adds several (the paper uses 10 to 100, as many as fit). The model simply continues the text, and its continuation is the answer.

Try it: build a prompt

Reading it: the grey box is exactly the text the model would receive. Drag K from 0 up: at K = 0 it is a zero-shot prompt, at 1 one-shot, above that few-shot. The last line always ends where the model must continue, and the expected continuation is shown in the readout. The token count is estimated at about 4 characters per token and compared with GPT-3's 2,048-token context window, which is what capped K in the paper. The example words and numbers are new ones chosen for this page.

The math: what the model is doing

Underneath, GPT-3 does one thing: it gives a probability for the next token given everything before it. The probability of a whole piece of text is the product of those next-token probabilities.

In words: “the chance of a whole text is the chance of its first token, times the chance of the second given the first, and so on to the end.”

With the numbers: if the model gives “the” 0.1, then “cat” 0.3 given “the”, then “sat” 0.5 given “the cat”, the three-token text has probability 0.1 × 0.3 × 0.5 = 0.015. A few-shot prompt is simply x_<t: the demonstrations change which next token looks likely.

In Python:

# P(the), P(cat | the), P(sat | the cat)
P_next = [0.1, 0.3, 0.5]
P_x = 1
# Π over t
for p in P_next:
    P_x *= p
round(P_x, 3)  # → 0.015

Why it matters today

Prompting is still how most applications steer a model: a system prompt describes the task and a few examples pin down the format. Modern chat models are additionally trained to follow instructions (see InstructGPT), so zero-shot works far better than it did for GPT-3, but few-shot examples remain the most reliable way to show an exact output format.

2.1 Model and architectures · original

Everyday picture

GPT-3 is not a new design. It is GPT-2 (a decoder-only Transformer) scaled up, trained in eight sizes from 125 million to 175 billion parameters so the authors could watch how abilities change with size. The one architectural tweak: layers alternate between full attention and a cheaper, locally banded sparse attention.

Tiny example: counting the parameters

Almost all of a Transformer's weights live in the attention and feed-forward matrices, and a standard block has about 12·d_model² of them. Pick a size to check the paper's parameter count against this rule of thumb and to see the training compute.

Reading it: each position of the slider is one of the paper's eight models (layer count, width and heads restated from Table 2.1). The first computed row applies the 12 × layers × width² rule; the second adds the embedding table (50,257 tokens × width), and together they land close to the paper's stated size. The last rows estimate training compute with the 6 × parameters × tokens rule, since every model saw 300 billion tokens. For the 175B model this gives 3.15 × 10²³ FLOPs, matching the paper's own figure of 3.14 × 10²³.

In words: “a Transformer's size is about twelve times the number of layers times the width squared; training costs about six arithmetic operations per parameter for every token of data.”

With the numbers: 12 × 96 × 12,288² ≈ 174 billion parameters. Then 6 × 175 × 10⁹ × 300 × 10⁹ = 3.15 × 10²³ operations, about 3,646 petaflop/s-days (the paper's own table, from its exact counts, lists 3,640): a machine doing 10¹⁵ operations per second would need about 10 years.

In Python:

n_layer, d_model = 96, 12_288
N = 12 * n_layer * d_model ** 2
# billions of parameters
round(N / 1e9)  # → 174
# parameters, training tokens
N, D = 175e9, 300e9
C = 6 * N * D
f"{C:.3g}"  # → '3.15e+23'
# petaflop/s-days
round(C / (1e15 * 24 * 3600))  # → 3646
# years at 10^15 operations per second
round(C / 1e15 / (365 * 24 * 3600))  # → 10

Why it matters today

These two rules of thumb (12·L·d² and 6·N·D) are how practitioners estimate model size and training cost on the back of an envelope. The transformer lesson counts parameters exactly and checks the rule; the scaling laws companion asks how to split a compute budget between N and D.

2.2 Training dataset · original

“This essentially accepts a small amount of overfitting in exchange for higher quality training data.”Brown et al. (2020), §2.2

Everyday picture

A reading list for a model. Most of the pages come from a filtered web crawl, but the authors deliberately hand the model the better books more often than their size alone would suggest, like a student who skims the newspaper once but rereads the good textbooks three times.

Reading it: each row is one source. The bar is its share of what the model actually read during training; the right-hand column shows how big the source is and how many times it was read in full over the 300 billion training tokens. Common Crawl makes up 60% of training but was not even read once in full (0.44 epochs), while Wikipedia, a small 3 billion tokens, was read 3.4 times. Numbers restated from Brown et al. (2020), Table 2.2.

Why it matters today

Data curation, weighting and deduplication are still among the most important and least visible parts of training a language model. The paper also removed near-duplicate documents so the held-out set stayed honest, a theme that returns in §4.

2.3 Training process · original

Larger models used larger batches and smaller learning rates: the 175B model used 3.2 million tokens per batch and a peak learning rate of 0.6 × 10⁻⁴, against 0.5 million tokens and 6.0 × 10⁻⁴ for the smallest. To fit in memory, each model was split across many GPUs, both within each matrix multiply and across layers, on a Microsoft cluster of V100 GPUs.

Everyday picture: a huge model takes careful, small steps (low learning rate) but averages each step over far more examples (big batch), like a large ship that turns slowly but steadily. The optimizers lesson shows why step size matters.

2.4 Evaluation · original

Everyday picture

For a multiple-choice question, you don't ask the model to write an answer; you ask how likely it finds each option as a continuation, and pick the most likely. One trap: a long answer is a product of many probabilities below 1, so it always looks less likely than a short one. The fix is to compare the average per token.

In words: “for each candidate answer, add up the log-probabilities of its tokens and divide by how many tokens it has; pick the candidate with the highest average.”

With the numbers: answer A is two tokens with probabilities 0.5 and 0.4: ln 0.5 + ln 0.4 = −0.693 − 0.916 = −1.609, average −0.805. Answer B is one token with probability 0.3: ln 0.3 = −1.204. By total, B would win (−1.204 > −1.609); by average, A wins (−0.805 > −1.204). Length normalization stops long answers being penalized just for being long.

In Python:

import math
# c: the candidate's token probabilities
def score(c):
    # (1/|c|) Σ_t log P(c_t | ...)
    return sum(math.log(p) for p in c) / len(c)
# totals
round(math.log(0.5) + math.log(0.4), 3), round(math.log(0.3), 3)  # → (-1.609, -1.204)
# averages
round(score([0.5, 0.4]), 3), round(score([0.3]), 3)  # → (-0.805, -1.204)

The paper uses per-token averaging for most multiple-choice tasks, draws the K demonstrations at random from each task's training set, and uses K between 10 and 100 depending on how many fit in 2,048 tokens.

Why it matters today

Scoring options by likelihood is still how many benchmarks evaluate base models. The loss functions lesson explains log-probabilities and why they are summed rather than multiplied.

3.1 LAMBADA and completion tasks · original

Everyday picture

LAMBADA is a fill-in-the-last-word test where you must read a whole paragraph to know the word: a test of long-range understanding. Few-shot examples help in a surprising way: they show the model that the answer must be exactly one word, which a plain continuation might not respect.

An 18-point jump over the previous state of the art, from examples in a prompt, with no training on the task.

3.2 Closed-book question answering · original

Everyday picture

“Open book” systems search documents for the answer first (that is retrieval-augmented generation); “closed book” means answering from memory alone. On TriviaQA, GPT-3 few-shot scored 71.2% closed-book, above the 68.0% of a fine-tuned open-book RAG system at the time (Table 3.3). On Natural Questions, which asks about more obscure facts, it scored 29.9% against RAG's 44.5%.

Why it matters today

This split still holds: large models remember popular facts well and obscure or recent facts badly, which is exactly why production systems retrieve documents instead of trusting memory. See the RAG lesson.

3.9.1 Arithmetic · original

Everyday picture

Nobody taught GPT-3 to add. The test asks questions in plain English (“Q: What is 23 plus 58? A:”) to see whether a model that only predicts text has picked up arithmetic along the way.

Reading it: each bar is GPT-3 175B's accuracy on one kind of problem, restated from Brown et al. (2020), Table 3.9. Toggle between settings. Two-digit addition is essentially solved few-shot (100%), three-digit addition is right about 80% of the time, and accuracy falls steeply with more digits (about 9% at five digits) and for multiplication (29%). Examples in the prompt help a great deal: two-digit addition goes from 77% zero-shot to 100% few-shot. The pattern looks like partial skill rather than a memorized table, but not like a reliable calculator either.

Why it matters today

Arithmetic is where the tokenizer quirks from the tokenization lesson bite: numbers split into irregular chunks. Modern systems either train heavily on maths, write out intermediate steps, or simply call a calculator tool.

3.9.4 News article generation · original

Everyday picture

People were shown short news articles (about 200 words), some real and some written by models of each size, and asked which were machine-written. If they could tell perfectly they would score 100%; pure guessing scores 50%.

Reading it: the x-axis is model size on a log scale, from 125 million to 175 billion parameters; the y-axis is how often people correctly spotted the machine-written article (restated from Brown et al., 2020, Table 3.11). The flat line is chance. As models grow, detection falls towards the line, reaching 52% for GPT-3 175B: barely better than a coin toss. A deliberately bad control model was spotted 86% of the time.

Why it matters today

This result drove the paper's long discussion of misuse, and it is why detecting machine-written text remains an unsolved problem rather than a solved one.