An Image is Worth 16x16 Words, annotated

How to read this page

Nothing here assumes you already know the jargon. Three things help:

• Any dotted word explains itself when you hover it, tab to it, or tap it.
• Every symbol inside an equation does the same, and each equation is followed by a table of its symbols, a sentence reading it aloud, the numbers of a tiny example, and the same example in a few lines of Python.
• The pictures are live: hover or tap the parts of the architecture diagram, change the patch size, and move the query patch in the attention picture.

The running example is the one the multimodal lesson builds in code: a 4×4 grayscale picture whose pixels are the numbers 0 to 15, cut into four 2×2 patches.

Abstract

“We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks.”Dosovitskiy et al. (2020), Abstract. Read the original

Everyday picture

Cut a photo into a grid of small square tiles, as if it were a jigsaw, and lay the tiles out in a row like the words of a sentence. Now hand that row to the same transformer that reads text. That is the whole idea of the Vision Transformer. The title's joke is the arithmetic: each tile is 16×16 pixels, and each tile becomes one “word”.

What the paper claims

• A transformer with almost no image-specific design can classify images as well as the best convolutional networks.
• The catch: it needs a lot of pre-training data. On about a million images it trails similar-sized ResNets; on tens to hundreds of millions it overtakes them.
• Once there is enough data it is also cheaper: the best ViT reached 88.55% on ImageNet using several times less pre-training compute than the convolutional networks it beat.

Why it matters today

The image half of most vision-language models is a ViT, often the one trained inside CLIP; diffusion transformers generate images as grids of patches; video models cut patches through time. The multimodal lesson builds the ViT front end from scratch.

1 Introduction · original

“We find that large scale training trumps inductive bias.”Dosovitskiy et al. (2020), §1

Everyday picture

A convolution is a flashlight that sweeps a photo looking for one small pattern, and it comes with two beliefs built in before it has seen a single image: nearby pixels matter most, and a pattern means the same thing wherever it appears. Those beliefs are its inductive bias. A transformer has almost none: every patch may look at every other patch from the first layer on, and it must learn which neighbours matter. A student who is told the rules learns faster from a few examples; a student who has to discover them needs many more examples, but is not stuck with rules that are only mostly right.

Tiny example

The paper tests that trade with three pre-training sets of very different size: ImageNet with 1.3 million images, ImageNet-21k with 14 million, and an internal set, JFT-300M, with 303 million. From the smallest to the largest is a factor of about 230.

In Python:

# pre-training set sizes, in millions of images
imagenet, imagenet_21k, jft = 1.3, 14, 303
round(jft / imagenet)  # → 233

Why it matters

“Given enough data, fewer built-in assumptions win” became one of the decade's recurring lessons. It is why a single architecture now reads text, images, audio and video, and why data, rather than clever structure, is where so much of the effort goes.

2 Related work · original

“Naive application of self-attention to images would require that each pixel attends to every other pixel.”Dosovitskiy et al. (2020), §2

Everyday picture

Self-attention compares every token with every other, so its cost grows with the square of the number of tokens. Treat every pixel as a token and a photo becomes a sentence tens of thousands of words long. Earlier work shrank the problem with special attention patterns (only nearby pixels, only along rows and columns), which worked but needed custom code to run fast. The paper's shortcut is blunt: make the tokens bigger.

Tiny example

A 224×224 image has 50,176 pixels. Attention over pixels would score 50,176² ≈ 2.5 billion pairs in every layer. Cut into 16×16 patches it is 196 tokens, 197 with the extra classification token below, and 197² = 38,809 pairs: about 65,000 times fewer.

In Python:

pixels = 224 * 224
pixels  # → 50176
# every pixel scored against every pixel
print(f"{pixels ** 2:,}")  # → 2,517,630,976
# 196 patches plus the classification token
tokens = (224 // 16) ** 2 + 1
tokens ** 2  # → 38809
round(pixels ** 2 / tokens ** 2)  # → 64872

What sits nearby

• Cordonnier et al. (2020) used 2×2 patches and full self-attention: the same model in spirit, but only for small images, and without the large-scale pre-training that makes ViT competitive.
• Image GPT (Chen et al., 2020) ran a transformer over pixels of shrunken images as a generative model, reaching 72% on ImageNet.
• Many hybrids added attention to CNNs, or ran attention on a CNN's output, rather than replacing the CNN.

Why it matters

The patch is the unit every later vision and multimodal transformer inherited, and the quadratic cost is why patch size is still a cost dial: see the attention lesson for where the n² comes from.

3 Method · original

“In model design we follow the original Transformer (Vaswani et al. 2017) as closely as possible.”Dosovitskiy et al. (2020), §3

Everyday picture

Read the picture below from the bottom up, the way data flows. The only new parts are at the bottom (cutting and flattening the image, one learned projection) and at the top (reading off a class). Everything in between is the transformer encoder of 2017, unchanged.

Reading it: start at the bottom. The image is cut into a grid of patches (nine here), each patch is flattened into one long row of pixel values, and one shared linear projection E turns every row into a vector of the model's width. An extra learned token, marked 0*, is put in front; then a learned position vector is added to each of the ten slots. The encoder (the dashed box, repeated L times) is the transformer's: Norm, multi-head attention, add the input back; Norm, MLP, add back. The two wires that bypass a block and rejoin at the + are the residual connections. At the top, only the class token's final state (slot 0) goes on to the head; the patch tokens' outputs are not used for classification at all.

Why it matters today

Following the original transformer “as closely as possible” was a bet on engineering as much as science: every efficient transformer implementation already written for text could be reused unchanged.

3.1 Vision Transformer (ViT) · original

Cutting: how many tokens?

Everyday picture: lay graph paper over the photo and cut along every sixteenth line. The number of tiles is the number of rows of tiles times the number of columns.

In words: “the number of patches is the image's area divided by one patch's area.” It is also the length of the sequence the transformer reads.

With the numbers: the worked 4×4 picture in 2×2 patches: 16 / 4 = 4. A 224×224 photo in 16×16 patches: 50,176 / 256 = 196. The same photo in 14×14 patches (ViT-H/14): 256.

In Python:

# the worked 4×4 picture in 2×2 patches
H, W, P = 4, 4, 2
H * W // P ** 2  # → 4
H, W, P = 224, 224, 16
H * W // P ** 2  # → 196
P = 14
H * W // P ** 2  # → 256

Try it: patch size is a cost dial

Reading it: the picture is a 224×224 image (drawn here, not a photo from the paper) with the patch grid laid over it. Each square becomes one token. Pick a patch size and read the numbers: halving the patch side quadruples the tokens, and the attention work, which grows with the square of the sequence length, goes up about sixteen-fold. That is why the paper notes that models with smaller patches are more expensive, and why ViT-B/32 is the cheapest model in its study and ViT-H/14 the dearest. Each token also starts life wider when the patch is bigger (P × P × 3 numbers).

Embedding: patches, a class token, and positions

Everyday picture: every tile fills in the same questionnaire (the projection E) and the answers become its card. A blank card that belongs to no tile goes on top of the pile (the class token): it will collect a summary of the whole picture. Then every card gets a sticker saying which slot it sits in (the position embedding), so shuffling the pile loses nothing.

Tiny example: the lesson's picture. The four patches, flattened, are (0, 1, 4, 5), (2, 3, 6, 7), (8, 9, 12, 13) and (10, 11, 14, 15). With a 4 × 2 projection E whose first column adds a patch's top row and whose second adds its bottom row, and positions (row, column), the tokens are (1, 9), (5, 14), (18, 25) and (22, 30). For the class token take, for illustration, x_class = (1, 1) with position (0, 0). All of these are learned in a real model; the numbers here are chosen so you can check them by hand.

In words: “project every flattened patch with the same matrix E, stack the results under one extra learned class vector, and add a learned position vector to every row.”

With the numbers: the top-right patch (2, 3, 6, 7) times E is (2 + 3, 6 + 7) = (5, 13); plus its position (0, 1) gives (5, 14). The whole of z₀ is five rows: (1, 1), (1, 9), (5, 14), (18, 25), (22, 30).

In Python:

patches = [[0, 1, 4, 5], [2, 3, 6, 7], [8, 9, 12, 13], [10, 11, 14, 15]]
# E: column 1 adds a patch's top row, column 2 its bottom row
E = [[1, 0], [1, 0], [0, 1], [0, 1]]
x_class = [1, 1]
# E_pos: one row per slot, the class slot first
E_pos = [[0, 0], [0, 0], [0, 1], [1, 0], [1, 1]]
# x_p^i E: each output number is a dot product with one column of E
projected = [[sum(x * E[r][c] for r, x in enumerate(p)) for c in range(2)] for p in patches]
projected  # → [[1, 9], [5, 13], [17, 25], [21, 29]]
# [x_class; x_p^1 E; ...] + E_pos
z0 = [[a + b for a, b in zip(row, pos)] for row, pos in zip([x_class] + projected, E_pos)]
z0  # → [[1, 1], [1, 9], [5, 14], [18, 25], [22, 30]]

The size of E is what the paper's shorthand encodes: for ViT-B/16, each 16 × 16 × 3 patch has 768 numbers and the model width is also 768, so E is a 768 × 768 matrix, 589,824 learned numbers shared by every patch.

In Python:

P, C, D = 16, 3, 768
# E is (P²·C) × D
print(f"{P * P * C} x {D} = {P * P * C * D:,}")  # → 768 x 768 = 589,824

In code: the lesson's worked_example_patches runs these steps on this picture, and VisionEncoder.embed is the formula for a real image (without the class token, since a vision-language model uses every patch's output rather than one summary).

The encoder: two sub-layers, each wrapped the same way

Everyday picture: each layer is a meeting followed by desk work. In the meeting (multi-head attention) every token hears from every other; at the desk (the MLP, two layers with a GELU between) each token thinks alone. Before each, the token's numbers are tidied to a standard scale (layer norm); after each, the result is added to what the token already had, as a correction rather than a replacement.

Tiny example: follow the token (5, 14). Layer norm maps it to (−1, 1): its mean is 9.5 and its spread 4.5. Say attention, reading the normalized tokens, proposes the change (0.4, −0.2) for this one (an illustrative number); adding it back gives (5.4, 13.8). Normalize again, and say the MLP proposes (0.1, 0.3): the token leaves the layer as (5.5, 14.1).

In words: “in each of the L layers, normalize the tokens, let them attend to each other, and add the result back; then normalize again, pass each token through the MLP, and add that back too.”

With the numbers: LN(5, 14) = (−1, 1); (5, 14) + (0.4, −0.2) = (5.4, 13.8); LN(5.4, 13.8) = (−1, 1) again; (5.4, 13.8) + (0.1, 0.3) = (5.5, 14.1).

In Python:

import math
# LN: shift to mean 0, scale to spread 1 (the learned gain and bias left at 1 and 0)
def LN(v):
    mean = sum(v) / len(v)
    sd = math.sqrt(sum((a - mean) ** 2 for a in v) / len(v))
    return [round((a - mean) / sd, 6) for a in v]
z = [5, 14]
LN(z)  # → [-1.0, 1.0]
# MSA(LN(z)) for this token: an illustrative proposal
msa = [0.4, -0.2]
z_prime = [a + b for a, b in zip(msa, z)]
z_prime  # → [5.4, 13.8]
LN(z_prime)  # → [-1.0, 1.0]
# MLP(LN(z')): another illustrative proposal
mlp = [0.1, 0.3]
[round(a + b, 6) for a, b in zip(mlp, z_prime)]  # → [5.5, 14.1]

In code: TransformerBlock in the transformer lesson is exactly these two lines (pre-norm, GELU inside FeedForward), and VisionEncoder stacks it with the causal mask switched off, because a picture has no “future”.

The output: read the class token

Everyday picture: after the last layer, the blank card on top of the pile has been filled in by listening to every tile, L times over. Normalize it and hand it to a small classifier.

Tiny example: say the class token leaves the last layer as (3, 1). Normalized it is (1, −1). A two-class head with weights [[1, −1], [−1, 1]] scores it (2, −2): class 1 wins.

In words: “the image's representation is the class token's state after the last layer, normalized.” A head turns it into one score per class: an MLP with one hidden layer during pre-training, a single linear layer when fine-tuning.

With the numbers: LN(3, 1) = (1, −1); scores (1 × 1 + (−1) × (−1), 1 × (−1) + (−1) × 1) = (2, −2).

In Python:

import math
def LN(v):
    mean = sum(v) / len(v)
    sd = math.sqrt(sum((a - mean) ** 2 for a in v) / len(v))
    return [(a - mean) / sd for a in v]
# z_L^0: the class token after the last layer
y = LN([3, 1])
y  # → [1.0, -1.0]
# a linear head, one column per class
head = [[1, -1], [-1, 1]]
[sum(y[r] * head[r][c] for r in range(2)) for c in range(2)]  # → [2.0, -2.0]

Inductive bias · original

“In ViT, only MLP layers are local and translationally equivariant, while the self-attention layers are global.”Dosovitskiy et al. (2020), §3.1

Count where ViT is told anything about images being two-dimensional: once when the image is cut into patches, and once when position embeddings are resized for fine-tuning (§3.2). Everything else is learned. Its position embeddings start random, carrying no idea of which patch is next to which; a CNN, by contrast, has locality and translation equivariance in every layer.

Hybrid architecture · original

Instead of raw pixels, the patches can be cut from a CNN's feature map, even 1×1 “patches” (each position of the map becomes a token). The paper calls this the hybrid, and tests it alongside pure ViTs in §4.4.

Why it matters today

This is the whole architecture. Today's vision encoders are still this recipe, often with the class token dropped in favour of averaging the patch outputs or passing all of them on, which is what a vision-language model's projector does in the multimodal lesson.

3.2 Fine-tuning and higher resolution · original

Everyday picture

Pre-train once on a big general dataset, then adapt to a small specific one: transfer learning. Two practical details make it work. The old classifier is thrown away and a new one is started at zero, so it begins with no opinion. And the adapted model is shown sharper images than it was trained on, which helps; but the position stickers were made for the old grid, so they are stretched to fit the new one.

Tiny example

Pre-training at 224×224 with 16-pixel patches gives a 14 × 14 grid: 196 learned positions. Fine-tuning at 384×384 with the same patches gives a 24 × 24 grid: 576 slots. The 196 learned vectors are laid out on their 14 × 14 grid and smoothly interpolated in two dimensions to fill 24 × 24. In one dimension: stretching the values (0, 1) over three slots gives (0, 0.5, 1). The new head, D × K and all zeros, scores every class 0 at first.

In Python:

P = 16
[(side // P) ** 2 for side in (224, 384)]  # → [196, 576]
# stretching two learned values over three slots: straight-line interpolation
old = [0.0, 1.0]
new = [old[0] + (old[1] - old[0]) * i / 2 for i in range(3)]
new  # → [0.0, 0.5, 1.0]
# a zero-initialized head scores every class 0, whatever the image
y, head = [1.0, -1.0], [[0, 0, 0], [0, 0, 0]]
[sum(y[r] * head[r][c] for r in range(2)) for c in range(3)]  # → [0.0, 0.0, 0.0]

Why it matters today

Pre-train big, fine-tune small is now the default in vision as it is in language (see fine-tuning in the training stages lesson). Resizing position embeddings is also why many vision encoders can accept images at more than one resolution, at the cost the multimodal lesson counts: more tokens.

4 Experiments · original

Everyday picture

Three kinds of model are put through the same course: ResNets (the reigning CNNs), ViTs, and hybrids. Each is pre-trained on datasets of different sizes and then tested on many smaller tasks. The questions are simple: who wins, how much data does each need, and what does each cost to train?

4.1 Setup · original

Datasets

Pre-training uses ImageNet (1.3 million images, 1,000 classes), ImageNet-21k (14 million, 21,000 classes) and JFT (303 million, 18,000 classes), with any images that also appear in the test sets removed. The models are then tested on ImageNet (with the original labels and the cleaned-up ReaL labels), CIFAR-10 and CIFAR-100, Oxford-IIIT Pets, Oxford Flowers-102, and VTAB.

Model variants

ViT-Large sits between them (24 layers, width 1024). Base and Large copy BERT's shapes; Huge is new. A name like ViT-L/16 means the Large model with 16 × 16 patches. You can check the parameter count yourself from the architecture of §3.1: count every matrix and bias, then add the patch projection, the class token and the position table.

In Python:

def vit_params(L, D, D_mlp, P, image=224, C=3):
    # query, key, value and output matrices, with their biases
    attention = 4 * D * D + 4 * D
    # the MLP's two layers, with their biases
    mlp = D * D_mlp + D_mlp + D_mlp * D + D
    # two layer norms per layer, each a gain and a bias
    norms = 2 * 2 * D
    N = (image // P) ** 2
    # the projection E with its bias, the class token, the position table, the final norm
    embed = P * P * C * D + D
    return L * (attention + mlp + norms) + embed + D + (N + 1) * D + 2 * D
# ViT-B/16, without the classification head
print(f"{vit_params(12, 768, 3072, 16):,}")  # → 85,798,656
# ViT-H/14
print(f"{vit_params(32, 1280, 5120, 14):,}")  # → 630,764,800

That is 85.8 million for Base (the paper's 86M) and 630.8 million for Huge (the paper's 632M; this count leaves out the classification head). The baselines are ResNets with two changes the paper credits with better transfer: batch normalization swapped for group normalization, and standardized convolutions. The paper calls them ResNet (BiT).

Training and evaluation

Every model, ResNets included, is pre-trained with Adam (β₁ = 0.9, β₂ = 0.999), batch size 4,096 and a high weight decay of 0.1, with linear warmup and decay. Fine-tuning uses SGD with momentum at batch size 512. Results are reported as fine-tuning accuracy, or as few-shot linear accuracy, a quick check that fits a simple linear map on frozen features (a close relative of a linear probe).

Why it matters today

The same backbone with the same optimizer and recipe for every model makes the comparison about architecture and data, not about who tuned harder. The Adam-for-ResNets choice was unusual enough that the paper checks it separately (Appendix D.1).

4.2 Comparison to state of the art · original

“The smaller ViT-L/16 model pre-trained on JFT-300M outperforms BiT-L (which is pre-trained on the same dataset) on all tasks, while requiring substantially less computational resources to train.”Dosovitskiy et al. (2020), §4.2

Everyday picture

Two champions to beat: BiT-L, a very large ResNet trained on the same JFT images, and Noisy Student, a large EfficientNet trained with extra unlabelled images. Cost is counted in TPUv3-core-days.

Reading it: each bar is one model's pre-training compute in TPUv3-core-days, on the same scale, with its ImageNet accuracy beside it. The two convolutional champions are the long bars. ViT-H/14 is more accurate than both while using about a quarter of BiT-L's compute (2.5k against 9.9k), and ViT-L/16 on JFT edges out BiT-L on ImageNet (87.76% against 87.54%) at about a fourteenth of its compute (0.68k). The shortest bar, ViT-L/16 on the public ImageNet-21k, could be trained on one 8-core cloud TPU in about 30 days.

In Python:

# TPUv3-core-days, in thousands
bit_l, vit_h, vit_l = 9.9, 2.5, 0.68
round(bit_l / vit_h, 1)  # → 4.0
round(bit_l / vit_l, 1)  # → 14.6

Why it matters today

The paper is careful to add that compute also depends on schedule, optimizer and weight decay, and runs a controlled study (§4.4) to separate them. Even so, “better and cheaper” is what made the field switch.

4.3 Pre-training data requirements · original

Everyday picture

If ViT has to learn what a CNN is told, it should need more examples. The paper tests this directly: pre-train on ever larger datasets and watch which models improve most.

Reading it: the x-axis is the size of the pre-training set in millions of images, on a log scale: ImageNet (1.3M), ImageNet-21k (14M) and JFT-300M (303M). The y-axis is ImageNet accuracy after fine-tuning. Each line is one ViT. Look at the left end: pre-trained on ImageNet alone, the Large models are worse than the Base ones (76.5% against 77.9% for the /16 pair), even with regularization; there is not enough data to fill a bigger model. In the middle they are level-ish; at the right, on JFT, ViT-L/16 leads ViT-B/16 by three points. Plotted from Table 5 of the paper, which is its Figure 3 in numbers; the figure also shades the range of BiT ResNets, which beat ViT at the left and are overtaken at the right. ViT-H/14 was not trained on ImageNet alone; it reaches 85.1% (21k) and 88.0% (JFT).

A second test: slices of the same dataset

Different datasets differ in more than size, so the paper also trains on random subsets of JFT of 9, 30 and 90 million images and the full 300 million, with no extra regularization. ViT-B/32 (slightly faster than a ResNet-50) is much worse on 9 million images but better from 90 million up. The paper's reading: convolution's built-in assumptions help on small data, and on large data learning the patterns directly is enough, even better.

Why it matters today

This result is why ViTs are almost always pre-trained on very large collections, and why a line of later work set out to train them well on ImageNet alone with stronger augmentation and regularization.

4.4 Scaling study · original

“First, Vision Transformers dominate ResNets on the performance/compute trade-off.”Dosovitskiy et al. (2020), §4.4

Everyday picture

With data no longer the bottleneck (everything pre-trained on JFT), the fair question is value for money: for a given amount of pre-training compute, which family gives the best accuracy?

Reading it: the x-axis is total pre-training compute in exaFLOPs (log scale); the y-axis is ImageNet accuracy after fine-tuning. Each bend in a line is one model: seven ResNets, six ViTs, five hybrids. At any compute, the ViT line sits above the ResNet line: at around 200 exaFLOPs, ViT-L/32 and ViT-B/16 reach about 84% where ResNet-50x2 reaches 82%. The hybrids (a ResNet stem feeding a ViT) lead at small budgets, but by the largest models the gap to pure ViTs is gone (both 87.12% at the top of their runs). Plotted from the ImageNet column of Table 6; the paper's Figure 5 averages five datasets and finds ViT needs about 2 to 4 times less compute for the same accuracy.

Why it matters today

The third observation pointed the way forward: the ViT curve had not flattened by the largest model tried, so bigger vision transformers were the obvious next step. It is the same story the scaling laws companion tells for language.

4.5 Inspecting the Vision Transformer · original

Everyday picture

If nobody tells the model that images are two-dimensional, does it work it out? The paper looks in three places. The projection E learns filters that look like sensible building blocks for small patches (shown with PCA). The position embeddings learn the grid: comparing them with cosine similarity, nearby patches have similar vectors, and patches in the same row or column do too. And attention learns how far to look.

Tiny example: attention distance

How far does a head look? For one query patch, take the distance in pixels to every patch, weight each distance by how much attention goes there, and add up. Say a query gives half its attention to itself (0 pixels away), 0.3 to a neighbour 16 pixels away and 0.2 to a patch 64 pixels away: 0 + 4.8 + 12.8 = 17.6 pixels. The paper averages this over many images and query patches, for every head at every layer. The formula below is written from that description (Appendix D.7).

In words: “a query's attention distance is the average distance to the patches it attends to, weighted by how much it attends to each.” It plays the part a receptive field plays in a CNN.

With the numbers: 0.5 × 0 + 0.3 × 16 + 0.2 × 64 = 17.6 pixels.

In Python:

# A_qj: where one query's attention goes
A = [0.5, 0.3, 0.2]
# ‖p_q − p_j‖ in pixels
dist = [0, 16, 64]
round(sum(a * d for a, d in zip(A, dist)), 1)  # → 17.6

Reading it: the grid is the 14 × 14 patches of a 224-pixel image; the outlined square is the query patch, and each patch is shaded by the share of attention it receives. The attention pattern here is made up (a smooth bump around the query, whose width you set), so the picture is illustrative; the distance in the readout is computed with the formula above. Narrow the focus and the head behaves like a small convolution: an attention distance of about 6 pixels. Widen it and the head looks at the whole image: over 80 pixels for a query in the middle, and about 145 from a corner. The paper finds both kinds of head already in the first layer of a trained ViT, something a convolution cannot do, and finds that distances grow with depth. The very local heads are rarer in hybrids, whose ResNet stem already does the local work.

Why it matters today

Pictures like this are a first look inside a vision transformer. The paper's attention maps from the class token (computed with attention rollout) land on the objects that decide the class.

4.6 Self-supervision · original

Everyday picture

Language transformers learn from unlabelled text by filling in hidden words, as BERT does. The image version: hide some patches and ask the model to guess what was there. The paper tries this as a first experiment, masked patch prediction.

Tiny example

Half of the patch embeddings are corrupted; of those, 80% become a learned [mask] vector, 10% another patch's embedding and 10% are left alone. The target is each corrupted patch's mean colour, rounded to 3 bits per channel: 8 levels for each of red, green and blue, so a choice among 8 × 8 × 8 = 512 colours.

In Python:

# 3 bits per channel, three channels
levels = 2 ** 3
levels ** 3  # → 512
# on a 196-patch image, half are corrupted
196 // 2  # → 98

The result

ViT-B/16 pre-trained this way reached 79.9% on ImageNet: 2 points better than training from scratch, but 4 behind supervised pre-training. The paper leaves contrastive pre-training for future work.

Why it matters today

Later work closed much of this gap, and self-supervised and image-text pre-training became common ways to train vision encoders. CLIP, for instance, trains a ViT against captions instead of class labels, and it is that ViT which sits inside many vision-language models.

5 Conclusion · original

“This simple, yet scalable, strategy works surprisingly well when coupled with pre-training on large datasets.”Dosovitskiy et al. (2020), §5

Everyday picture

Treat an image as a sentence of patches, use the text model as it is, and feed it enough data. No special vision machinery beyond the first cut.

Why it matters today

The open problems the authors listed (detection and segmentation, self-supervised pre-training, more scaling) each became an active line of research. The bigger consequence is one they only hint at: once images are tokens, a single transformer can read images and text together, which is the premise of the multimodal lesson.

Appendices, briefly · original

What they contain

• A, multi-head self-attention: the standard definition, decoded below. For the attention formula itself, see the Attention Is All You Need companion.
• B, training details: batch 4,096 and 10,000 warmup steps for every pre-training run; dropout 0.1 and weight decay 0.03 to 0.3 on the smaller datasets; fine-tuning at 384×384 with SGD and momentum 0.9.
• C, the tables behind Figures 4 and 5, which the charts above are plotted from.
• D, ablations: a few are summarised below.

D, k = 768, 12
D_h = D // k
D_h  # → 64
# U_qkv for one head is D × 3·D_h; U_msa is k·D_h × D
per_head_qkv = D * 3 * D_h
u_msa = k * D_h * D
print(f"{k * per_head_qkv + u_msa:,}")  # → 2,359,296
# the same total with 16 heads of 48
k = 16
print(f"{k * D * 3 * (D // k) + k * (D // k) * D:,}")  # → 2,359,296

What changed since 2020

The core, patches as tokens read by a plain transformer, is everywhere. Around it:

Glossary

Every term with hover guidance on this page, in one place.