CLIP, annotated

2.1 Learning from captions · original

Learning from natural language has two advantages over fixed labels. It scales, because nobody has to label anything: the captions already exist. And the model learns to connect images to language, so at test time you can describe any category in words instead of retraining.

2.2 A 400-million-pair dataset · original

Existing caption datasets were either small (about 100,000 photos) or noisy (a 100-million-photo set shrank to 15 million after keeping only images with real English descriptions). So the authors built WIT (WebImageText): 400 million image-text pairs, gathered by searching for pairs whose text contained one of 500,000 queries, with at most 20,000 pairs per query to keep it balanced. Its total word count is similar to the text dataset used to train GPT-2.

2.3 The contrastive objective · original

Everyday picture

First they tried to make the model write each image's caption word for word. That is very hard, because the same photo could be captioned a thousand ways. The easier game is matching: shuffle a stack of photos and a stack of captions, and learn to pair them up. That turned out to be far more efficient. Predicting only the bag of words, ignoring order, learned 3× faster than writing the caption, and switching to matching gave another 4×.

Reading it: a batch of N image-caption pairs goes in, here N = 4. The image encoder turns each image into a vector I₁ to I₄ (the rows); the text encoder turns each caption into a vector T₁ to T₄ (the columns). Every cell is the similarity of one image with one caption: all N × N combinations. The N cells on the green diagonal are the real pairs; the other N² − N are mismatches. Training pushes diagonal cells up and everything else down. In the real paper N is 32,768, so each image is contrasted with over 32,000 wrong captions at once.

Tiny example: two pairs

Two images and their two captions. After normalizing every vector to length 1, the similarities are: image 1 with caption 1 = 0.9, with caption 2 = 0.1; image 2 with caption 1 = 0.2, with caption 2 = 0.8. Read row 1 as a two-way multiple-choice question: softmax(0.9, 0.1) gives the right caption probability e^0.9 / (e^0.9 + e^0.1) = 2.46 / 3.57 = 0.690, a loss of −ln 0.690 = 0.371. Row 2 gives 0.646 and 0.437. The columns ask the reverse question, “which image matches this caption?”, and give 0.403 each. The average of the two directions is 0.404.

The math

In words: “score every image against every caption with a scaled dot product of unit vectors; for each image, the right caption should win a softmax over all captions, and for each caption, the right image should win a softmax over all images; average the two losses.”

With the numbers: with the scale at 1, the two-pair example gives image-to-text losses 0.371 and 0.437 (mean 0.404) and text-to-image losses 0.403 and 0.403 (mean 0.403); the total is ½(0.404 + 0.403) = 0.404. At CLIP's starting scale of 1/0.07 ≈ 14.3, the same similarities give a loss below 0.001, because scaling stretches the gap between right and wrong.

In Python:

import math
# I_i · T_j: row i is image i, column j is caption j
sims = [[0.9, 0.1],
        [0.2, 0.8]]
def loss(scale):
    # ℓ_ij = e^t · (I_i · T_j)
    l = [[scale * s for s in row] for row in sims]
    N = len(l)
    img = [-math.log(math.exp(l[i][i]) / sum(math.exp(l[i][j]) for j in range(N))) for i in range(N)]
    txt = [-math.log(math.exp(l[j][j]) / sum(math.exp(l[i][j]) for i in range(N))) for j in range(N)]
    # ½ (image-to-text + text-to-image)
    return img, txt, (sum(img) / N + sum(txt) / N) / 2
img, txt, L = loss(1)
[round(x, 3) for x in img], [round(x, 3) for x in txt]  # → ([0.371, 0.437], [0.403, 0.403])
round(sum(img) / 2, 3), round(sum(txt) / 2, 3), round(L, 3)  # → (0.404, 0.403, 0.404)
# CLIP's starting scale, 1/0.07 ≈ 14.3
loss(1 / 0.07)[2] < 0.001  # → True

The paper notes this is the same batch trick as the multi-class N-pair loss and InfoNCE, earlier used for medical images: the other items in the batch serve as free negatives. See the losses lesson.

Try it: the similarity matrix, live

Reading it: rows are images, columns are captions, and each cell shows the probability that this caption is the right one for this image (a softmax along the row). Darker cells mean higher probability, and the green-bordered diagonal holds the correct pairs. At the default τ = 0.07 each row is nearly certain, even though the dog and cat photos are fairly similar to each other's captions (both are animals). Drag τ towards 1 and the rows go soft: the same cosine similarities now give much less decisive probabilities. CLIP learns τ during training, starting at 0.07 and capped so the scale 1/τ never exceeds 100. Hover a cell for its raw cosine similarity.

Why it matters

Because the loss only cares about ranking right pairs above wrong ones within a batch, bigger batches mean more, harder competition. That is why CLIP used a batch of 32,768, and why contrastive training is hungry for scale. The contrastive lesson shows how adding hard negatives sharpens this further.

2.4–2.5 Models and training · original

• Image encoders: five ResNets (modified, with attention pooling at the end) and three Vision Transformers, which cut the image into patches and treat each patch as a token.
• Text encoder: a Transformer of 63 million parameters, 12 layers, 512 wide, 8 heads, reading lower-cased byte-pair-encoded text with a 49,152-token vocabulary, capped at 76 tokens. The vector at the end-of-text token becomes the caption's embedding.
• Simple projections: each encoder's output is mapped into the shared space by one linear layer and normalized to length 1.
• Training: 32 epochs, batch 32,768, Adam with decoupled weight decay (AdamW) and cosine learning-rate decay. The largest ResNet took 18 days on 592 V100 GPUs; the largest Vision Transformer 12 days on 256.
• Learned temperature: τ is a trained parameter, initialized at 0.07 and clipped so logits are never scaled by more than 100, which the authors found necessary for stability.

The best model, ViT-L/14 fine-tuned briefly at 336-pixel resolution, is the one reported as “CLIP” in most results.

Classifying with text · original

Everyday picture

To sort photos of pets into breeds, you don't retrain CLIP. You write one caption per breed, “a photo of a beagle”, “a photo of a pug”, and so on, and for each photo ask: which caption fits best? The captions become the classifier.

In words: “embed the image and one caption per class; the probability of each class is a softmax over the image's cosine similarity with each caption, sharpened by the temperature.”

With the numbers: with two classes, cosines 0.30 (dog) and 0.25 (cat), and τ = 0.01, the logits are 30 and 25, so p(dog) = 1 / (1 + e⁻⁵) = 0.993. A 0.05 difference in cosine becomes a confident answer.

In Python:

import math
# cos(I, T_k) for each class caption
cos = {"dog": 0.30, "cat": 0.25}
tau = 0.01
logits = {k: c / tau for k, c in cos.items()}
{k: round(z) for k, z in logits.items()}  # → {'dog': 30, 'cat': 25}
# Σ_j exp(cos(I, T_j) / τ)
total = sum(math.exp(z) for z in logits.values())
# p(y = dog | image)
round(math.exp(logits["dog"]) / total, 3)  # → 0.993
# the same thing, written with the gap of 5
round(1 / (1 + math.exp(-5)), 3)  # → 0.993

The paper points out that this is a linear classifier whose weights are written by the text encoder from the class names. That is why no training examples are needed.

Reading it: pick a stand-in photo. Each bar is the probability CLIP's zero-shot rule gives to one class, computed from the cosine similarity between the photo's embedding and the embedding of “a photo of a {class}”. Press “add a class” to add “bird”: the probabilities are recomputed over the new set, and nothing is retrained. Notice that the dog photo gives the cat caption, and then the bird caption, a little probability, far more than car or pizza: they share the “animal” direction.

Against the earlier attempt · original

76.2% top-1 matches the original fully supervised ResNet-50; CLIP's top-5 accuracy reached 95%. The authors stress this is context, not a controlled comparison: CLIP used about 10× more data and far more compute.

Prompt engineering and ensembling · original

Everyday picture

A single word is ambiguous. “Crane” could be a bird or a machine; “boxer” a dog or an athlete. And web captions are rarely one word. Wrapping the label in a sentence tells CLIP what kind of thing you mean.

• “A photo of a {label}.” alone improved ImageNet accuracy by 1.3%.
• Task hints helped more: “a photo of a {label}, a type of pet.” on pet breeds; “a satellite photo of a {label}.” for satellite images; quotes around text for reading digits.
• Ensembling: average the text embeddings of 80 different prompts (“a photo of a big {label}”, “a photo of a small {label}”, …) into one per class, which costs nothing extra at prediction time. That added 3.5% on ImageNet.

Together, almost 5 points on ImageNet, which the paper likens to the gain from 4× more compute, for free.

Why it matters today

This was an early, clean demonstration that the wording of what you ask a model changes the answer, for vision as much as for language. Averaging prompt embeddings is still a common trick.

How good is zero-shot? · original

• Against a fully supervised linear probe on ResNet-50 features, zero-shot CLIP won on 16 of 27 datasets.
• It did especially well on everyday objects and scenes, and poorly on specialized or abstract tasks: satellite images, tumour detection, counting objects, traffic signs, distance estimation.
• Zero-shot CLIP matched a 4-shot linear probe on its own features: describing a class in words was worth about four labelled examples per class.