LoRA and QLoRA, annotated

How to read this page

Nothing on this page assumes you already know the jargon.

• Any dotted word explains itself when you hover it, tab to it, or tap it.
• Every symbol inside an equation does the same. Hover the B in the first equation and see.
• The diagrams and sliders are live. Drag the rank slider in §4.1 and watch the parameter count collapse.

Each idea climbs the same ladder: an everyday picture, a tiny example you can check by hand, a diagram, the math, and why it still matters. The training stages lesson builds LoRA from scratch in NumPy.

LoRA: Abstract · original

“We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.”Hu et al. (2021), Abstract

Everyday picture

You own a 1,000-page textbook and want it to suit a new course. Reprinting the whole book for every course is absurd. Instead you keep the book exactly as it is and add a thin booklet of corrections for each course. Swap the booklet, and the same book serves a different class. LoRA is that booklet for a neural network: the huge weights stay frozen, and a tiny set of new numbers carries each task's changes.

What the paper claims

• Compared with full fine-tuning of GPT-3 175B with Adam, LoRA cuts the number of trainable parameters by 10,000× and the GPU memory needed by 3×.
• Quality is on par with or better than full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3.
• No extra delay when the model runs, unlike adapter layers, because the booklet can be folded back into the book.

Why it matters today

LoRA is the default way to customize open-weight language models. When someone ships a “LoRA” for a model, they are shipping a few megabytes that turn a general model into a specialist.

1 Introduction · original

“We hypothesize that the change in weights during model adaptation also has a low ‘intrinsic rank’, leading to our proposed Low-Rank Adaptation (LoRA) approach.”Hu et al. (2021), §1

Everyday picture

Adapting a model that already knows English to write SQL, or to summarize conversations, probably doesn't require rewriting everything it knows. It needs a few focused adjustments. The authors bet that the change to each weight matrix can be described with very few independent “directions” of change. That count of directions is what rank means.

Tiny example: a big table from two short lists

Take a column of 4 numbers, B = (1, 0, 2, 0), and a row of 4 numbers, A = (0.5, 1, 0, −1). Multiply every entry of B by every entry of A and you get a full 4 × 4 table, 16 numbers from only 8:

Every row is a multiple of (0.5, 1, 0, −1), so the table has rank 1. At the scale of a real model, a 4,096 × 4,096 weight matrix has 16.8 million entries, but a rank-1 change to it needs only 4,096 + 4,096 = 8,192 numbers. That ratio, 2,048 to 1, is the whole trick.

Four advantages the paper lists

• Many tasks, one model: keep one frozen copy and swap small A and B pairs per task.
• Cheaper training: no gradients or optimizer state for the frozen weights, which lowers the hardware needed by up to 3×.
• No inference latency: the update can be added into the weights before serving.
• Composable: it combines with other methods such as prefix tuning.

Why it matters

The claim is not only “cheaper”. It is that the useful part of fine-tuning really is low-dimensional, which §7 tests directly.

2 Problem statement · original

Everyday picture

Fine-tuning means showing the model many examples of a task, such as a question paired with its SQL, and nudging its weights so that the right answer becomes more likely, one token at a time.

In words: “adjust all the weights so that, across every training pair, the model gives as much probability as possible to each correct next token of the target, given the prompt and the target tokens so far.”

With the numbers: if the target SQL is 3 tokens long and the model gives them probabilities 0.5, 0.8 and 0.9, this pair contributes ln 0.5 + ln 0.8 + ln 0.9 = −0.69 − 0.22 − 0.11 = −1.02. Training pushes that number up towards 0.

In Python:

import math
# P_Φ(y_t | x, y_<t) for t = 1, 2, 3
P = [0.5, 0.8, 0.9]
[round(math.log(p), 2) for p in P]  # → [-0.69, -0.22, -0.11]
# Σ over t of log P
round(sum(math.log(p) for p in P), 2)  # → -1.02

The problem: full fine-tuning learns a change ΔΦ as large as the model itself. For GPT-3 that is about 175 billion numbers per task. LoRA instead describes the change with a much smaller set of numbers Θ; for GPT-3, Θ can be as small as 0.01% of the model.

Why it matters

Storage and switching cost scale with the size of what you train. Shrink that, and serving hundreds of custom models from one base becomes practical. See the training stages lesson for where fine-tuning sits among pretraining, SFT and preference tuning.

3 Why not adapters or prompts? · original

Everyday picture

Two earlier ways to avoid full fine-tuning each had a catch. Adapters insert small extra layers into the network: like adding extra stops on a train line, every journey now takes longer, and at a batch size of 1 the stops dominate. Prefix tuning learns special tokens that sit in the prompt: like reserving seats on the train for luggage, it takes up space the passengers (your actual input) could have used, and the paper found it hard to optimize.

The latency cost, measured

With big batches the overhead shrinks (about 2 to 3% at batch 32), because the GPU is busy anyway. Online serving usually has small batches, where adapters hurt most.

Why it matters

The goal was an adaptation method with no serving penalty and no loss of usable context. That design constraint is what led to a change that can be merged into the existing weights.

4 The method · original

# d × r column, with r = 1
B = [1, 0, 2, 0]
# r × d row
A = [0.5, 1, 0, -1]
x = [2, 1, 0, 1]
# α = r, so the scale α/r is 1
alpha, r = 1, 1
# squeeze d numbers down to r = 1
Ax = sum(a_j * x_j for a_j, x_j in zip(A, x))
Ax  # → 1.0
# expand: the correction (α/r)·BAx
[alpha / r * b_i * Ax for b_i in B]  # → [1.0, 0.0, 2.0, 0.0]

B = [1, 0, 2, 0]
A = [0.5, 1, 0, -1]
# the identity
W0 = [[1 if i == j else 0 for j in range(4)] for i in range(4)]
# W = W₀ + BA
W = [[W0[i][j] + B[i] * A[j] for j in range(4)] for i in range(4)]
# still 4 × 4
len(W), len(W[0])  # → (4, 4)
x = [2, 1, 0, 1]
# W x
[sum(W[i][j] * x[j] for j in range(4)) for i in range(4)]  # → [3.0, 1.0, 2.0, 1.0]

5 Experiments · original

Everyday picture

The test is simple: does the booklet teach as well as reprinting the book? The paper compares LoRA with full fine-tuning and other cheap methods on language understanding (the GLUE benchmark), text generation (E2E NLG) and three GPT-3 tasks: turning questions into SQL (WikiSQL), judging whether one sentence follows from another (MultiNLI) and summarizing chats (SAMSum).

Reading it: each row pair compares full fine-tuning with LoRA on the same model. The bars show trainable parameters on a log scale (a bar's length follows how many digits its number has), so even a modest gap in length means hundreds of times fewer parameters. LoRA trains between a hundred and tens of thousands of times fewer numbers and lands within a point of full fine-tuning, often above it. Scores come from different benchmarks, so compare rows within a pair, not across pairs.

One more finding: not every cheap method improves with more trainable parameters. Prefix methods got worse past a few dozen to a few hundred special tokens, while LoRA stayed stable.

Why it matters

“As good as full fine-tuning with a fraction of the parameters” is the result that made LoRA the default. The metrics lesson explains BLEU and ROUGE, two of the scores used here.

7 Understanding the low-rank updates · original

Having shown that LoRA works, the paper asks why. All three studies use GPT-3 175B.

# top direction of the r = 8 run (length 1)
u = [0.8, 0.6]
# top direction of the r = 64 run (length 1)
v = [1, 0]
i = j = 1
# Uᵀ U is a 1 × 1 table here
overlap = sum(u_k * v_k for u_k, v_k in zip(u, v))
# ‖·‖_F² / min(i, j)
round(overlap ** 2 / min(i, j), 2)  # → 0.64

8 Conclusion · original

The authors conclude that LoRA keeps quality while removing inference latency and cutting the input-length cost of prompt methods, and that it lets you switch tasks quickly on a shared model. Their open questions: combining LoRA with other methods, understanding the mechanism more deeply, choosing which matrices to adapt on a principled basis, and whether W itself is rank-deficient.

QLoRA (Dettmers et al., 2023) · original

“We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.”Dettmers et al. (2023), Abstract

Everyday picture

LoRA shrank what you train, but you still had to hold the whole frozen model in GPU memory at 16 bits per number. QLoRA stores the frozen book in shorthand, 4 bits per number, and writes the corrections booklet in full precision. When a page is needed, it is expanded from shorthand on the fly.

Try it: how much GPU memory does fine-tuning need?

Reading it: each bar is the memory for the model's weights and training state, on a log scale; the dashed marker is a single 48 GB GPU. At 65B parameters the rule of thumb gives 780 GB for full fine-tuning, matching the paper's “more than 780 GB”, and about 34 GB for QLoRA's frozen weights, which leaves room on one 48 GB card for the adapters and activations. Slide down to 7B and even 16-bit LoRA fits on one GPU.

QLoRA adds three tricks, described next: a 4-bit data type shaped for neural network weights, quantizing the quantization constants, and paging optimizer memory to the CPU.

4-bit NormalFloat (NF4) · original

Everyday picture

With 4 bits you get only 16 possible values. Suppose you must describe everyone's height with just 16 labels. Spacing the labels evenly from shortest to tallest wastes most of them on rare extremes, while crowds of average-height people share a few labels. Better: space the labels so each covers the same number of people. Neural network weights follow a bell curve, so NF4 puts its 16 levels where the weights actually are.

Tiny example

Take a block of weights (0.02, −0.10, 0.05, 0.31). Divide by the largest magnitude, 0.31, so they fit in [−1, 1]: (0.065, −0.323, 0.161, 1.0). Round each to the nearest NF4 level: 0.080, −0.284, 0.161, 1.0. Store the four level numbers (4 bits each) plus the one scale, 0.31. To use a weight, multiply back: 0.080 × 0.31 = 0.025, close to the original 0.02.

In words: “cut the bell curve into equal-probability slices and put each level halfway between two neighbouring cut points.”

With the numbers: with k = 4 there are 2⁴ = 16 levels and 2⁴ + 1 = 17 slices. Straight from the formula, q₈ sits halfway between the cut points Q_X(8/17) = −0.074 and Q_X(9/17) = 0.074, at 0, and its neighbour q₉ = 0.148 is close by; out in the tail, q₁₄ = 1.058 and q₁₅ = 1.376 are 0.318 apart. After a small fix so that zero is represented exactly, and rescaling into [−1, 1], the paper's NF4 levels run −1, −0.696, −0.525, −0.395, −0.284, −0.185, −0.091, 0, 0.080, 0.161, 0.246, 0.338, 0.441, 0.563, 0.723, 1: dense near zero, sparse at the tails.

In Python:

from statistics import NormalDist
# Q_X: the bell curve's quantile function
Q_X = NormalDist().inv_cdf
k = 4
# halfway between two neighbouring cut points
def q(i):
    return (Q_X(i / (2**k + 1)) + Q_X((i + 1) / (2**k + 1))) / 2
round(Q_X(8 / 17), 3), round(Q_X(9 / 17), 3)  # → (-0.074, 0.074)
# near zero: levels packed tight
round(q(8), 3), round(q(9), 3)  # → (0.0, 0.148)
# in the tail: levels spread out
round(q(14), 3), round(q(15), 3)  # → (1.058, 1.376)

Reading it: the grey bars are 25,600 weights drawn from a bell curve (computed live in your browser with a fixed seed), each block of 64 divided by its largest magnitude, which is how QLoRA normalizes them. The vertical lines are the 16 levels every weight must be rounded to. Toggle between NF4 and evenly spaced levels and read the error: NF4 packs levels where the bars are tall, so the average rounding error is lower and the 16 levels are used more evenly. With so few levels, where you put them matters.

Perplexity measures how surprised a model is by real text. Same 4 bits, better placement, a less surprised model.

Double quantization · original

Everyday picture

Every block of 64 weights needs its own scale factor (the 0.31 in the tiny example), stored as a 32-bit number. That overhead adds up, so QLoRA compresses the scale factors too: a note about the notes.

In words: “one 32-bit scale per 64 weights costs half a bit per weight; storing those scales in 8 bits, with one 32-bit scale per 256 of them, cuts it to about an eighth of a bit.”

With the numbers: the saving is 0.5 − 0.127 = 0.373 bits per parameter. For a 65B model that is 65 × 10⁹ × 0.373 / 8 ≈ 3 GB, which the paper also reports.

In Python:

# one 32-bit scale per 64 weights
before = 32 / 64
# 8-bit scales, plus one 32-bit scale per 256 of them
after = 8 / 64 + 32 / (64 * 256)
before, round(after, 3)  # → (0.5, 0.127)
# bits saved per parameter
round(before - after, 3)  # → 0.373
# bytes saved on a 65B model, in GB
round(65e9 * 0.373 / 8 / 1e9, 1)  # → 3.0

Paged optimizers · original

Everyday picture

A long training example can briefly spike memory use and crash the run. Paged optimizers keep the optimizer's state in memory that the GPU driver can move to ordinary CPU RAM when the GPU runs short, and page it back when needed, the way an operating system swaps memory to disk.

Why it matters

It turns rare out-of-memory crashes into a small slowdown, which is what makes “fine-tune a 65B model on one GPU” reliable rather than lucky.

QLoRA in one equation · original

In words: “expand the 4-bit frozen weights back to 16-bit on the fly, multiply the input by them, and add the LoRA correction computed in 16-bit.”

With the numbers: it is §4.1's equation with W₀ replaced by its decompressed 4-bit copy. In the tiny example, the stored level 0.080 and scale 0.31 are expanded to 0.025 just before the multiply. Gradients flow through that decompressed copy but are only kept for L₁ and L₂.

In Python:

# a W^NF4 entry and its block's scale
level, scale = 0.080, 0.31
# doubleDequant: back to a usable 16-bit weight
w = level * scale
round(w, 3)  # → 0.025

Results · original

What the paper shows

• No quality loss from 4 bits: with NF4 and double quantization, QLoRA matched 16-bit full fine-tuning and 16-bit LoRA on GLUE and Super-NaturalInstructions, and matched 16-bit LoRA on 5-shot MMLU for LLaMA 7B to 65B (mean 53.1 versus 53.0).
• Adapter coverage beats rank: putting LoRA on every linear layer of each block was needed to match full fine-tuning; the rank r mattered little.
• 65B on one GPU: memory for fine-tuning a 65B model drops from more than 780 GB to under 48 GB.
• Guanaco: their best model reached 99.3% of ChatGPT's score on the Vicuna benchmark after 24 hours of fine-tuning on a single GPU.
• Data quality over quantity: a 9,000-example dataset (OASST1) beat a 450,000-example one (a subsample of FLAN v2) for chatbot quality.

The authors also warn that chatbot benchmarks judged by another model are not fully trustworthy, and they publish examples of where their model fails.

Why it matters today

QLoRA made fine-tuning large open models a single-GPU job, and “bigger model at lower precision beats smaller model at higher precision” became a common rule of thumb. The inference lesson builds int8 and int4 quantization from scratch.

What changed since

Glossary

Every term with hover guidance on this page, in one place.