LoRA Fine-Tuning: Smaller Checkpoints, Frozen Weights

Two thin matrices beside a frozen model

Seven authors at Microsoft, one of them also at Carnegie Mellon, proposed leaving the pretrained weight matrix alone and expressing the adaptation as the product of two much thinner matrices. A weight matrix of size d by k gets a companion pair of size d by r and r by k, where r is small. Only that pair receives gradients. The B matrix starts at zero, so the adapted model begins identical to the base model, and the update is scaled by a constant over r. [1]

The cost falls because optimizer state and gradients are needed only for the small pair. The paper's headline figures are for GPT-3 at 175 billion parameters: trainable parameters reduced by a factor of ten thousand, training memory from 1.2TB to 350GB, a 25 percent training speedup, and, at rank 4 adapting only the query and value projections, a per-task checkpoint of 35MB against 350GB. Switching tasks means subtracting one pair and adding another. [1]

The experiments are narrower than the abstract's language suggests. The paper limits itself to adapting attention weights and freezes the feed-forward modules, and its ablation covers ranks of 1, 2, 4, 8 and 64, finding that very small ranks already compete. Evaluation spans RoBERTa, DeBERTa, GPT-2 and GPT-3, with the claim being parity or better against full fine-tuning rather than a quality tradeoff. The now-common practice of attaching adapters to every linear layer is later convention, not this paper's recommendation. [1]

Zero added latency, and when that stops being true

The distinguishing claim against earlier adapter methods is that this one costs nothing at inference. Because the update is a plain matrix product, it can be folded into the base weights once and served as an ordinary model of exactly the original shape. Adapter layers, by contrast, add sequential computation to every forward pass. [1]

The paper states the exception itself, in one sentence: once the matrices are absorbed, it is not straightforward to batch inputs for different tasks with different adapters in a single forward pass. The systems consequence was worked out later. S-LoRA, in 2023, argues that merging is actively the wrong choice when serving many adapters, because it forces multiple copies of the base weights and gives up batching, and instead computes the adapter term on the fly. [1][5]

Atlas interpretation: The zero-latency property and the many-adapters property are therefore alternatives, not a package. A single fine-tune served as its own model gets the former for free. A service hosting thousands of customer-specific adapters on shared hardware gives it up deliberately, which is why serving systems for exactly that shape exist as their own small field. [5][3]

How it became the default

Hugging Face released PEFT in February 2023 with this method as its lead technique, demonstrating a 3B-parameter model tuned on an 11GB consumer card with an adapter checkpoint of 19MB and under 0.2 percent of parameters trainable. Three months later QLoRA backpropagated through a frozen 4-bit quantized base model into the same adapters, reporting a 65B model fine-tuned on a single 48GB GPU while preserving 16-bit fine-tuning task performance. [3][2]

Atlas interpretation: Those two steps are what turned a 2021 paper about deployment economics at GPT-3 scale into the reason a hobbyist could fine-tune an open-weight model overnight. The method was published about twenty months before the open-weight ecosystem it is now associated with existed, and it was aimed at a different problem: a lab that did not want to store a 350GB copy of a model per customer. [1][2][3]

Where parity breaks

The parity claim is regime-dependent. A 2024 TMLR paper reports that on code and mathematics, in both instruction tuning and continued pretraining, this method substantially underperforms full fine-tuning, while forgetting less outside the target domain. It also finds that full fine-tuning learns weight changes of far higher rank than the ranks typically used in practice, which is a mechanical account of why a low-rank constraint would bind on the harder tasks. [4]

Atlas interpretation: Read together with the original results, the honest summary is that a small number of trainable parameters is sufficient for adapting style, format and task framing, and not always sufficient for teaching a genuinely new capability. The paper's own evaluation stayed on the first kind of task. [4][1]