LoRA: How to Fine-Tune AI Without Retraining Everything
See how LoRA fine-tunes a huge model by training a tiny add-on instead of every weight.
Fine-tuning is powerful, but it has a practical snag. Classic fine-tuning updates every weight in the model. For a 70-billion-parameter model that means holding the weights, their gradients and the optimiser’s bookkeeping in memory at once, several times the size of the model itself. And every fine-tuned variant is another full copy: one for legal documents, one for customer support, one in your company’s tone of voice.
In 2021, Microsoft researchers proposed a shortcut called LoRA, low-rank adaptation. Leave the original weights completely frozen. Next to each big weight matrix, add a pair of thin matrices, B and A, whose product has the same shape as the big one. Train only those.
The key is the rank r: the thin dimension the two strips share. A 4,096 × 4,096 matrix holds 16.8 million numbers. With r = 16, B is 4,096 × 16 and A is 16 × 4,096: 131,072 numbers between them, under 1% of the full matrix. Their product can’t express every possible change, only fairly “simple” ones built from 16 basic patterns. But the changes fine-tuning makes usually are simple in exactly that way: you’re nudging an already capable model towards a style or a task, not teaching it the world from scratch.
The payoff is big. Training needs far less memory, so models can be fine-tuned on a single GPU, especially with a 4-bit quantized base model (a combination called QLoRA). The result is a small file, an adapter, often just tens of megabytes, that can be clipped onto the base model in an instant. One server can hold one base model and hundreds of adapters, one per customer.
Key takeaways
- Full fine-tuning updates every weight: memory-hungry, and every variant is a full copy.
- LoRA freezes the original weights and learns the change as two thin matrices, B × A.
- The rank r sets their size; a small r captures most fine-tuning changes with under 1% of the parameters.
- Adapters are small, swappable files; with a quantized base (QLoRA), fine-tuning fits on a single GPU.
Check your understanding
Go deeper (more technical)
LoRA (Hu and colleagues, 2021) adds ΔW = B·A, scaled by a factor α/r. B starts at zero, so training begins exactly from the base model. It’s usually applied to the attention matrices, often to the feed-forward ones too. After training, B·A can be merged into W, so the adapted model runs at full speed with no extra cost.
QLoRA (Dettmers and colleagues, 2023) fine-tuned a 65-billion-parameter model on a single 48 GB GPU by quantizing the frozen base to 4 bits. The simulation finds the best rank-r approximation with a singular value decomposition; real LoRA training finds its B and A with gradient descent instead.