LoRA: low-rank adapters, rank and alpha, trained on a laptop
LoRA bets that the change fine-tuning needs is simple enough to write as the product of two thin matrices. The original weights stay frozen and the layer's output becomes the original output plus the thin pair's. Training only the pairs updates well under 1% of the model, which fits on a laptop.
What this page answers
How does LoRA work?
It freezes each weight matrix and adds the product of two thin trainable matrices, such as 896 by 8 and 8 by 896, to the layer's output. Only those thin pairs are trained. Read it in “Train a small patch, not the model”
Why does LoRA start with no effect on the model?
One of its two matrices starts at zero, so before training the add-on contributes nothing and the model behaves exactly as before. Read it in “LoRA in a dozen lines”
Train a small patch, not the model
Each layer of a transformer multiplies its input by big weight matrices: in this model, the attention's query projection is 896 by 896, about 800,000 numbers. LoRA's bet is that the change fine-tuning needs is simple, so it can be written as the product of two thin matrices: one 896 by 8 and one 8 by 896, about 14,000 numbers. The original matrix stays frozen; the layer's output becomes the original output plus the thin pair's output. Train only the thin pairs, and you train well under 1% of the model.
LoRA in a dozen lines
This is the exact code used for every fine-tune in this track. B starts at zero, so before training the add-on contributes nothing and the model behaves exactly as it did.
import math, torch, torch.nn as nn
class LoRA(nn.Module):
def __init__(self, base, r=8, alpha=16):
super().__init__()
self.base, self.scale = base, alpha / r # base: the frozen nn.Linear
self.A = nn.Parameter(torch.randn(r, base.in_features) / math.sqrt(base.in_features))
self.B = nn.Parameter(torch.zeros(base.out_features, r)) # zero: starts as a no-op
def forward(self, x):
y = self.base(x)
return y + ((x.float() @ self.A.T) @ self.B.T * self.scale).to(y.dtype)
model.requires_grad_(False) # freeze everything
for layer in model.model.layers:
layer.self_attn.q_proj = LoRA(layer.self_attn.q_proj)
layer.self_attn.v_proj = LoRA(layer.self_attn.v_proj)
params = [p for m in model.modules() if isinstance(m, LoRA) for p in (m.A, m.B)]
opt = torch.optim.AdamW(params, lr=2e-4)Teaching a habit
The habit: given a bare product review with no instructions at all, reply with exactly one word, positive or negative. In Track E, lesson E3, a bigger model needed instructions to do this. I fine-tuned on 32 reviews I wrote, for 2 passes, then tested on the 40 different reviews from lesson E3.
Predict first. After 64 training steps on 32 examples, how many of the 40 unseen bare reviews get a one-word label?
- 30 or more
- 10 to 29
- fewer than 10
Answer. 40 of 40, up from 0 before training. A habit is easy to teach: the model already knew what positive and negative reviews look like; it only had to learn the reply format.
How to read the charts: left, the training loss at each step (how surprised the model was by the correct one-word answer; lower is better), with a smoothed line. Right, out of 40 unseen reviews, how many replies were exactly one label word (outlined) and how many were the right label (solid), before and after.

The loss fell from 7.4 over the first four steps to 0.02 over the last eight. Before training, the bare reviews got replies like “I'm glad you found the product you were looking for. If you have any”; after, “positive”. Of the 40 one-word replies, 31 had the right label: the format was learned completely, the judgement less so. But look at what else changed. On 20 general-knowledge questions the untouched model scored 14; after this fine-tune, 3. The habit didn't stay with reviews. Asked “Who wrote Romeo and Juliet?”, the tuned model now says “English”; asked “What is the square root of 81?”, it says “positive”. Sixty-four training examples, every one answered with a single word, taught “answer with one word” to the whole model. That's catastrophic forgetting, measured, and lesson B5 shows how to catch it; mixing some ordinary examples into the training data is the usual fix.
Continue the lesson
The whole lesson, with its drawings and its questions. The rest of this section and 2 more after it.

