How LLM Fine-Tuning Works: Adapting a Base Model to a Task

What Fine-Tuning Is

Fine-tuning is more training, starting from the pretrained weights, on a much smaller dataset chosen for a specific behavior. The machinery is the same as pretraining — next-token prediction, cross-entropy, backpropagation, AdamW — but four things change: the starting point is a trained model instead of random weights, the data is curated examples instead of the raw web, the loss usually counts only the response, and training runs for thousands of steps instead of hundreds of thousands.

The most important thing to understand is what fine-tuning mostly teaches. The LIMA study fine-tuned a 65B-parameter LLaMA model on just 1,000 carefully curated prompt–response pairs (about 750,000 tokens), with no reinforcement learning, and in a controlled human study its answers were equivalent to or preferred over GPT-4's in 43% of cases. The authors' conclusion: almost all of a model's knowledge comes from pretraining, and fine-tuning mainly teaches it which format and style to use when drawing on that knowledge. OpenAI's InstructGPT work points the same way from another angle: after supervised fine-tuning on demonstrations plus reinforcement learning from human feedback, outputs from a 1.3B model were preferred by human raters over the original 175B GPT-3's.

Define the Goal

Common goals are instruction following (turning a base model into an assistant), domain adaptation (legal, medical, a company's own documents), a narrow task (classification, extraction, summarization in a fixed format), a consistent voice or format, or tool use. Before choosing fine-tuning, it's worth checking the cheaper options: a well-written prompt with examples often gets a capable model most of the way, and retrieval (fetching relevant documents at query time) is usually a better way to give a model facts that change. Fine-tuning shines at behavior — format, style, task structure — and at making a smaller, cheaper model match a larger one on a narrow job.

Build the Dataset

The dataset is a list of examples, each a prompt (optionally with a system message and context) and the ideal response. Typical sources are public instruction datasets, human-written demonstrations, domain documents turned into question–answer pairs, logs of real conversations, and responses generated by a stronger model and then filtered. Google's FLAN showed how much the mix of tasks matters: instruction-tuning a 137B model on over 60 NLP datasets, each phrased as natural-language instructions, improved zero-shot performance on tasks it had never seen and beat zero-shot 175B GPT-3 on 20 of 25 evaluations.

Curation matters more than volume — the LIMA result is the extreme case. The usual cleaning steps are the ones from pretraining, applied with more care: remove duplicates, fix broken formatting, drop low-quality or unsafe responses, make the style consistent, and remove any examples that overlap with your evaluation set, or your scores will be inflated by memorization.

Chat Templates and Loss Masking

The template

Each example is rendered into one token sequence with a fixed chat template: special tokens that mark where the system message, the user turn and the assistant turn begin and end.

The exact markers differ between model families. What matters is to use the same template in training and at inference, and to tokenize with the base model's own tokenizer, since its embeddings only mean something for the tokens it was trained on. Our experiment uses the simplest possible template, Q: …\nA: …, ending with the end-of-text token the base model already knows.

Labels and the prompt mask

Labels are built exactly as in pretraining — each position's target is the next token — with one crucial change: positions in the prompt get the label −100, which PyTorch's cross-entropy ignores. The model still reads the whole prompt (attention is unchanged), but it's only graded on the response. Otherwise it would spend capacity learning to predict users' questions, which is not the skill you want. Padding added to make batch rows the same length is masked the same way. Hover over the tokens:

In our run, the eight training examples contain 239 prompt tokens, all masked, and 117 response tokens that carry the entire training signal.

Full Fine-Tuning vs LoRA

Full fine-tuning updates every weight: θ ← θ − η·∇L, starting from θ₀. It's the most expressive option, but it needs the same memory as training — weights, gradients and AdamW's two moments for every parameter — and it produces a complete new copy of the model for each task. For a model the size of Llama 2 7B (6.74B parameters) with bf16 weights and fp32 optimizer state, that's roughly 16 bytes per parameter, about 108 GB before activations.

Parameter-efficient fine-tuning (PEFT) freezes the base model and trains a small number of new parameters. Adapters, one of the first approaches, insert small bottleneck layers into each block and came within 0.4% of full fine-tuning on the GLUE benchmark while adding 3.6% parameters per task. The approach that took over is LoRA.

How LoRA works

LoRA's bet is that the change a fine-tune needs to make to a weight matrix is low-rank — it can be written as the product of two thin matrices. For a frozen weight W (d_out × d_in), LoRA adds B (d_out × r) and A (r × d_in) with a small rank r:

The savings are large. The LoRA paper reports that, compared with fully fine-tuning GPT-3 175B with Adam, LoRA reduced trainable parameters by 10,000× and GPU memory by 3×. On a Llama-2-7B-shaped model, rank-16 adapters on the four attention projections of all 32 layers come to 16,777,216 parameters — 0.25% of the model. QLoRA goes further: it stores the frozen base model in a 4-bit format (NF4) and backpropagates through it into LoRA adapters, which made it possible to fine-tune a 65B model on a single 48 GB GPU while matching 16-bit fine-tuning performance.

Rough memory for weights and optimizer state only (activations come on top), for a 6.74B-parameter model:

Estimates from the arithmetic in CS336 Part 3: 16 bytes/parameter for full fine-tuning; 2 bytes/parameter for a frozen bf16 base or about 0.5 for 4-bit, plus full training state for the 16.8M adapter parameters. Real frameworks add overhead.

One Fine-Tuning Step

Each step is the familiar training loop, with the prompt mask and the frozen weights as the only differences:

1. Batch of formatted examples(B, T)Token IDs from the chat template, plus labels with the prompt set to −100.
2. Forward pass(B, T, V)Through the pretrained model (with adapters added, for LoRA) to logits. The model always sees the true previous tokens: teacher forcing.
3. Compare with target labels(B, T)Only response positions have real labels.
4. Cross-entropy on response tokensscalarL = (1/|Y|) · Σ over the graded positions only.
5. Backpropagation∇LGradients for every trainable parameter: all of them in full fine-tuning; only A and B with LoRA.
6. Optimizer update (AdamW)θ ← …With the learning-rate schedule and weight decay.

Hyperparameters

Fine-tuning settings differ from pretraining in consistent ways, all pointing toward gentler training — you're adjusting a model, not building one:

These are common starting points rather than rules; the right values depend on the model, the data and the method, and are found with small sweeps like the one below.

Forgetting

Fine-tuning on a narrow task can erode abilities the base model had — catastrophic forgetting. A careful study, LoRA Learns Less and Forgets Less, compared the two methods on programming and math with both instruction data and continued pretraining. In standard low-rank settings LoRA substantially underperformed full fine-tuning on the target domain, but it better preserved performance outside it, and mitigated forgetting more than weight decay or dropout did. Full fine-tuning learned weight changes with a rank 10–100× higher than typical LoRA configurations, which may explain the gap.

We measured forgetting in our experiment by tracking the model's loss on held-out text in the style of its pretraining data, before and after fine-tuning (base model: 0.146), across three learning rates for each method:

Three things show up, and one of them is a useful surprise:

Common ways to limit forgetting: a lower learning rate and fewer epochs, parameter-efficient methods, and mixing some general-purpose data back into the fine-tuning set.

Evaluation and Choosing a Checkpoint

Track validation loss on held-out examples during training, but judge the model on the task: accuracy for classification, ROUGE or BLEU for summarization and translation (both measure overlap with reference text, and both miss a lot), and human or model-graded evaluation for open-ended responses. Save checkpoints regularly and keep the best by validation metric rather than the last; stopping early when the metric stops improving is the simplest defense against overfitting.

Our experiment shows why the evaluation set must contain unseen examples. Both models answered all eight training questions exactly. Asked about the four facts they were never quizzed on, they answered with fragments of training answers: "When was the wheat harvested?" got "red ball." and "What covered the mountain?" got "the east." They had learned the question-and-answer format, and memorized eight answers, but not how to look up the facts they'd seen in pretraining. A model with billions of parameters and a broad fine-tuning set does learn that mapping, which is exactly the LIMA and FLAN result. Only a held-out test tells you which situation you're in.

What Comes Out

The result is a task-adapted model. With full fine-tuning it's a new set of weights; with LoRA it's a small adapter file — often a few megabytes to a few hundred — that can be merged into the base weights for serving, or kept separate so one base model can switch between many adapters. For assistants, supervised fine-tuning is usually the first post-training stage, followed by preference-based training (reinforcement learning from human feedback, or direct preference optimization) to refine helpfulness and safety — the recipe InstructGPT established.

Run the Experiment

The whole comparison in one script, on a laptop CPU in about two minutes: pretrain an 843K-parameter base model on twelve facts; build question–answer examples with prompt masking; implement LoRA from scratch (random A, zero B, α/r scaling, applied to every linear layer); fine-tune both ways on 8 facts; test on the other 4; and sweep learning rates to measure forgetting. Needs PyTorch 2.x.

"""Pretrain a tiny model on facts, then fine-tune it to answer questions: full fine-tuning vs LoRA."""
import copy, itertools, math, random
import torch
import torch.nn as nn
import torch.nn.functional as F

random.seed(0); torch.manual_seed(0)
torch.set_num_threads(4)

# ---------------- Step 0: a pretrained base model (condensed from the pretraining article) ----------------
facts = [
    ("The cat sat on the mat and watched the rain.",       "Where did the cat sit?",        "On the mat."),
    ("The dog chased a red ball across the park.",         "What did the dog chase?",       "A red ball."),
    ("Water boils at one hundred degrees Celsius.",        "When does water boil?",         "At one hundred degrees."),
    ("The sun rises in the east every morning.",           "Where does the sun rise?",      "In the east."),
    ("Bees collect nectar and turn it into honey.",        "What do bees make?",            "Honey."),
    ("The river flows into the quiet lake.",               "Where does the river flow?",    "Into the quiet lake."),
    ("Children read books in the library after school.",   "Where do children read?",       "In the library."),
    ("Trains leave the station every hour.",               "How often do trains leave?",    "Every hour."),
    ("The farmer harvested the wheat in autumn.",          "When was the wheat harvested?", "In autumn."),
    ("Birds build their nests in tall trees.",             "Where do birds build nests?",   "In tall trees."),
    ("The moon pulls on the ocean and causes the tides.",  "What causes the tides?",        "The moon."),
    ("Snow covered the mountain before the hikers left.",  "What covered the mountain?",    "Snow."),
]
EOT, V, T = 256, 257, 128
enc = lambda s: list(s.encode())
dec = lambda ids: bytes(i for i in ids if i < 256).decode(errors="replace")

pages = [" ".join(f for f, _, _ in c) for c in itertools.permutations(facts, 3)]
random.shuffle(pages)
stream = torch.tensor([t for p in pages[:600] for t in enc(p) + [EOT]])
heldout_text = torch.tensor([t for p in pages[600:700] for t in enc(p) + [EOT]])   # for measuring forgetting

d, H, L = 128, 4, 4
class Block(nn.Module):
    def __init__(self):
        super().__init__()
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.qkv, self.proj = nn.Linear(d, 3 * d), nn.Linear(d, d)
        self.up, self.down = nn.Linear(d, 4 * d), nn.Linear(4 * d, d)
    def forward(self, x):
        B_, T_, _ = x.shape
        q, k, v = self.qkv(self.ln1(x)).split(d, dim=-1)
        q, k, v = (t.view(B_, T_, H, d // H).transpose(1, 2) for t in (q, k, v))
        x = x + self.proj(F.scaled_dot_product_attention(q, k, v, is_causal=True).transpose(1, 2).reshape(B_, T_, d))
        return x + self.down(F.gelu(self.up(self.ln2(x))))
class GPT(nn.Module):
    def __init__(self):
        super().__init__()
        self.tok, self.pos = nn.Embedding(V, d), nn.Embedding(T, d)
        self.blocks = nn.Sequential(*[Block() for _ in range(L)])
        self.ln_f, self.head = nn.LayerNorm(d), nn.Linear(d, V, bias=False)
        self.head.weight = self.tok.weight
        for m in self.modules():
            if isinstance(m, (nn.Linear, nn.Embedding)): nn.init.normal_(m.weight, std=0.02)
            if isinstance(m, nn.Linear) and m.bias is not None: nn.init.zeros_(m.bias)
    def forward(self, idx):
        return self.head(self.ln_f(self.blocks(self.tok(idx) + self.pos(torch.arange(idx.shape[1])))))

def text_loss(model, s, n=30):
    model.eval()
    with torch.no_grad():
        starts = torch.randint(0, len(s) - T - 1, (n,), generator=torch.Generator().manual_seed(1))
        x = torch.stack([s[i:i + T] for i in starts]); y = torch.stack([s[i + 1:i + T + 1] for i in starts])
        return F.cross_entropy(model(x).reshape(-1, V), y.reshape(-1)).item()

base = GPT()
opt = torch.optim.AdamW(base.parameters(), lr=2e-3, weight_decay=0.1)
for step in range(500):
    i = torch.randint(0, len(stream) - T - 1, (32,))
    x = torch.stack([stream[j:j + T] for j in i]); y = torch.stack([stream[j + 1:j + T + 1] for j in i])
    loss = F.cross_entropy(base(x).reshape(-1, V), y.reshape(-1))
    opt.zero_grad(); loss.backward(); torch.nn.utils.clip_grad_norm_(base.parameters(), 1.0); opt.step()
print(f"base model: {sum(p.numel() for p in base.parameters()):,} params, held-out text loss {text_loss(base, heldout_text):.3f}")

# ---------------- Steps 3-9: task data -> training examples -> tokens -> labels with prompt masking ----------------
train_qa, test_qa = facts[:8], facts[8:]
def example(q, a):
    prompt, answer = enc(f"Q: {q}\nA: "), enc(a) + [EOT]
    ids = prompt + answer
    labels = [-100] * len(prompt) + answer                               # loss only on the answer
    return ids[:-1], labels[1:]                                          # shift: predict token t+1
batch = [example(q, a) for _, q, a in train_qa]
width = max(len(x) for x, _ in batch)
X = torch.tensor([x + [EOT] * (width - len(x)) for x, _ in batch])
Y = torch.tensor([y + [-100] * (width - len(y)) for _, y in batch])      # padding is ignored too
n_prompt = sum(len(enc(f"Q: {q}\nA: ")) for _, q, _ in train_qa)
print(f"SFT batch {tuple(X.shape)}: {(Y != -100).sum().item()} answer tokens train, {n_prompt} prompt tokens masked")

def ask(model, q, max_new=40):
    ids = enc(f"Q: {q}\nA: ")
    model.eval()
    with torch.no_grad():
        for _ in range(max_new):
            nxt = model(torch.tensor([ids]))[0, -1].argmax().item()      # greedy decoding
            if nxt == EOT: break
            ids.append(nxt)
    return dec(ids).split("A: ", 1)[1]

print("\nbase model, before fine-tuning:")
for _, q, a in facts[:2] + facts[8:9]:
    print(f"  Q: {q}  ->  {ask(base, q)!r}")

# ---------------- Step 6: the two tuning methods ----------------
class LoRALinear(nn.Module):
    """y = W x + (alpha / r) * B A x, with W frozen, A random, B zero (so training starts at the base model)."""
    def __init__(self, base_linear, r=4, alpha=8):
        super().__init__()
        self.base = base_linear.requires_grad_(False)
        self.A = nn.Parameter(torch.randn(r, base_linear.in_features) / math.sqrt(base_linear.in_features))
        self.B = nn.Parameter(torch.zeros(base_linear.out_features, r))
        self.scale = alpha / r
    def forward(self, x):
        return self.base(x) + self.scale * (x @ self.A.T @ self.B.T)

def add_lora(model):
    model.requires_grad_(False)                                          # freeze everything...
    for blk in model.blocks:                                             # ...then add adapters to every linear
        for name in ("qkv", "proj", "up", "down"):
            setattr(blk, name, LoRALinear(getattr(blk, name)))
    return model

# ---------------- Steps 10-17: fine-tune, evaluate ----------------
def finetune(model, lr, steps=150):
    params = [p for p in model.parameters() if p.requires_grad]
    opt = torch.optim.AdamW(params, lr=lr, weight_decay=0.0)
    for step in range(steps):
        warm = min(1.0, (step + 1) / 15)                                 # short warmup, then constant
        for g in opt.param_groups: g["lr"] = lr * warm
        loss = F.cross_entropy(model.train()(X).reshape(-1, V), Y.reshape(-1), ignore_index=-100)
        opt.zero_grad(); loss.backward(); torch.nn.utils.clip_grad_norm_(params, 1.0); opt.step()
    return sum(p.numel() for p in params), loss.item()

for name, make, lr in [("full fine-tuning", lambda: copy.deepcopy(base), 1e-4),
                       ("LoRA (r=4)", lambda: add_lora(copy.deepcopy(base)), 1e-3)]:
    torch.manual_seed(0); model = make()
    n_train, final = finetune(model, lr)
    total = sum(p.numel() for p in model.parameters())
    seen = sum(ask(model, q) == a for _, q, a in train_qa)
    print(f"\n{name}: trains {n_train:,} of {total:,} params ({100 * n_train / total:.1f}%), lr {lr}")
    print(f"  final SFT loss {final:.3f} | train questions exactly right: {seen}/8")
    print(f"  held-out text loss {text_loss(model, heldout_text):.3f} (base: {text_loss(base, heldout_text):.3f})")
    for (_, q, a), got in zip(test_qa, (ask(model, q) for _, q, _ in test_qa)):
        print(f"  unseen Q: {q:<30} -> {got!r}   (expected {a!r})")

# ---------------- Forgetting vs learning rate ----------------
print("\nlearning-rate sweep (held-out text loss = forgetting; base = %.3f):" % text_loss(base, heldout_text))
for name, make, lr in [("full", lambda: copy.deepcopy(base), 1e-4), ("full", lambda: copy.deepcopy(base), 3e-4),
                       ("full", lambda: copy.deepcopy(base), 1e-3), ("LoRA", lambda: add_lora(copy.deepcopy(base)), 3e-4),
                       ("LoRA", lambda: add_lora(copy.deepcopy(base)), 1e-3), ("LoRA", lambda: add_lora(copy.deepcopy(base)), 3e-3)]:
    torch.manual_seed(0); model = make()
    _, final = finetune(model, lr)
    seen = sum(ask(model, q) == a for _, q, a in train_qa)
    print(f"  {name:<4} lr {lr:<7} train {seen}/8   SFT loss {final:.3f}   held-out text loss {text_loss(model, heldout_text):.3f}")

Output (PyTorch 2.14 on CPU; multithreaded CPU math can shift later decimals between runs):

base model: 842,624 params, held-out text loss 0.146
SFT batch (8, 53): 117 answer tokens train, 239 prompt tokens masked

base model, before fine-tuning:
  Q: Where did the cat sit?  ->  'coneat con thon mun. The river flows int'
  Q: What did the dog chase?  ->  'coneat con the mat an watched the rain.'
  Q: When was the wheat harvested?  ->  'coneatat con the mat and watched the rai'

full fine-tuning: trains 842,624 of 842,624 params (100.0%), lr 0.0001
  final SFT loss 0.070 | train questions exactly right: 8/8
  held-out text loss 0.558 (base: 0.146)
  unseen Q: When was the wheat harvested?  -> 'red ball.'   (expected 'In autumn.')
  unseen Q: Where do birds build nests?    -> 'the library.'   (expected 'In tall trees.')
  unseen Q: What causes the tides?         -> 'y.'   (expected 'The moon.')
  unseen Q: What covered the mountain?     -> 'the east.'   (expected 'Snow.')

LoRA (r=4): trains 32,768 of 875,392 params (3.7%), lr 0.001
  final SFT loss 0.214 | train questions exactly right: 8/8
  held-out text loss 0.929 (base: 0.146)
  unseen Q: When was the wheat harvested?  -> 'eastey.'   (expected 'In autumn.')
  unseen Q: Where do birds build nests?    -> 'the mast.'   (expected 'In tall trees.')
  unseen Q: What causes the tides?         -> 'east.'   (expected 'The moon.')
  unseen Q: What covered the mountain?     -> 'the east.'   (expected 'Snow.')

learning-rate sweep (held-out text loss = forgetting; base = 0.146):
  full lr 0.0001  train 8/8   SFT loss 0.070   held-out text loss 0.558
  full lr 0.0003  train 8/8   SFT loss 0.009   held-out text loss 0.954
  full lr 0.001   train 8/8   SFT loss 0.001   held-out text loss 2.508
  LoRA lr 0.0003  train 5/8   SFT loss 0.323   held-out text loss 0.714
  LoRA lr 0.001   train 8/8   SFT loss 0.214   held-out text loss 0.929
  LoRA lr 0.003   train 8/8   SFT loss 0.205   held-out text loss 1.376

Before fine-tuning, the base model can't answer at all — asked "Where did the cat sit?" it rambles on in the style of its pretraining text. After 150 steps on eight examples, both methods produce short, correctly formatted answers and get all eight right. LoRA did it while training 32,768 parameters: 3.7% of the model, a far larger fraction than on a real LLM because this model is tiny.

⚠️ Notes

🔗 References