Supervised fine-tuning

Supervised fine-tuning is next-token prediction on instruction-response text. InstructGPT is the well-known title for the broader post-training stack that starts with SFT. Your replica is smaller: pack prompts, train, and refuse to call it aligned just because loss went down.

What SFT actually does

• You concatenate instruction and answer, mask the prompt tokens in the loss if you want the model to learn the answer more than the prompt.
• You keep a held-out set that is not in the synthetic dump from week 4.
• You start from the week 5 checkpoint, not from random, unless you are studying that failure on purpose.

def sft_labels(input_ids, prompt_len, ignore_index=-100):
    labels = input_ids.clone()
    labels[..., :prompt_len] = ignore_index
    return labels

Use SyntheticInstruction rows plus your 20 human rows. If the model starts parroting the template and ignoring the question, your packing or mask is wrong. Fix the data before you reach for preference tuning.

Assignment

• Fine-tune the small LM for a short token budget. Log train loss and held-out loss.
• Sample 10 prompts from the held-out set. Read them. Mark useful / refuse / waffle. No invented win-rate.
• Write one example that got worse after SFT. That is the honest artifact.

Opinion checkpoint

Core project 5: SFT on typed synthetic plus a few human rows.

Next: Preference tuning, PPO, DPO, and RLHF intuition.