Preference tuning, PPO, DPO, and RLHF intuition

RLHF intuition, then a method you can run. The classic stack: SFT model, a reward model on preference pairs, then PPO against that reward with a KL penalty back to the SFT policy. That stack is real and heavy. Direct Preference Optimization (DPO) is the well-known title for a method that skips the explicit RL loop and trains on pairs with a closed-form objective.

You need the intuition even if you only run DPO this week. PPO needs rollouts, a value head or advantage estimate, and careful KL. It is easy to reward-hack. DPO needs pairs: chosen and rejected completions for the same prompt. The loss pushes up the likelihood of chosen relative to rejected, under a reference model.

Coding sketch

import torch.nn.functional as F

def dpo_loss(policy_chosen, policy_rejected, ref_chosen, ref_rejected, beta=0.1):
    policy = policy_chosen - policy_rejected
    ref = ref_chosen - ref_rejected
    logits = beta * (policy - ref)
    return -F.logsigmoid(logits).mean()

Those four numbers are log-probabilities of full completions, summed over answer tokens. If you compute them on the prompt too, you add noise. If you skip the reference model, you no longer have DPO.

Assignment

• Build 100 preference pairs. Some can be synthetic if you label them as such. Ten should be ones you ranked by hand.
• Run a short DPO step from the SFT checkpoint. Sample the same 10 prompts as week 7.
• Write whether the model got more sycophantic, more terse, or unchanged. Use examples, not a fake percentage.

Opinion checkpoint

Core project 6: a DPO step you can explain on a whiteboard.

Next: vLLM internals and production serving.