Scaling laws, data, and synthetic data

Scaling-law talk is cheap when nobody logged tokens and compute. Scaling Laws for Neural Language Models is the well-known title for the idea that loss tracks scale in a regular way across model size, data, and compute. You will not reproduce the original plots. You will learn to stop treating data as an afterthought.

Synthetic data is a tool. It is not a substitute for a quality bar. If you generate instructions with a larger model, you inherit that model's failures. Type the rows. Filter them. Keep a human slice.

Coding: typed synthetic instructions

from pydantic import BaseModel, Field, ValidationError

class SyntheticInstruction(BaseModel):
    instruction: str = Field(min_length=8)
    input: str = ""
    output: str = Field(min_length=1)
    source: str = "synthetic"
    quality: float = Field(ge=0.0, le=1.0)

def accept_row(raw: dict, min_quality: float = 0.6) -> SyntheticInstruction | None:
    try:
        row = SyntheticInstruction.model_validate(raw)
    except ValidationError:
        return None
    if row.quality < min_quality:
        return None
    if row.output.strip() == row.instruction.strip():
        return None
    return row

Write a generator that emits JSONL. Reject empty outputs, copied instructions, and quality below your threshold. Store source so you can drop the synthetic slice later. That is intelligent use: typed, filtered, attributed.

Topics

• Tokens versus examples. Packing multiple short instructions into one sequence changes the loss you think you are measuring.
• Dedup and contamination. If the eval leaked into train, your curve is a vanity plot.
• Compute-optimal allocation as an intuition, not a number you invent. You will pick a width and a token budget you can finish.
• Mixture weights. Code, prose, and instructions fight for the same gradient.

Assignment

• Build a 2k-row JSONL of synthetic instructions with SyntheticInstruction. Report how many the filter dropped.
• Hand-write 20 rows. Compare a model-written sample to a human sample without claiming a win-rate you did not measure.
• Write the token budget for week 5: context length, batch tokens, steps, and total tokens. Use arithmetic, not vibes.

Opinion checkpoint

Next: Pretrain a small language model.