---
title: "Build a transformer from scratch (TinyGPT)"
lesson_id: "03"
---

# Build a transformer from scratch (TinyGPT)

- **Lesson ID:** 03
- **Goal:** Attention is matmuls plus a causal mask. Write TinyGPT until you can explain every tensor without hand-waving.
- **Human lesson:** [03-tinygpt-from-scratch.html](03-tinygpt-from-scratch.html)
- **Phase:** Phase 2 Model Internals

## Prerequisites

- The previous week's artifact, or a written note if this is week 1.
- Python 3.11+ and a laptop. CUDA is useful from week 5 onward and not required to read.

## Inputs, outputs, and artifacts

- **Inputs:** The replica from the prior week.
- **Outputs:** Core project 2: TinyGPT that trains on a tiny corpus.
- **Artifacts:** Lesson notes plus the code named in the human page.

## Agent build steps

1. State the week 1 principle: you do not need to pretrain an 800B model to understand the shape of the problem; you do need progressively realistic replicas.
2. Teach the topics on the human page without inventing paper URLs or news claims.
3. Keep starter code in fenced blocks that match the human lesson.
4. End with the assignment and the opinion checkpoint.
5. Use Edge FDE tone: ship, do not sightseeing. Opinion checkpoints matter.

## Constraints

Plain spoken English. No quizzes. No em dashes. No invented metrics, dates, or citations. Brand Edge FDE only. Name well-known papers by title only: Attention Is All You Need, InstructGPT, Direct Preference Optimization, ZeRO, PagedAttention/vLLM, FlashAttention, Scaling Laws for Neural Language Models.

## Key concepts

- A decoder block is norm, causal attention, residual, norm, MLP, residual.
- QKV are linear maps. The mask enforces the next-token contract.
- Attention scores scale with 1/sqrt(head dim) to keep softmax stable.
- TinyGPT is the replica. 800B is the same diagram with logistics.

## Takeaways

- Type CausalSelfAttention and train it on a tiny corpus.
- Fail the causal mask on purpose so you know the cheat.
- Name the paper by title: Attention Is All You Need. Then ship the code.

## Acceptance checks

- Human HTML has Key concepts and Takeaways.
- Opinion checkpoint is present.
- No em dash and no fake URL.
- [ ] Proceed to [lesson 04 brief](04-scaling-laws-and-synthetic-data.llms.md).