Proof-based linear algebra for deep learning

Core principle, keep it for the whole twelve weeks: you do not need to pretrain an 800B model to understand the shape of the problem. You do need to build progressively realistic replicas of the whole stack. Week 1 is the replica of the math the rest of the stack pretends you already know.

This is not a sightseeing tour of Wikipedia linear algebra. Every object you write down should show up again in attention, in backprop, or in a training log. If you cannot prove a one-line identity, you will hand-wave a transformer later.

Weekly rhythm

Use this rhythm every week. Monday: read and prove the claim. Tuesday and Wednesday: code the replica. Thursday: finish the assignment. Friday: write the opinion checkpoint in your own words. Weekend stretch is optional. Ship the artifact, do not collect tabs.

Topics

Vector spaces, bases, and linear maps. A layer is a map. A residual is a sum of maps.
Inner products, norms, and orthogonality. Cosine similarity is an inner product after you normalise.
Projections. Residual connections and least squares are the same geometric move.
Eigenvalues and SVD. Low-rank adapters and compressed KV ideas start here.
Gradients, Jacobians, Hessians. Backprop is a Jacobian-vector product, not magic.

Write the identities you will reuse. The squared Euclidean norm of x is x^Tx. A projection onto the column space of A with full column rank is A(A^TA)^-1A^T. Softmax is invariant to adding a constant to logits, which is why you can subtract the row max before the exp.

Coding

Stay in NumPy first. Do not hide behind autograd until you can compute a gradient by hand for a two-layer map.


import numpy as np

def project(A, y):
    """Least-squares projection of y onto col(A)."""
    gram = A.T @ A
    return A @ np.linalg.solve(gram, A.T @ y)

def svd_lowrank(X, rank):
    U, s, Vt = np.linalg.svd(X, full_matrices=False)
    return (U[:, :rank] * s[:rank]) @ Vt[:rank]

def softmax(logits):
    z = logits - logits.max(axis=-1, keepdims=True)
    exp = np.exp(z)
    return exp / exp.sum(axis=-1, keepdims=True)

Then write a tiny autograd for y = Wx and L = ||y - t||^2. The gradient wrt W is an outer product. If that sentence is fuzzy, stop and derive it on paper.

Assignment

Prove that the residual y - P y is orthogonal to the column space of A when P is the projection above.
Measure the condition number of A^TA on a skinny random matrix and on a near-collinear one. Write one paragraph on what that does to a learning rate.
Implement SVD truncation and plot reconstruction error versus rank. No invented accuracy claims. Report the numbers you measured.

Opinion checkpoint

Write this down. If someone says they understand transformers but cannot say what a Jacobian-vector product is, they are sightseeing. The 800B run is a logistics problem sitting on linear maps you can replica this week on a laptop.

Core project 1 starts here: a linear algebra and tiny autograd lab.

Next week you will put a non-convex optimiser on top of these maps.

Next: Non-convex optimisation.

Key concepts

A network layer is a linear map plus a cheap nonlinearity.
Inner products and norms turn geometry into the numbers you log.
SVD is the honest way to talk about rank and compression.
Backprop is Jacobian-vector products, not a framework feature.
You do not need an 800B pretrain to learn the shape of the stack.

Takeaways

Prove one projection identity and implement it. That is the week.
Condition numbers explain more training instability than lore does.
Keep a written opinion: replicas beat tourism.