← All courses ← Course home

Lesson 03 of 04

Truncate safely, keep one model

You can keep only the first 512, 256, or 128 numbers of each vector to save space. After cutting, the vector is no longer length 1, so normalize it again.

Agent brief (llms.md)

Environment setup

Use the same local-lab folder from lesson 01 (uv, Python 3.11+, and uv add numpy). If you skipped it, do the setup in lesson 01 first.

Install llama.cpp. On macOS: brew install llama.cpp. On Linux or Windows, download a release from github.com/ggml-org/llama.cpp/releases. Check it with llama-server --version.

Shorter vectors

EmbeddingGemma 2 is trained with Matryoshka Representation Learning. The EmbeddingGemma 2 model card says you can keep the first 512, 256, or 128 dimensions. Quality is close to full down to 256. 128 is best for text-only work and you should test it on your data.

768dUnit vector
SliceFirst 256
Lengthabout 0.68
NormalizeLength 1.0

On the test box, the length of one document vector after slicing was:

512d norm before renormalize: 0.8626
256d norm before renormalize: 0.6799
128d norm before renormalize: 0.5944

If you skip normalizing and score with a dot product, longer vectors win more often, and ranking gets worse with no error. The model card calls this out too.

import numpy as np

def truncate(vectors: np.ndarray, dim: int) -> np.ndarray:
    out = vectors[:, :dim]
    return out / np.linalg.norm(out, axis=1, keepdims=True)

One model per index

Vectors from two models, two prefixes, or two dimensions are not comparable. A 768-dimensional query cannot be scored against a 128-dimensional index, and vectors made without prefixes do not match vectors made with them. Mixing them gives bad results, not errors.

index_guard.py from lesson 02 stores the setup next to the vectors and refuses to load an index built with a different one:

from index_guard import SETUP, load_index, save_index

save_index("notes_index", doc_vectors_256, ids)   # writes vectors.npy and index.json
vectors, ids = load_index("notes_index")          # raises if SETUP changed
caught: index was built with {'model': 'ggml-org/embeddinggemma-2-GGUF:Q8_0', 'query_pre...}, app uses {...}. Re-embed everything.

When you change the model, a prefix, or the dimension, build a new index from scratch. Do not update it a few rows at a time.

Sources

Key concepts

  • Matryoshka vectors can be cut to 512, 256, or 128 dimensions.
  • A cut vector is shorter than length 1 and must be normalized again.
  • Query and document vectors must use the same model, prefixes, and dimension.
  • Mixing setups gives bad ranking, not errors.

Takeaways

  • Slice, then normalize, every time.
  • Store model, prefixes, dimension, and normalization next to the vectors.
  • Rebuild the whole index when any of them changes.