A recall test and the speed trap

A recall test

Write pairs of a real user query and the document that should come first. Embed both, rank, and count how often the right document is first (recall@1).

"""Recall test for local embeddings served by llama-server (--embedding)."""
import json
import urllib.request

import numpy as np

URL = "http://127.0.0.1:8094/v1/embeddings"
QUERY_PREFIX = "task: search result | query: "
DOC_PREFIX = "title: none | text: "

DOCS = {
    "refund": "You can ask for a refund within 30 days. Open Orders, pick the order, and tap Request refund.",
    "password": "To reset your password, tap Forgot password on the sign-in screen and follow the email link.",
    "shipping": "Standard shipping takes 3 to 5 working days. Express shipping arrives the next working day.",
    "battery": "If the battery drains fast, turn off background refresh and lower the screen brightness.",
    "export": "You can export your notes as PDF or Markdown from Settings, then Export.",
    "delete": "To delete your account, go to Settings, Privacy, and tap Delete account. This cannot be undone.",
    "offline": "The app works offline. Changes sync the next time you are online.",
    "language": "Change the app language in Settings, General, Language.",
}
QUERIES = [
    ("how do I get my money back", "refund"),
    ("I forgot my login", "password"),
    ("when will my parcel arrive", "shipping"),
    ("phone dies quickly", "battery"),
    ("save notes as a pdf file", "export"),
    ("remove my profile for good", "delete"),
    ("does it work without internet", "offline"),
    ("switch to Spanish", "language"),
]


def embed(texts):
    body = json.dumps({"input": texts}).encode()
    req = urllib.request.Request(URL, body, {"Content-Type": "application/json"})
    data = json.load(urllib.request.urlopen(req))["data"]
    return np.array([row["embedding"] for row in data], dtype=np.float32)


def truncate(vectors, dim, renormalize):
    out = vectors[:, :dim]
    if renormalize:
        out = out / np.linalg.norm(out, axis=1, keepdims=True)
    return out


def recall_at_1(q, d, ids):
    scores = q @ d.T
    hits = [ids[int(np.argmax(row))] == want for row, (_, want) in zip(scores, QUERIES)]
    return sum(hits) / len(hits)


def run():
    ids = list(DOCS)
    docs_plain = embed([DOCS[i] for i in ids])
    qs_plain = embed([q for q, _ in QUERIES])
    docs_pref = embed([DOC_PREFIX + DOCS[i] for i in ids])
    qs_pref = embed([QUERY_PREFIX + q for q, _ in QUERIES])

    print("norm of a full vector:", round(float(np.linalg.norm(docs_pref[0])), 4))
    print("recall@1 no prefixes:  ", recall_at_1(qs_plain, docs_plain, ids))
    print("recall@1 with prefixes:", recall_at_1(qs_pref, docs_pref, ids))
    for dim in (512, 256, 128):
        d_raw = truncate(docs_pref, dim, renormalize=False)
        print(f"{dim}d norm before renormalize:", round(float(np.linalg.norm(d_raw[0])), 4))
        print(f"{dim}d recall@1 renormalized:", recall_at_1(
            truncate(qs_pref, dim, True), truncate(docs_pref, dim, True), ids))


if __name__ == "__main__":
    run()

norm of a full vector: 1.0
recall@1 no prefixes:   1.0
recall@1 with prefixes: 1.0
512d norm before renormalize: 0.8626
512d recall@1 renormalized: 1.0
256d norm before renormalize: 0.6799
256d recall@1 renormalized: 1.0
128d norm before renormalize: 0.5944
128d recall@1 renormalized: 1.0

Every number is 1.0 on this toy set of 8 documents. That is a real result, and it shows a real lesson: a small, easy set cannot tell setups apart. Use 50 or more real queries from your app, including hard ones where two documents are close. Then the test shows a difference when you change the prefix, the dimension, or the model.

Run the test in CI whenever you change the model, prefixes, dimension, or runtime. For a larger eval setup, see the eval harness course.

The speed trap in llama.cpp

llama.cpp #30398: with flash attention on (-fa on) on CUDA, EmbeddingGemma 2 was about 10 times slower when the input length was not a multiple of 256 tokens. 3,840 tokens took 0.17 s and 3,841 took 1.96 s. It was fixed by PR #30416, merged on 12 October 2026. Use a build from after that date, or -fa off on older builds.

Time your own setup at lengths just below and above a multiple of 256:

"""Time embeddings at token counts just below and above a multiple of 256."""
import json
import time
import urllib.request

URL = "http://127.0.0.1:8094"


def post(path, body):
    req = urllib.request.Request(URL + path, json.dumps(body).encode(), {"Content-Type": "application/json"})
    return json.load(urllib.request.urlopen(req))


text = " ".join(["The quick brown fox jumps over the lazy dog."] * 500)
tokens = post("/tokenize", {"content": text})["tokens"]
post("/embedding", {"content": tokens[:256], "cache_prompt": False})  # warm-up
for n in [1024, 1025, 2048, 2049, 2048, 2049]:
    start = time.perf_counter()
    # cache_prompt false: otherwise slot reuse hides the difference
    post("/embedding", {"content": tokens[:n], "cache_prompt": False})
    print(f"{n} tokens: {time.perf_counter() - start:.3f}s")

On the test box (CPU only, build b11624, -fa on) the gap was much smaller: 2,048 tokens took about 2.1 s and 2,049 took about 3.1 s. 1,024 and 1,025 were close. Your GPU and build will differ, so measure.

Sources

• Model card: google/embeddinggemma-2
• llama.cpp server README