← All courses ← Course home

Lesson 01 of 04

Use the right task prefixes

EmbeddingGemma 2 expects a short instruction in front of each text. Queries and documents get different ones. The docs disagree, so use the model card.

Agent brief (llms.md)

Environment setup

Do this once before the first runnable example. You need Python 3.11 or newer and uv.

# 1. Install uv (skip if you have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Make a project folder with Python 3.11+
mkdir local-lab && cd local-lab
uv init --bare --python 3.11
uv venv

# 3. Add the course packages
uv add numpy

# 4. Optional: PyTorch, only if you also want to run models directly in Python.
#    On Apple Silicon this wheel uses MPS. On other machines it uses the CPU.
uv add torch
uv run python -c "import torch; print('mps' if torch.backends.mps.is_available() else 'cpu')"

Install llama.cpp. On macOS: brew install llama.cpp. On Linux or Windows, download a release from github.com/ggml-org/llama.cpp/releases. Check it with llama-server --version.

On iOS or Android you would run the same model with LiteRT-LM. The rules in this course are the same.

Run every script with uv run python script.py. The examples were checked on llama.cpp build b11624 and Ollama 0.40.4 on a Linux CPU box. Your numbers will be different on your machine. That is the point: measure on your own hardware.

Start the embedding server

llama-server -hf ggml-org/embeddinggemma-2-GGUF:Q8_0 --embedding --port 8094 -c 8192 -b 8192 -ub 8192 -np 1

--embedding turns on the /v1/embeddings endpoint. The batch sizes must fit your longest input, because an embedding is computed in one pass.

curl -s localhost:8094/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"input": ["task: search result | query: how do I get my money back"]}' \
  | python3 -c "import json,sys,math; v=json.load(sys.stdin)['data'][0]['embedding']; print(len(v), round(math.sqrt(sum(x*x for x in v)), 4))"
# 768 1.0

On the test box (llama.cpp b11624) the server returned 768 numbers with length 1.0. So the vectors come back normalized.

The prefixes

The EmbeddingGemma 2 model card says the model is trained with task prefixes. Leaving them out still works but gives worse results. For search:

query:    task: search result | query: {query}
document: title: {title} | text: {content}     (use "title: none" when there is no title)

Other tasks on the card: question answering (task: question answering | query: ), code search (task: code retrieval | query: ), and for symmetric tasks the same prefix on both sides, such as task: sentence similarity | query: or task: classification | query: . Prefixes are for text only. Images, video, and audio take no prefix.

The docs disagree

LiteRT-LM #3929 shows that the LiteRT-LM guide lists different prefixes. For example, the guide uses task: search query | text: for queries and task: search result | text: for documents. The model card and config_sentence_transformers.json use the strings above, and LiteRT-LM's own demo app uses the model card pair. The issue is still open.

Rule: use the model card prefixes. Store them in one place in your code, next to the model name.
QUERY_PREFIX = "task: search result | query: "
DOC_PREFIX = "title: none | text: "

def query_text(q: str) -> str:
    return QUERY_PREFIX + q

def doc_text(body: str, title: str | None = None) -> str:
    return f"title: {title or 'none'} | text: {body}"

If you change a prefix later, every stored document vector is out of date. Lesson 03 shows how to catch that.

Sources

Key concepts

  • EmbeddingGemma 2 gives 768-dimensional vectors.
  • Search uses different prefixes for queries and documents.
  • The LiteRT-LM guide and the model card disagree (LiteRT-LM #3929).
  • The model card and its config file are the source the model was trained with.

Takeaways

  • Use task: search result | query: and title: none | text: for search.
  • Keep prefixes in one constant next to the model name.
  • Re-embed everything if a prefix changes.