Evolve your agent's prompt with GEPA

Source: Grok conversation: GEPA explained and how to implement it (14 Oct 2026), discussing Artem's post on a reported inference-cost cut, the GEPA repo, and the GEPA paper. Pairs with lessons 04 and 05.

Goal

GEPA (Genetic-Pareto prompt optimisation) evolves a system prompt by reading full traces and written feedback, then keeping prompts that win on different slices of your data. Use it so a small model you can afford to run at the edge scores close to a larger model on one specialized task. Leave the weights alone until the prompt has had its turn.

The worked task is an edge alert reviewer. Each row is one device alert. The label is ### page (a person should look now) or ### hold (it can wait). Swap in your own agent task if you keep the same three files, the same metric shape, and a held-out test set.

The task model is the cheap model you will ship at the edge. The reflection model is stronger. It proposes prompt edits. You do not ship the reflection model. You ship the small model and the evolved prompt.

Prerequisites

• Python 3.11 or newer.
• uv, the same tool lesson 03 uses.
• An API key for the models you call, set in the provider's env var (for example OPENAI_API_KEY for openai/... model ids). Or a local model behind an OpenAI-compatible chat API. A local server still needs some value in OPENAI_API_KEY, such as sk-local. With no key set, the run fails with a confusing AttributeError ... 'choices' instead of a clear auth error.
• A small labeled set you can split into train, validation, and test. A few dozen careful rows are enough to start. Lessons 04 and 05 are the harness version of the same idea: a golden row, a written bar, then a prompt change.

Steps

1. Install GEPA.

   uv venv --python 3.11
   source .venv/bin/activate
   uv pip install "gepa[full]"

   Use gepa[full], not plain gepa. Plain gepa installs no dependencies, so step 5 fails with ImportError: tqdm must be installed and then No module named 'litellm'. --python 3.11 is the lowest version that works. A newer Python is fine too. Check the install with python -c "import gepa, litellm, tqdm".

   The library is gepa on PyPI. The project lives at github.com/gepa-ai/gepa. Docs: the GEPA quickstart.

2. Build train, validation, and test JSONL. One JSON object per line. input is the alert. answer is the label the metric looks for. Keep test.jsonl out of the optimizer.

   {"input": "pump-4 temp 92C for 6 minutes", "answer": "### page"}
   {"input": "pump-4 temp 41C, inside the normal band", "answer": "### hold"}
   {"input": "gateway lost heartbeat for 3 cycles", "answer": "### page"}
   {"input": "daily humidity report, no threshold crossed", "answer": "### hold"}

   import json
   
   def load_jsonl(path):
       rows = []
       with open(path, encoding="utf-8") as handle:
           for line in handle:
               line = line.strip()
               if not line:
                   continue
               row = json.loads(line)
               row.setdefault("additional_context", {})
               rows.append(row)
       return rows
   
   trainset = load_jsonl("train.jsonl")
   valset = load_jsonl("val.jsonl")
   testset = load_jsonl("test.jsonl")  # never pass this to gepa.optimize
   
   def count_labels(rows):
       counts = {}
       for row in rows:
           counts[row["answer"]] = counts.get(row["answer"], 0) + 1
       return counts
   
   print("train", count_labels(trainset))
   print("val", count_labels(valset))
   print("test", count_labels(testset))

   Hold out the test file first. Split the rest between train and validation. When the remaining set is small, an even split is a sound start. GEPA uses train rows for feedback and validation rows to choose which prompt survives.

3. Write a seed prompt. Ask for one final line so the metric can parse it. Save it as prompts/edge_alert.seed.txt.

   You review one edge-device alert. Decide whether a person must act now.
   Reply with exactly one final line: ### page or ### hold.
   ### page means a person should look now. ### hold means the alert can wait.
   Use only the alert text. Do not invent sensor history.

4. Write a metric that returns a score and textual feedback. GEPA reads the feedback the way a person would read a grader's note. A bare 0 or 1 throws away the reason. This callable matches the default adapter's evaluator: it receives the example and the model response, and it returns EvaluationResult.

   from gepa.adapters.default_adapter.default_adapter import EvaluationResult
   
   def edge_alert_metric(data, response):
       expected = data["answer"]
       other = "### hold" if expected == "### page" else "### page"
       has_expected = expected in response
       has_other = other in response
       snippet = response[:400]
   
       if has_expected and not has_other:
           feedback = f"Correct. The response includes {expected} and not {other}."
           return EvaluationResult(score=1.0, feedback=feedback)
   
       if has_expected and has_other:
           feedback = (
               f"Both labels appear. Expected only {expected}. "
               "Ask for one final line and no second label. "
               f"Response: {snippet}"
           )
       elif has_other:
           feedback = (
               f"Wrong label. Expected {expected} but the response includes {other}. "
               f"Response: {snippet}"
           )
       else:
           feedback = (
               f"Missing label. Expected a line with {expected}. "
               f"Response: {snippet}"
           )
       return EvaluationResult(score=0.0, feedback=feedback)

   The default grader, when you omit evaluator, is a substring check with shorter feedback. The custom metric is what teaches the reflection model why a row failed. For a multi-step agent, DSPy exposes the same idea as dspy.GEPA: a metric that returns dspy.Prediction(score=..., feedback=...), then compile(...). See the quickstart. Install DSPy only if you take that path.

5. Run gepa.optimize with a budget of 100 to 200. max_metric_calls is the evaluation budget. The sample uses 150, inside that range. task_lm is the edge model. reflection_lm is the larger model. Run this in the same file as the loader and the metric.

   import gepa
   
   seed = open("prompts/edge_alert.seed.txt", encoding="utf-8").read()
   # LiteLLM model ids. Replace both with the models you actually call.
   task_lm = "openai/gpt-4o-mini"
   reflection_lm = "openai/gpt-4o"
   
   result = gepa.optimize(
       seed_candidate={"system_prompt": seed},
       trainset=trainset,
       valset=valset,
       task_lm=task_lm,
       reflection_lm=reflection_lm,
       evaluator=edge_alert_metric,
       max_metric_calls=150,
       run_dir="gepa_runs/edge_alert",
       display_progress_bar=True,
   )
   
   best = result.best_candidate["system_prompt"]
   open("prompts/edge_alert.gepa.txt", "w", encoding="utf-8").write(best)
   print(best)
   print("val score", result.val_aggregate_scores[result.best_idx])
   print("metric calls", result.total_metric_calls)

   Do not put testset in trainset or valset. If validation is omitted, GEPA reuses the train set and your selection score is no longer honest.

   Local model. To use a local OpenAI-compatible server, set its base URL and name the model with the openai/ prefix:

   export OPENAI_API_BASE=http://localhost:8080/v1
   export OPENAI_API_KEY=sk-local   # any value works for most local servers

   Then set task_lm = "openai/<your-model-name>". The same works for reflection_lm.

   Run directory. If run_dir already holds an earlier run, GEPA quietly resumes it instead of starting fresh. Delete the folder or pick a new name for each fresh run. After a run it holds files such as candidates.json, run_log.txt, run_log.json, gepa_state.bin (the resume state), candidate_tree.html, and a generated_best_outputs_valset/ folder.

6. Evaluate on the held-out test set. Score the seed prompt and the GEPA prompt with the same edge model. Score the larger model once, with the seed prompt, so you have a bar to match. This block uses edge_alert_metric, testset, task_lm, and reflection_lm from the earlier steps. It does not call gepa.optimize.

   import time
   from gepa.lm import LM
   
   def score_prompt(prompt, rows, model_name):
       lm = LM(model_name)
       messages = [
           [
               {"role": "system", "content": prompt},
               {"role": "user", "content": row["input"]},
           ]
           for row in rows
       ]
       started = time.perf_counter()
       responses = lm.batch_complete(messages)
       elapsed = time.perf_counter() - started
       tokens_in = lm.total_tokens_in
       tokens_out = lm.total_tokens_out
       if not tokens_in:
           # Fallback when the endpoint reports no usage: about 4 characters per token.
           tokens_in = sum(len(prompt) + len(row["input"]) for row in rows) // 4
   
       correct = 0.0
       page_total = 0
       page_hit = 0.0
       characters = 0
       for row, response in zip(rows, responses):
           result = edge_alert_metric(row, response)
           correct += result.score
           characters += len(prompt) + len(row["input"])
           if row["answer"] == "### page":
               page_total += 1
               page_hit += result.score
   
       n = len(rows) or 1
       return {
           "accuracy": correct / n,
           "page_recall": (page_hit / page_total) if page_total else 0.0,
           "mean_characters": characters / n,
           "tokens_in": tokens_in,
           "tokens_out": tokens_out,
           "cost": lm.total_cost,
           "seconds": elapsed,
       }
   
   seed_prompt = open("prompts/edge_alert.seed.txt", encoding="utf-8").read()
   gepa_prompt = open("prompts/edge_alert.gepa.txt", encoding="utf-8").read()
   
   print("edge + seed", score_prompt(seed_prompt, testset, task_lm))
   print("edge + gepa", score_prompt(gepa_prompt, testset, task_lm))
   print("larger + seed", score_prompt(seed_prompt, testset, reflection_lm))

   LM.batch_complete returns strings. The LM object also keeps running totals: lm.total_tokens_in, lm.total_tokens_out, and lm.total_cost (in dollars, when LiteLLM knows the price). The sample makes a new LM per call, so the totals cover just that run. If the endpoint reports no usage, the totals stay at 0 and the code falls back to characters divided by 4.

7. Compare cost, tokens, and accuracy. Fill this table from the held-out run. Cost is the published price times measured tokens when the endpoint reports usage. A local edge model has no API bill. Record seconds per alert from the timer instead, and say that the column is latency.

   System	Accuracy	Page recall	Tokens in / out	Cost or latency
   Edge model, seed prompt	record	record	record	record
   Edge model, GEPA prompt	record	record	record	record
   Larger model, seed prompt	record	record	record	record

   Adopt the new prompt when page recall holds and accuracy on the test set improves. A public report of a 17x inference-cost cut was one workload. It is a story about the method, not your target number.

Expected result

• A prompts/edge_alert.gepa.txt chosen on the validation set. It can be longer than the seed. If the seed already scores perfectly, GEPA may keep it unchanged.
• A filled table for the held-out test set: seed versus GEPA versus the larger model, with accuracy, page recall, tokens, and cost or latency.
• A gepa_runs/edge_alert directory you can read. Validation score and result.total_metric_calls printed beside the prompt.

Gains depend on your labels. If the edge model was already matching the larger one, the table should say so.

Self-check

• Held-out data. Was test.jsonl absent from gepa.optimize? If the test rows were in train or validation, the table is optimistic.
• Written feedback. Does edge_alert_metric return a sentence that says what was wrong, not only 0 or 1? GEPA mutates from that text.
• Class balance. What share of the test rows are ### page? If almost every row is ### hold, an always-hold prompt looks accurate. Page recall is the check.
• One change GEPA made. From the run directory, quote one sentence the new prompt added or removed. Say which failure the feedback described. candidates.json lists every prompt GEPA tried, run_log.txt has the feedback, and candidate_tree.html shows which prompt came from which.

Solution notes

GEPA and fine-tuning work together. The search keeps a Pareto front: several prompts that are strong on different slices, such as alert type or severity. result.per_val_instance_best_candidates is that map. Pick by page recall first, then accuracy, then prompt size.

If the GEPA prompt closes most of the gap to the larger model, ship the small model and the new prompt to the edge. If a slice still fails, keep the evolved prompt as the system prompt and fine-tune the small model on that slice. The prompt search makes that later training job smaller.

A multi-step agent with several text fields can use dspy.GEPA instead of the single-prompt adapter. The metric still returns a score plus feedback. The call is still a budget of metric calls, not a weight update.