A labelled set and option order bias

The test set

One JSON object per line: the text and the right label. Include easy cases and hard ones that sit between two labels.

{"id": "t01", "text": "I was charged twice for my order last week.", "label": "billing"}
{"id": "t02", "text": "My card was declined but the money left my account.", "label": "billing"}
{"id": "t03", "text": "Please send me a VAT invoice for March.", "label": "billing"}
{"id": "t04", "text": "Can I get a refund for the yearly plan?", "label": "billing"}
{"id": "t05", "text": "The parcel says delivered but it is not here.", "label": "shipping"}
{"id": "t06", "text": "Can you ship to Norway?", "label": "shipping"}
{"id": "t07", "text": "My order has been in transit for two weeks.", "label": "shipping"}
{"id": "t08", "text": "The box arrived damaged and the cup is broken.", "label": "shipping"}
{"id": "t09", "text": "The app crashes when I open the camera.", "label": "technical"}
{"id": "t10", "text": "I cannot log in, it says error 500.", "label": "technical"}
{"id": "t11", "text": "Sync stopped working after the last update.", "label": "technical"}
{"id": "t12", "text": "The export to PDF button does nothing.", "label": "technical"}
{"id": "t13", "text": "I paid for express delivery but it came late. I want the extra fee back.", "label": "billing"}
{"id": "t14", "text": "The tracking link in your email gives a 404 page.", "label": "shipping"}
{"id": "t15", "text": "After I updated my card, the app shows a blank screen.", "label": "technical"}
{"id": "t16", "text": "Where is my order?", "label": "shipping"}

Tickets t13 to t16 are the hard ones. For example, t13 is about a late delivery, but the customer wants money back.

The eval script

For each ticket the script asks once in the normal order and then again with the options shuffled. If the pick changes, that is an order flip. It also counts how many answers pass a confidence threshold and times every call.

"""Test a local decision model on a labelled set before it goes into a pipeline."""
import argparse
import json
import random
import statistics
import time
import urllib.request

OPTIONS = {
    "billing": "Payments, charges, invoices, and refunds.",
    "shipping": "Delivery, tracking, and damaged parcels.",
    "technical": "App errors, crashes, login, and sync problems.",
}
INSTRUCTIONS = "Which team should handle this support ticket?"


def ask(url, text, order):
    criteria = {name: OPTIONS[name] for name in order}
    body = {
        "state": text,
        "questions": {"team": {"type": "choice", "instructions": INSTRUCTIONS, "criteria": criteria}},
    }
    req = urllib.request.Request(
        url + "/v1/systemone", json.dumps(body).encode(), {"Content-Type": "application/json"}
    )
    start = time.perf_counter()
    answer = json.load(urllib.request.urlopen(req))["answers"]["team"]
    return answer, (time.perf_counter() - start) * 1000


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--url", default="http://127.0.0.1:8080")
    p.add_argument("--data", default="tickets.jsonl")
    p.add_argument("--orders", type=int, default=3, help="option orders to try per ticket")
    p.add_argument("--threshold", type=float, default=0.5)
    p.add_argument("--seed", type=int, default=7)
    args = p.parse_args()
    rng = random.Random(args.seed)
    rows = [json.loads(line) for line in open(args.data, encoding="utf-8") if line.strip()]

    correct = flips = auto = auto_correct = 0
    latencies = []
    for row in rows:
        picks = []
        first = None
        for i in range(args.orders):
            order = list(OPTIONS)
            if i > 0:
                rng.shuffle(order)
            answer, ms = ask(args.url, row["text"], order)
            latencies.append(ms)
            picks.append(answer["choice"])
            if first is None:
                first = answer
        if len(set(picks)) > 1:
            flips += 1
        correct += first["choice"] == row["label"]
        if first["confidence"] >= args.threshold:
            auto += 1
            auto_correct += first["choice"] == row["label"]
        print(f'{row["id"]} want={row["label"]:9} got={first["choice"]:9} '
              f'conf={first["confidence"]:.2f} picks={picks}')

    n = len(rows)
    lat = sorted(latencies)
    print("\naccuracy:", f"{correct}/{n}")
    print("order flips:", f"{flips}/{n}", "(answer changed when options were shuffled)")
    print(f"auto-handled at confidence >= {args.threshold}:", f"{auto}/{n}",
          "| accuracy on those:", f"{auto_correct}/{auto}" if auto else "n/a")
    print("sent to a human:", n - auto)
    print(f"latency ms p50={statistics.median(lat):.0f} p95={lat[int(0.95 * (len(lat) - 1))]:.0f}")


if __name__ == "__main__":
    main()

uv run python eval_decision.py --url http://127.0.0.1:8080

Kev-0.8B Q8_0 on the test box:

t12 want=technical got=technical conf=0.84 picks=['technical', 'technical', 'technical']
t13 want=billing   got=shipping  conf=0.46 picks=['shipping', 'shipping', 'shipping']
t14 want=shipping  got=shipping  conf=0.35 picks=['shipping', 'technical', 'technical']
t15 want=technical got=technical conf=0.97 picks=['technical', 'technical', 'technical']

accuracy: 15/16
order flips: 1/16 (answer changed when options were shuffled)
auto-handled at confidence >= 0.5: 14/16 | accuracy on those: 14/14
sent to a human: 2
latency ms p50=207 p95=283

Two things to see. The only wrong answer (t13) had low confidence. The only order flip (t14) also had low confidence: with the options shuffled the model picked technical. Low confidence and order flips point at the same hard cases.

What to do with flips

• Send any ticket that flips to a human, or ask in two orders and only act when both agree.
• Rewrite option descriptions that overlap. Short, distinct descriptions help.
• Grow the set from real mistakes. The agent harness course shows a loop for that.

Sources

• llama.cpp server README