← All courses ← Course home

Lesson 04 of 05

Track hit rate

A cache you cannot measure will drift. Count hits, misses, stale rows, and wrong hits that users undo.

Agent brief (llms.md)

Hit rate is hits divided by lookups. That number is not enough. You also need wrong-hit rate: the user asked again, edited, or tapped "try again" after a reuse.

LookupEvery question
HitReused answer
MissFresh generate
StaleExpired row
UndoWrong hit
SavedGenerate time skipped
struct CacheStats: Sendable {
    var lookups = 0
    var hits = 0
    var misses = 0
    var stale = 0
    var undos = 0
    var savedNanoseconds: UInt64 = 0

    var hitRate: Double {
        lookups == 0 ? 0 : Double(hits) / Double(lookups)
    }

    var wrongHitRate: Double {
        hits == 0 ? 0 : Double(undos) / Double(hits)
    }
}

Log one line per lookup: threshold, best score, decision, and a short reason. Keep it on device. Do not upload questions.

If hit rate is high and undo rate rises, raise the threshold. If hit rate is near zero, your questions are unique or the embedder is weak. Do not "fix" that by lowering the gate until undo rate is known.

EDGE FDE metric. Ship a debug screen with hit rate, undo rate, and last ten scores. That is enough for a first review.

Next, put the cache in front of the session.

Key concepts

  • Hit rate is hits divided by lookups. Also track wrong hits via undos and retries.
  • Log threshold, best score, decision, and reason on every lookup.
  • High hit rate plus a rising undo rate means raise the threshold.
  • Ship a debug screen with hit rate, undo rate, and the last ten scores.

Takeaways

  • Hit rate alone is not enough. Watch wrong-hit rate.
  • Do not lower the gate until undo rate is known.
  • One log line per lookup is enough for a first review.