EDGE FDE
BeginnerReuse answers that are almost the same
Lower latency and battery use by returning a past answer when the new question is close enough.
Embedding similarity cache + hit-rate tracking for on-device sessions.
Build a small on-device cache that embeds the question, checks nearby past answers, and records hit rate. iOS 27 and macOS 27 only.
Lessons
Key concepts
- An app-owned semantic cache sits in front of
LanguageModelSessionon iOS 27 and macOS 27. - Embed the question, compare with cosine similarity, and return a stored answer on a hit.
- This is not Apple's token KV cache (
usage.input.cachedTokenCount). - Track hit rate, wrong-hit rate, and stale evictions before you lower the threshold.
Takeaways
- You can skip
session.respondwhen a new question is close enough to a past one. - You can lower latency and battery use while measuring whether reuse is safe.
- You can wire a cache-first path with stats and a debug screen.
Agent briefs
Related courses
Route work to the right Apple model · Stream tokens and measure speed · Version prompts like product copy