EDGE FDE work on the phone is paid in heat, latency, and battery. A LanguageModelSession that answers "What is my next meeting?" and then "When is my next meeting?" should not always run the model twice.
Two caches, two jobs
Apple's session can report usage.input.cachedTokenCount. That is a token KV cache inside the model run. It speeds a generate that still happens. A semantic cache is different. It is a table you own. You store a question vector and the answer text. You skip the generate when a new question is near an old one.
Do not mix the two in logs. KV cache hits mean the model reused tokens. Semantic hits mean the app never called respond.
When reuse is wrong
- The answer depends on a clock, a calendar, or a file that changed.
- The user asked for a rewrite, not the same fact.
- The new question is only loosely related (near in words, different intent).
Those cases are misses. Lesson 03 sets a threshold. Lesson 04 tracks how often you were right.
WWDC26 Foundation Models talks about usage and context. It does not ship a semantic cache for you. You build that layer. Next, embed the question on the device.
Key concepts
- Repeat questions in different words should not always trigger a full generate.
- A semantic cache is your table of question vectors plus answer text.
usage.input.cachedTokenCountis the model's KV cache inside a generate, not a semantic skip.- Reuse fails when answers depend on time, files, rewrites, or a different intent.
Takeaways
- Do not mix KV cache hits and semantic hits in logs.
- Treat reuse as a product choice, not a free speed win.
- Know when a close question is still a miss.