← All courses ← Course home

Lesson 03 of 05

Rank memory, search, and tools

Give every slice a score, a token cost, and a lane. Fill leftover budget from the top. Drop the rest.

Agent brief (llms.md)

Three lanes compete:

ScoreRelevance now
CostToken count
FillUntil leftover is gone
enum SliceLane: String, Sendable { case memory, search, tool }

struct ContextSlice: Sendable {
    let id: String
    let lane: SliceLane
    let text: String
    let score: Float
    let tokens: Int
}

func pack(slices: [ContextSlice], leftover: Int, perLaneCap: Int = 4) -> [ContextSlice] {
    var used = 0
    var taken: [SliceLane: Int] = [:]
    var packed: [ContextSlice] = []
    for slice in slices.sorted(by: { $0.score == $1.score ? $0.tokens < $1.tokens : $0.score > $1.score }) {
        let count = taken[slice.lane, default: 0]
        if count >= perLaneCap { continue }
        if used + slice.tokens > leftover { continue }
        packed.append(slice)
        used += slice.tokens
        taken[slice.lane] = count + 1
    }
    return packed
}

Score memory and search with the same embedder you use in the semantic cache course, or with the retrieval ranker. Tools get a hard allow-list from the current app state, not a similarity score. A calendar tool does not belong in a packing pass about a photo.

Cap each lane so one noisy index cannot eat the window. Four memories and four hits is a sane start.

Drop, do not summarize yet. First ship drops low slices. Summaries come later, and they need their own eval.

Next, put the packed slices into a Dynamic Profile.

Key concepts

  • Three competing lanes: memory, search hits, and tools.
  • Each slice gets a score, a token cost, and a lane.
  • Fill leftover budget from the top. Drop the rest.
  • Cap each lane. Four memories and four hits is a sane start.

Takeaways

  • Score memory and search with your ranker. Tools use a hard allow-list.
  • Tool schemas cost tokens too.
  • Drop low slices first. Do not summarize until you have eval for summaries.