Hybrid Recall for Agent Memory

An agent that remembers things about its owner has a retrieval problem that most search tutorials do not cover. The memories are short. “The owner prefers metric units.” “The staging site lives at this address.” Each one is a sentence, and the questions asked of them are shorter still: “Who is Priya?”

Embedding models are weak at this. Short sentences that share a template (“X’s brother is Y”, “X’s sister is Z”) land close together in vector space, so the right fact can sit at rank three or five behind its look-alikes. Keyword search has the opposite profile: it finds a name exactly and has no idea that “wife” and “spouse” mean the same thing.

This post describes the memory design I moved to: both searches over the same facts, merged by rank.

Requirements

• Short facts as the unit. Not documents, not chunks.
• Find a fact by a name, and also by a paraphrase.
• No duplicates. An agent will state the same fact many times.
• One place for the data, with writes that cannot race.
• Runs on Cloudflare with no external database.

Architecture

A single Durable Object holds the whole memory. At this scale (tens of facts, about a hundred long-form pages) that is not a scaling decision. It buys strict serialisation, so a background job and an interactive write can never interleave.

Storage uses the Durable Object’s SQLite database:

• A facts table holds the text, a type (fact, skill or log), a source and a timestamp. It is the source of truth.
• A facts_fts table is an FTS5 index over the same text, with Porter stemming, ranked by BM25.
• Vectorize holds one embedding per fact, from Workers AI bge-large-en-v1.5 (1024 dimensions, cosine). Vector ids are prefixed fact: so facts and long-form page chunks can share an index.

Durable Object SQLite supports FTS5, but not triggers and not explicit BEGIN statements. The index is therefore kept in sync in code, and the two inserts run inside transactionSync.

The agent reaches the memory over a service binding with a shared-secret header, not over a public URL.

Recall by rank fusion

Recall runs both searches and merges them with reciprocal rank fusion:

1. Remove stop words from the query and join the remaining terms with OR into an FTS5 MATCH. Take the top 30 by BM25.
2. Embed the query and take the top 30 Vectorize matches of the same type.
3. Score every fact as the sum of 1 / (60 + rank) across the lists it appears in. Sort and return the top N.

Rank fusion needs no score calibration. BM25 scores and cosine similarities are on unrelated scales, and fusing by rank sidesteps the problem. A fact that either search ranks highly will surface, and a fact that both rank well rises to the top. Each result carries a via tag (kw, vec or both), which makes a bad recall easy to debug.

Writes that deduplicate

An agent writes the same fact repeatedly, in slightly different words. Two checks run before a fact is stored:

• Exact: the same type and the same text, ignoring case.
• Near: the nearest neighbour in Vectorize has the same type and a cosine score of 0.97 or above.

Either one returns the existing id instead of inserting. Deletion removes the row, the FTS entry and the vector together.

Results

I measured it on the memory I actually had: 48 facts and 24 questions written the way the agent asks them (“What units does the owner prefer?”, “When is the renewal due?”). I compared the previous vector-only recall with hybrid recall.

	Correct fact in top 5	Correct fact at rank 1
Vector only	22 of 24	20 of 24
Hybrid	24 of 24	15 of 24

Hybrid recall found every fact. Two hits were rescues: vector-only missed the top 5 entirely on a bare first name and on a question about a person’s role. Hybrid found both (at rank 1 and rank 3). It also moved two bare-name queries from rank 3 up to rank 1.

It also ranked the right fact first less often on questions that share a template: 8 of the 24 landed at rank 2, nearly all of them relationship questions, and one landed at rank 3. I have not isolated the cause. My working explanation is that fusion lifts look-alike facts that share query terms, but I did not test it. The agent does not use a single top result. It injects the top 15 into the model’s context, so being found matters more than being first, and that is why this trade is acceptable here. A system that shows only the first result should not take it without tuning.

The sample is small. Treat the numbers as evidence that the approach works on short personal facts, not as a benchmark.

Design principles

• Match the retrieval to the data. Short, templated sentences are where embeddings are weakest, and where keyword search is strongest.
• Fuse ranks, not scores. It removes the need to compare unrelated scales.
• Deduplicate on write. A memory that grows with every repeated statement gets worse to search.
• Keep the old path as a fallback. In the agent, hybrid hits come first, the older vector index fills in behind, and the results are deduplicated by text. If the memory Worker is unreachable, recall behaves as before.

What this does not claim

• It is not a benchmark. Forty-eight facts and twenty-four questions show direction, not scale.
• It has not been tested at tens of thousands of facts. One Durable Object holding everything is a deliberate choice for a personal-scale memory and would need sharding beyond it.
• The fusion constant (60) and the 0.97 duplicate threshold are the common defaults and my first choices. I did not tune them.
• Rank 1 accuracy got worse, as the table shows.

References

• Durable Objects SQLite storage
• Vectorize
• Workers AI embedding models
• SQLite FTS5
• Cormack, Clarke and Buettcher, Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods