How persistent AI memory actually works

A model's weights are frozen. Talking to it does not teach it anything, and closing the tab does not make it forget — there was never anywhere for the memory to go. What you experience as an assistant remembering you is a much more mundane arrangement: text was saved somewhere, and before the model answered, a program went and found the relevant pieces and pasted them in.

That is the whole trick. But the quality of an AI memory is almost entirely the quality of the fetching step, and that is where products in this category differ wildly while describing themselves identically.

Context window is not memory

The two get confused constantly, so: the context window is the model's working desk. Everything it can consider right now, this turn. It is large in current models, and it is completely erased between conversations.

Memory is a store that outlives the conversation. The relationship between them is that memory *feeds* the window — selectively.

And it has to be selective, because the naive alternative fails on both cost and accuracy. If you had a year of notes and dumped all of them in every turn, you would pay for those tokens on every single message, and the model would do worse, not better: relevant detail buried in a hundred pages of irrelevant detail gets diluted. Retrieval exists because less, chosen well, beats more.

Storing: the part that looks trivial and isn't

Two decisions get made when something is written, and both haunt you later.

What to keep. Saving whole transcripts is cheap and nearly useless — you end up searching a haystack you built yourself. Saving distilled facts is far more retrievable but requires a judgement call about what mattered, and any automatic extraction will sometimes discard the thing you cared about. Most products sit somewhere on this spectrum, and where they sit predicts how they feel after six months more than any feature list does.

How to cut it up. Text goes in as chunks, and a chunk is the unit that gets retrieved. Too large, and a hit drags in three unrelated topics. Too small, and the sentence comes back without the context that made it meaningful. There is no correct answer, only a tuned one.

Finding: embeddings, and their blind spot

A second model — an embedding model, not the one you chat with — turns each chunk into a vector of a few hundred to a few thousand numbers. Texts with similar meaning land near each other. Your question gets the same treatment, and the search becomes geometry: return the nearest chunks.

Done exhaustively that is slow, so real systems use an approximate index — HNSW is the common one, a navigable graph that reaches a good answer without visiting every point. It trades a small amount of recall for orders of magnitude of speed, and in pgvector it is a one-line index.

Here is the blind spot nobody mentions in the pitch. Semantic search is *only* semantic. Ask for an exact invoice number, a surname, a library version, a variable name — the thing where you know the precise string — and vector similarity will happily hand you five things that are *about* that topic and not the one containing the string.

Which is why a keyword index, BM25 or a database full-text search, is not the legacy approach vectors replaced. It is the half that vectors are bad at.

Hybrid search and reranking

A serious retrieval pipeline runs both searches and merges them. The standard merge is reciprocal rank fusion: each result is scored by its *rank* in each list rather than by a raw score, which sidesteps the fact that a cosine distance and a BM25 score are not measured in the same units and cannot be added.

Then comes the step that produces most of the visible quality, and that many products skip because it costs compute. Fusion gives you maybe thirty candidates. A cross-encoder reranker reads the question and each candidate *together* and scores how well that one answers this one. It is far more accurate than comparing two independently-made vectors, and far too slow to run over the whole store — which is exactly why it runs last, on a shortlist.

One warning if you work in more than one language. Both stages are model-dependent, and plenty of embedding and reranking models are English-first. A pipeline that scores well in English can quietly degrade in French or German, and you will experience that as "it forgot", not as "the reranker was trained on the wrong distribution". If you are not working in English, check that the models are genuinely multilingual — multilingual-e5 is one such family.

When we moved our own pipeline from plain vector search to hybrid fusion plus a multilingual cross-encoder, mean reciprocal rank on our evaluation set improved by 139%. Treat that as what it is: our measurement, on our data, with our chunking. It says the step is worth taking; it is not a number you can expect to reproduce on a different corpus.

The four ways this fails

1. It was never stored. The most common failure by a distance. Nothing in the retrieval stack can find a fact that was never written, and automatic extraction misses things.
2. No threshold. If the system always returns its top five, then when nothing relevant exists it returns five irrelevant things — and the model, being agreeable, weaves them in. A similarity floor that returns nothing is a feature, and its absence is why some assistants confidently misremember.
3. Stale contradictions. You moved city; both the old and new fact are in the store, both retrievable. Without recency weighting or explicit supersession, the model may pick either one.
4. The needle problem. Exact identifiers, in a vector-only system. See above.

What to ask a memory product

Five questions, and every one has a factual answer a vendor either gives you or dodges:

• Keyword search as well as vector, or vector only?
• Is there a reranking stage, and what model?
• Is there a relevance threshold below which it returns nothing?
• Are the embedding and reranking models multilingual, and evaluated in my language?
• Can I see, edit and delete an individual memory — or is the store opaque to me?

That last one is not about retrieval quality, but it is the one you will care about first, the first time something wrong gets remembered about you.

Pryzm runs hybrid fusion with a multilingual cross-encoder reranker and a relevance threshold, on PostgreSQL with HNSW indexes, and every memory is individually visible and deletable. How the alternatives compare is in our side-by-side.