Pryzm Memory benchmarks: method, results and where it loses

1. Retrieval on a synthetic dataset

Original run 2 October 2026; engine v3 measured from 3 October 2026.

Question: when a question arrives, does Pryzm's search return the useful memory, and at what rank? No language model is involved: this measures “finding”, not “answering”.

Method: a fictional person (a freelancer who co-founded a fictional SaaS), no real data. 93 core memories and 1,907 template-generated distractors (72% French, 28% English) that share vocabulary with the answers without answering. 223 questions: 135 simple (French paraphrases, another language, indirect wording, exact identifiers), 24 needing two memories, 24 about a fact that changed, 40 with no answer. Memory sizes from 100 to 2,000. Confidence intervals: percentile bootstrap, 2,000 resamples over questions; comparisons are paired (same questions). Baselines: keyword search alone (BM25) and semantic search alone with the same embedding model (e5-small).

The v3 row comes from the published production run, which reports R@1 and R@5 only.

Against keyword search, the original engine gains +0.165 (+0.089 to +0.237) MRR, mostly when the question is not in the memory's language. Against semantic search alone with the same model, it gained +0.022 (−0.035 to +0.075): not significant. Engine v3 does better on this point (+0.085 (+0.030 to +0.140) on the same set).

Because v3 was tuned on this dataset, it was then checked once on a second dataset generated beforehand with different templates, whose questions were never seen during tuning:

Pasting your notes instead: with a fixed text of the most recent memories (20,000 characters), every useful memory is present for 100% of questions at 100 memories, more than Pryzm's top 5 (78% (72–84%)). At 2,000 memories it falls to 26% (20–33%), while Pryzm's top 5 holds 64% (58–71%) (original engine). This fixed text is an approximation of a summarised memory, not how ChatGPT or Claude actually work.

Where it loses:

• Facts that changed: with the original engine, the old version ranked first 71% of the time. Engine v3 ranks the updated fact above the old one 79% of the time on this set, 71% on the held-out set.
• Questions that need two memories: 8% complete in the top 5 with the original engine; 67% with v3 on this set, 42% on the held-out set.
• Questions with no answer: the search never says “nothing found” (0 of 40). A “nothing reliable” signal was tried and not shipped: 28% false alarms on the held-out set.
• Speed: the 95th-percentile search time was 550 to 835 ms depending on the run, on a server shared with production. It missed our target and is not a service guarantee.

2. LoCoMo, a public benchmark

Run on 4 October 2026, production engine settings.

Question: on a dataset we did not write, does the evidence come back? LoCoMo contains 10 long conversations between two people; each question lists the dialogue turns that hold the answer. Every turn became a dated memory (5,882 memories); we check whether the evidence turns are in Pryzm's results. Categories 1 to 4 (1,532 questions); the adversarial category is excluded, as is usual. No language model anywhere.

Pryzm scores above both baselines in every category; on open-domain questions the sample is small and the intervals overlap. Multi-hop and open-domain questions remain weak.

Not comparable with the “J” scores other memory systems publish for LoCoMo: those grade an answer written and judged by language models. Here only retrieval is measured.

3. Reliability: what counts, and “nothing was decided”

Measured on 10 October 2026 and 11 October 2026.

Question: when the notes hold an old amount, a proposal that was never approved and the final decision, does the AI answer with what counts? And when nothing was decided, does it say so instead of filling the gap?

Method: 60 fictional scenarios in 8 families (old → proposal → decision, revised decision, never settled, never discussed, cancellation…), three questions each. Tuned on 26 scenarios; the figures below come from a test part written afterwards: 16 scenarios, 5 unseen projects, 48 questions, measured once. Claude Sonnet, with no tools, receives either the project's full notes or Pryzm's search results; two runs, 96 answers per condition. Graded by rules, then by a Claude Haiku judge for unclear cases; 30 judgements checked by hand, 28 agreed.

On topics never decided or never discussed (24 answers per condition), no invented answer in any condition. In the search itself, the version that counts comes first 75% / 72% of the time (300 / 1,500 memories), against 44% before.

Limits: fictional, short data; with 96 answers per condition, a 5-point gap is fragile. One answering model. “Never discussed” is not detected by the engine: the AI says it, from the results and the tool's instructions.

4. Tokens: memory versus pasting all your notes

Measured on 11 October 2026.

Question: at equal accuracy, does connecting a memory make the AI read fewer tokens than pasting the whole notes file into the conversation?

Method: long fictional memories, like the ones an AI writes (672 characters on average). Test part written after tuning: 4 unseen projects, 128 memories, 32 questions, with 256 other memories as noise (384 in all), measured once. Claude Sonnet, no tools, two runs: 64 answers per condition. Input tokens are those billed for the question (instructions + notes received).

Limits: connecting Pryzm adds a fixed 6,730 tokens to each conversation (tool descriptions and instructions, measured in Claude Code), so the saving depends on how large the notes you used to paste are: with a small notes file, pasting it can cost less. One answering model; with 64 answers per condition, a 3-point gap is fragile. Search results later gained a few fields (+4% output, not re-measured at answer level).

5. The AI's built-in memory

Measured on 13 October 2026.

Question: do Claude's or ChatGPT's own memories get in the way of Pryzm?

Method: Claude Sonnet with a test Pryzm server exposing the real tools and instructions, fictional memories. 44 messages in four groups, two runs: 88 conversations per setup. The built-in memory is simulated by a summary of past chats added to the conversation, deliberately out of date on one fact; an answer is outdated when it quotes the old value (8 checks per setup).

With the built-in memory on, the AI answers from its own older summary instead of checking Pryzm. Pasting Pryzm's instructions limits the damage but does not remove it: this is why the setup guide asks you to turn the built-in memory off. Limits: a simulation (the real apps inject their memory differently), one model, a small sample.