Every number, with its methodology attached.
This is the evidence base behind the comparison page: how the tests work, what the variance looks like, and the full sweep — including the benchmarks where Memomee trails. Every figure is sourced to a committed evaluation run; the raw artifacts and harness will be published, and each table below names the run it comes from.
Three tiers, three different strengths.
Not all measurements are equal. Read every figure against the tier it comes from. The decision-grade head-to-head lives on the comparison page.
Clean-fixture suites with zero-tolerance gates: multi-hop bridge recall, contradiction precedence, abstention-on-conflict, cross-user isolation, governed forgetting. No live model — a green run means the same thing every time. Gated on every change.
End-to-end runs on a real model with the answerer, grader, embedder, sample, and seeds held constant across systems — only the memory strategy changes. Real answer quality, recall, and cost. The comparison page's decision-grade head-to-head lives in this tier.
One lever at a time: extraction fidelity, the anti-saturation breadth sweep, the accuracy-vs-token trade-off, the MCC protocol. Advisory, committed, and the source of most tables on this page.
The setup that makes comparisons fair.
One shared sample, one answer-writer (deepseek-v4-flash), one grader (deepseek-v4-pro[1m]), one embedder (Voyage), one turn granularity. Only the memory strategy differs — so differences within a test are unbiased. Absolute scores are not comparable across tests.
Head-to-heads run three independent seeds (11 / 12 / 13), 50 questions each; sweep runs are n=3 over 20-question samples (MemoryAgentBench spans 145). Tables show per-run standard deviation, and we call a lead inside run-to-run noise a tie — including when it's ours.
Every token figure here is the assembled read-context — the tokens memomee actually hands the answer model per question — not answer-generation tokens and not transcript size. One honest 'tokens read per question' number, page-wide.
Session-level recall asks whether the right evidence session reached the model; the stricter turn-level recall@k asks whether the exact gold turn survived ranking. Both are honest; we label which is which and never mix them in one figure.
One engine, every venue — including the losses.
Two of the twelve — the LongMemEval and LoCoMo head-to-heads against RAG, mem0, and a full-context reader — live on the comparison page, where they do their decision-grade work. The other ten are here: long-conversation stress tests, multi-hop QA, conversational memory, and the million-token tier.
Long-conversation stress tests · run eval-results/e2e/2026-07-04 (+ 2026-07-15-mab-clean) · n=3, sample-20 (MAB: 145)
| Benchmark | memomee | full-context | recall@k | Read tok | vs full-context |
|---|---|---|---|---|---|
| PersonaMem | 0.65 | 0.55 | 1.00 | 2,410 | edges |
| BEAM | 0.30 | 0.25 | 1.00 | 2,416 | edges |
| MemoryAgentBench | 0.45 | 0.60 | 0.26‡ | 2,286 | ~parity‡ |
| BABILong | 0.30 | 0.50 | 1.00 | 226 | trails |
| Memora | 0.146 | 0.174 | 1.00 | 1,421 | trails |
| AMABench | 0.10 | 0.20 | 1.00 | 2,417 | trails |
Multi-hop QA · run eval-results/e2e/2026-07-05 · SQuAD answer-F1 · questions that need two or more facts chained
| Benchmark | memomee | full-context | recall@k | Read tok | vs full-context |
|---|---|---|---|---|---|
| HotpotQA | 0.668 | 0.674 | 1.00 | 1,518 | ~parity |
| 2WikiMultihopQA | 0.438 | 0.596 | 1.00 | 976 | trails |
| MuSiQue | 0.317 | 0.553 | 1.00 | 2,072 | trails |
Conversational long-term memory · ConvoMem · run eval-results/e2e/2026-07-05
ConvoMem's dataset is CC‑BY‑NC‑4.0 (non-commercial), so we run it for internal reference only — opt-in, never part of a default run. Memomee matches the full-context reader here (0.95) reading 878 tokens of assembled context: one of its strongest results.
Beyond a million tokens · BEAM‑1M · run eval-results/e2e/2026-07-06-beam1m · the tier Mem0 headlines
At around a million tokens, a full-context reader can't hold the conversation at all — so Memomee edges it (0.35 vs our matched 0.30 baseline) reading a compact 2,414 tokens of assembled context out of the >1M in the transcript. This is the large tier some vendors headline; read our 0.35 against our own full-context baseline in the same setup, never against a vendor number from a different one.
The vs full-context column is the quick read: Memomee edges the reader on PersonaMem and BEAM — including BEAM's million-token tier, where a full transcript no longer fits at all — matches it on ConvoMem, runs about level on HotpotQA and MemoryAgentBench, and trails on the rest, all while reading a compact ~0.2–2.4k-token context per question instead of the whole conversation. On nearly every test retrieval still finds the evidence (recall@k near 1.0), so where Memomee trails, the gap is in writing the answer — a known synthesis deficit on multi-hop categories, not a retrieval deficit. MemoryAgentBench's 0.26 recall is internal pool-coverage, not a competitive gap: the full-context baseline's recall is 1.0 by construction, so the two aren't comparable. On the paper-comparable metric — SubEM on Accurate-Retrieval, an LLM-judge elsewhere — Memomee scores 0.45 vs 0.60, with about two-thirds of the exact-match misses already having the answer in context, paraphrased rather than quoted; only ~15% is a genuine retrieval gap.
MemoryAgentBench's evidence is chunk-tagged differently from the other venues, and the full-context baseline reads the whole document. Test-Time-Learning is scored separately on the paper's dedicated exact-label MCC protocol — below — never folded into the 0.45.
Where memomee sits among named systems
cross-regime · not apples-to-applesMemoryAgentBench's classification metric — MCC, the Matthews correlation coefficient, a chance-corrected measure of classification quality — is the one place we can line memomee up against published competitor numbers — run eval-results/mab-ttl/2026-07-17 (n=3 × 5 datasets, pooled 95% CIs). Read it as rough placement, not a controlled test.
memomee beats the commercial memory agents (Mem0, Cognee, MIRIX) and trails HippoRAG-v2 and the long-context oracle. But the figures are not comparable: memomee's 0.49 (95% CI ±0.11) uses its own DeepSeek answer model and discrete-event ingest, while the competitor and oracle numbers are quoted from the MemoryAgentBench paper (arXiv:2507.05257) on its own backbone. The long-context oracle here (the paper's 0.82–0.89 range, bar drawn at its 0.86 midpoint) is a different protocol from memomee's own 128-shot many-shot ceiling of 0.64 (±0.11).
How to read anyone's benchmark — including ours.
Five checks that disarm almost every misleading memory benchmark you'll meet. Apply them to this page first.
- 01Same answerer and grader across systems? If not, the model difference swamps the memory difference — that's why vendor leaderboards can't be compared to each other or to the tables above.
- 02Is the regime stated? Sample size, seeds, turn granularity, the exact questions. A score without a regime is a marketing number.
- 03Is variance reported, or is it one lucky run? A lead smaller than the run-to-run spread is a tie — ours included.
- 04Which recall is it? Session-level (did the right evidence reach the model) and turn-level (did the exact gold turn survive ranking) are both honest — and answer different questions. Check which one you're being shown.
- 05Who is the baseline? A matched full-context reader run in the same setup, or a number quoted from someone else's? Only the first is a comparison.
Committed artifacts, named runs, published soon.
Every figure on this page comes out of a versioned evaluation harness — the same harness that gates releases — and each run is committed as a machine-readable artifact with its config, seeds, and scores. The raw run files and the harness will be published; the tables above already name the exact run behind every figure, so nothing here is a hand-typed number.
What we don't claim
- The best raw recall. Memomee finds the right evidence about as often as a reader that sees everything — it does not out-retrieve a system that reads the whole transcript.
- A broad answer-quality win over full context. A full-context reader still scores higher on both head-to-head tests and on most of this sweep; Memomee edges it only on a few saturated or beyond-context venues — always at a large token discount.
- That memory broadly beats full context. Our own stress tests disprove that.
- What we do claim: parity on finding the right evidence, an edge over open-source memory in a fair test — decisive over mem0, within run-to-run noise over RAG — a 5–11× token discount on long conversations, and governance the alternatives don't offer.