LoCoMo benchmark
The numbers, unedited.
Every engine measured with Qwen3-8B running locally as both respondent and judge — the same ruler for all of them, so the head-to-head is fair. These numbers are not comparable to mem0 or Zep's official leaderboards, which use GPT-4o. What this measures is how each memory does against the others under identical conditions.
Full results
LoCoMo · 1,986 questions · accuracy, LLM-judged
| Category | What it measures | Corvusmemfree · offline | Embedding e5free · vector DB | mem0paid · cloud |
|---|---|---|---|---|
| Single-hop282 | Direct fact lookup | 23.0% | 30.5% | 25.2% |
| Temporal321 | Dates & ordering | 16.2% | 17.4% | 5.3% |
| Multi-hop96 | Cross-referencing facts | 17.7% | 24.0% | 17.7% |
| Open-domain841 | Fact + common sense | 41.6% | 44.7% | 35.0% |
| Adversarial446 | Knowing when to say "I don’t know" | 5.6% | 3.4% | 4.7% |
| Overall | weighted, all categories | 25.6% | 28.0% | 21.1% |
Bold marks the best score in each row. Embedding e5 leads most categories, but it needs a vector database to run. Corvusmem is free, offline and server-less, wins outright on adversarial questions (knowing when to abstain), and nearly doubles mem0 on temporal reasoning — see below.
Where it pulls ahead · temporal reasoning (cat2)
16.2%
Corvusmem
5.3%
mem0
+206% vs mem0
Corvusmem stores every fact with its explicit date and temporal context. That's why it almost doubles mem0 when a question depends on when something happened — the kind of recall long-running agents need, not just what happened.
Methodology
Same ruler for everyone
One local model, Qwen3-8B, answers every question using each memory's recall, and the same model judges every answer. No engine gets a stronger judge than another. That's what makes the comparison fair — and what makes it incomparable to leaderboards scored with GPT-4o.
The dataset
LoCoMo: 1,986 questions over long, multi-session conversations, split into five categories. Accuracy is the share of questions each engine answers correctly, as graded by the judge. Nothing is dropped or re-weighted in your favor.
A note on Zep / Graphiti
Running Zep over the full dataset locally wasn't feasible, so we only have a single-conversation sample (conv-1). It is not a full-dataset result and is kept separate so it never gets mixed into the table above.
Sample only · conv-1
| Corvusmem | 28.2% |
|---|---|
| Embedding e5 | 30.3% |
| mem0 | 14.9% |
| Zep / Graphiti | 18.5% |
Reproduce it
The table above isn't typed by hand — it's generated from the raw result files. From the repo root, point the comparison script at the three result sets and you get the exact same numbers:
Source of truth: the raw resultados-e2e-locomo-*.json files and BENCHMARK.md in the engine repo.