# engram / terminal
Agent memory · Paper
Engram
Does agent memory need LLM-extracted facts — or just the right raw turns?
- 77.0% vs 77.5% on 778 held-out LoCoMo questions — non-inferior under a pre-registered −5pt margin
- Raw turns are 3,061× cheaper to write than LLM-extracted memory
- Reranking adds +17.4 pts (LoCoMo) and +9.1 pts (LongMemEval) at k=3
Last login: Thu Oct 15 09:12:44 on ttys004
rishabh~/engrammain ✓make reproduce-v3
▸ stripping API keys from env ................. ok
▸ fetching LoCoMo @ pinned revision .......... ok
▸ fetching LongMemEval @ pinned revision ..... ok
▸ rebuilding numbers.json from committed results
✓ H1 non-inferior Δ −0.5 pt · one-sided 95% bound −3.0 > margin −5.0
✓ write cost 3,061× cheaper than LLM extraction
✓ check.py every number traces to a result file · 0 API calls
✓ built paper_v3/when-does-selection-replace-extraction.pdf
rishabh~/engrammain ✓
zero API calls — every number re-derived
- 77.0% vs 77.5% on 778 held-out LoCoMo questions — non-inferior under a pre-registered −5pt margin
- Raw turns are 3,061× cheaper to write than LLM-extracted memory
- Reranking adds +17.4 pts (LoCoMo) and +9.1 pts (LongMemEval) at k=3
make reproduce-v3rebuilds every number, table and the PDF with zero API calls
The question
Published results disagree on whether conversational memory for agents needs LLM-extracted facts or whether selecting the right raw turns is enough. Engram tests that question under a pre-registered plan, on LoCoMo conversations never used during development and on LongMemEval.
What I found
The context budget decides the answer. When the answer model reads few items, selection matters and raw turns hold their own: Turns + Jev at k=6 scored 77.0% against 77.5% for engram v2 at k=3 — non-inferior, with a one-sided 95% bound of −3.0 against a −5-point margin — while being 3,061× cheaper to write. When the model reads many items, extraction wins: engram v2 at k=20 is the most accurate system measured at 82.4%. The result held under blind human grading of discordant questions and with a second answer model (Llama 3.3 70B).
How it's built to be trusted
Every number in the paper traces to a committed result file. make reproduce-v3 rebuilds numbers.json, every table and figure, and the PDF, then runs a checker that fails on any number that doesn't trace back. Every result is labeled registered, exploratory or post-hoc. Along the way the study surfaced a real-world bug worth knowing: mem0 2.1.0 silently routes OpenAI calls to OpenRouter whenever an OpenRouter key is set.
Reception
Co-authored with Rishika Lall and published on arXiv with a Zenodo DOI and a pre-registration DOI. It was listed on Hugging Face Daily Papers, picked up its first independent academic citation within weeks, and the open-source PRISM project built an experiment around it. The engram-eval dataset (33k rows) has passed 775 downloads.