Five agent-memory papers, September 19–25: what memory costs, when it pays, and how far its evidence reaches
Five primary-source papers from September 19–25 show what published memory systems return on real repository work, what memory costs in dollars and seconds, what a poisoning defense bills on clean traffic, how an index of addresses beats a summary, and why an accepted skill has to stay inside the family that certified it.
From September 19 through September 25, 2026, five primary-source papers in AI agent memory converged on one accounting problem: what a memory mechanism returns has to be measured against the run that did without it, at the cost and latency it actually adds, on the traffic it actually meets, and inside the scope the evidence behind it covers. Two of this week's papers measure published memory systems losing to no memory at all, one redesigns a defense so its bill lands where it is being paid, and two propose mechanisms that come out ahead. The change of unit across all five is the same one, from the recalled fact to the finished task.
A poisoning defense's bill lands on the traffic nobody attacked
Date: September 19, 2026
What happened: A measurement study holds the memory backend, retrieval configuration, answering model, question set and judge fixed across conditions and changes only the defense. Five conversations truncated to 150 turns run three times each, giving 90 cells, and a judge from a different provider than the answering model scored the answers with Cohen's κ = 0.909 against human labels. The measure is a paired accuracy delta on traffic containing no attacks, so every point it moves is the defense damaging ordinary work. 1
Three write-time defenses — lexical input sanitization, provenance tags that down-weight lower-trust sources, and an LLM anomaly detector that quarantines candidate memories inconsistent with the live conversation — cost nothing the study can resolve on clean traffic; their 95% bootstrap intervals span about ±4.5 points and include zero. The read-time reranker, which re-scores retrieved memories against the query and against each other, lowers core accuracy by 4.4 points (95% CI [−9.0, −0.05]; McNemar p = 0.064) and quarantines legitimate memories on 33.6% of the items it adjudicates, reaching 106 false quarantines inside one conversation at 2.7% token overhead. Stacking all four does not compound the loss: the combined condition's interval includes zero, which suggests the write-time filters offset part of what the reranker discards. 1
Why it matters for builders: A defense is paid for on every interaction and earns its keep in a small minority of them, so the benign case is where the bill arrives. Instrument the read-time stage separately from the write gate: the write-time defenses here spend an LLM call per write and show no measurable accuracy cost, while the read-time filter spends accuracy and discards a third of the legitimate records it sees. Reproduce that false-quarantine rate on your own store before shipping a reranker, because discarding a third of good memories changes what the agent knows about the user.
A memory system's grade depends on the agent it is plugged into
Date: September 21, 2026
What happened: DolphinBench scores memory through task completion inside a simulated work environment, on the argument that a question announces which fact has to be retrieved while an action makes the agent work out that a fact is needed at all. Three personas — a startup CEO, an infrastructure engineer and a product manager — each carry several years of history, 3,400 to 5,128 user messages and roughly 500,000 tokens, with tools such as Notion, Gmail and GitHub simulated. The suite holds 600 tests, 200 per persona, each needing one to four separately graded actions, and every test is certified by oracle runs with the relevant history that must pass and runs without it that must fail. Accuracy, total cost and median latency are all required in the report, so a configuration lands on an accuracy-cost-latency frontier rather than in a single ranking. 2
The highest score in the table is 70.67%, for Hermes with Mem0 on GPT-5.6-Luna. Against Hermes' built-in memory (65.67%), Mem0 adds five points and cuts median task latency from 44.35 to 37.69 seconds while total cost rises from $61.48 to $96.21. Hindsight scores 69.50% at $84.65 and 55.31 seconds, against Honcho's 68.50% at $142.85 and 44.66 seconds. Supermemory is the most expensive configuration at $345.64 and scores 59.17%. The ordering changes with the harness: Mem0 leads under both Hermes models, and Honcho leads under Claude Code with Claude Sonnet 5, at 35.83% against Mem0's 32.33%. 2
Why it matters for builders: Cost and latency belong in the result because they decide which system wins, and an accuracy-only table hides a $345 configuration that scores below a $61 one. Test the pairing you intend to ship, because the same memory system ranks first under one harness and second under another, and because the model behind the harness moves the outcome more than the memory layer does: this Hermes-plus-Mem0 setup reaches 70.67% on GPT-5.6-Luna and 47.83% on MiniMax M3.
Real repository work is where the published memory systems lose to no memory
Date: September 20, 2026
What happened: VibeMemBench asks whether a memory system improves executable repository work, and builds a matched intervention to answer it: two runs of the same task share the model, prompt, tools, sandbox image, context budget and resource limits, and only the declared memory condition changes. Executable tests decide resolution, so the score is a pass rate on real code rather than a recall rate. The corpus is 111 targets from 90 SWE-rebench V2 repositories with 3,634 completed history trajectories, and a target is retained only when prior experience measurably improved a reference solver on it, which makes this a test of whether verified experience transfers. 3
Injecting one fixed, verified experience — bug class, root cause, fix pattern, reusable lesson — raised resolution on four of five held-out solvers by 1.1 to 4.5 percentage points and lowered agent steps on all five, with every interval crossing zero at 111 targets. The negative half is the part to read: when Mem0, SimpleMem, MemoryOS and A-MEM had to construct and retrieve experience from the same history, 11 of 12 solver-and-system pairings finished at or below the matched memory-off baseline, and glm-5 with Mem0 lost 5.5 points with an interval entirely below zero. 3
The mechanism sits in the headroom. 560 of the 1,332 retrieval pairings are targets the solver already resolved on all four seeds without memory, where an injected record can only remove a seed; a dose curve on Mem0 stays below the memory-off baseline at every budget from one record to five. Where the solver failed the target without memory and the record reached at least a directory or identifier anchor, the record raised resolution on 55.2% of pairings and lowered it on none. 3
Why it matters for builders: The sign of an injected record depends on what the agent could already do, which makes expected headroom the useful gate and a matched memory-off run on your own tasks the useful measurement. Count the targets your agent already passes before tuning retrieval: that stratum is where injection costs seeds and buys nothing. For the stratum that fails, compress the retrieved trajectory to the span that states the fix and keep the identifiers that anchor it to the repository.
An index is an address, not a summary
Date: September 23, 2026
What happened: EnSIMem keeps the original interaction text and builds an index that points into it. Neighbouring turns are grouped into theme-coherent episodes that split at a sustained topic or task change, so a speaker change alone keeps one episode together. Each episode is indexed as dialogue-grounded entity-property records — entity, type, property, and an optional concrete value, each pointing back to its episode — and properties are held at an intermediate granularity, broad enough to match a paraphrase and specific enough to keep an event's identity. At query time a planner breaks the question into atomic requirements, classifies it as point, temporal, compositional or aggregation, sets an evidence budget from that type, retrieves through the structured index and dense search together, and stops on requirement coverage rather than on a similarity threshold. Answers come from the retrieved source turns along with their speaker, timestamp and multimodal fields. 4
On LongMemEval the system averages 92.8% under both evaluation models, 5.4 points above the strongest baseline the paper reports (MEMORA at 87.4%) and far above the full-context baseline at 65.6%, with the largest gains on single-session-assistant and multi-session questions. On LoCoMo it reaches 90.6% with GPT-4o-mini judging and 90.0% with Qwen3-32B, against 86.3% for MEMORA and 82.5% for full context. Holding the answering and the evaluation model fixed, the structured access path scores 96.59% against 86.36% for dense retrieval over the same material. 4
The cost lands at query time: 14.4 seconds mean end-to-end latency, 1.92 retrieval steps per query on average, with planning at 71.1% of latency, retrieval at 11.0% and generation at 17.9%. The paper reports its own latency beside MEMORA's published 5.70 seconds and says the two figures come from different engines and execution settings, so it draws no comparison from them. 4
Why it matters for builders: An index of addresses pays at query time and keeps the source text intact, which is a different budget from summarizing history up front and trusting the summary to have carried the detail. The ablations put the gain in the access path rather than in denser retrieval: 96.59% against 86.36% with the same material and the same answering model. Budget for the planner, since it consumed 71% of this system's latency and is where a cheaper implementation would change the unit economics.
Certification scope and retrieval scope have to agree
Date: September 24, 2026
What happened: This paper names a failure it calls cross-family interference: an edit certified as valid inside one task family, deployed through a single compact global skill, changes behaviour on unrelated families. On ProcStream-RSI, a 12-round code-repair stream where a frozen Qwen3-Coder-Next agent rewrites a 360-character skill, one accepted boundary rule worth +0.398 hidden utility inside its own family moved missing, rotation, tie-breaking and chunk families by −0.461, −0.234, −0.148 and −0.117. Across eight main streams, deploying the eight accepted updates globally was harmful in six, and mean trajectory utility fell to 0.713, below the 0.775 of a static agent that never updates at all. 5
The gate in the study, Orthogonal Regression Control, compares incumbent against candidate on four public examples, four private exact-output tests and at least six answer-free metamorphic checks, and deploys only when a one-sided 95% paired-bootstrap lower bound clears a margin in every family group and every channel, so a gain in one channel cannot average away a regression in another. Then the paper changes exactly one thing: it keeps the same gate decisions and the same accepted skills, and retrieves each skill only for the family it came from. Mean trajectory utility rises to 0.816 with zero harmful deployments, a paired advantage of 0.103 over global deployment. 5
The full extension runs 27 paired 12-round streams. Scoped deployment reaches 0.848 mean trajectory utility against 0.785 for global deployment, accepts 63 updates against 12, produces multiple accepted updates in 19 of 27 streams against 2, and records 0 of 63 harmful acceptances against 6 of 12. On the same evidence, the gate's own positive predictive value is 1.00 on same-family probes and 0.25 on the next global checkpoint: the gate was right about the family it tested and wrong about everywhere else. 5
Why it matters for builders: A persistent store makes two decisions, and gates have mostly been built for the first one: whether an update is supported, and where it applies. Storing the family alongside an accepted rule and retrieving by family keeps a certified gain from leaking into unrelated work, and it turns a store that stops accepting after its first update into one that keeps adapting. Record the task family with every accepted rule and scope retrieval to it.
Issue summary
Three patterns run across the five items.
- The control condition became part of the result. VibeMemBench's headline is a comparison against the same agent with memory switched off on the same targets; DolphinBench refuses a ranking that omits cost and latency; the defense study measures utility on conversations with no attacks in them. Each of those decisions settles what a number means, and each is a run a builder can stand up on their own stack.
- Every mechanism has a place where it does harm. A record injected into a target the solver already resolves removes seeds; a read-time reranker discards legitimate memories on a third of the items it adjudicates; a boundary rule that pays inside its family loses points outside it. Each paper states the size of the damage, and in each case the damage sits in a stratum that can be counted before the mechanism is deployed.
- The bill arrives where the work happens. EnSIMem spends 71.1% of its latency in query planning and keeps the source text behind the index; the anomaly detector spends an LLM call per write and shows no measurable accuracy cost; the reranker spends accuracy on every retrieval. A component's position in the pipeline predicts its cost more reliably than whether it calls a model.
One concrete next action
Build a matched memory-off baseline before changing any retrieval code.
- Run the pair. Take 50 real tasks your agent performs today and run each twice with the same model, prompt, tools, context budget and seeds — once with memory, once without. Score resolution with whatever check already decides success, and record agent steps and input tokens beside it.
- Split by headroom. Mark which of the 50 the agent already completes without memory. That stratum is where an injected record can only cost you, and the share of tasks sitting in it tells you whether your write gate or your retrieval trigger needs the work first.
- Check the anchor. For the tasks the agent fails without memory, count how often the record your retriever would inject reaches a directory or identifier in the repository. That anchor is the case VibeMemBench measured at 55.2% of pairings with no observed loss, and a day with an annotated sample tells you whether you produce one.
References
- 1
- 2
- 3
- 4
- 5
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.