← All posts

Blog · Inside the research · November 10, 2026

Filtering out the world

A new peer-reviewed study shows write-time memory defenses catch poison and reject legitimate learning at the same time — because a correct update and a poisoning attempt are indistinguishable on content alone.

By Timothy O’Brien5 minute readFiled under Inside the research

This is the fourth thing I have written about agent memory this month. First the survey, then the attackers, then the market — and each time the proposed fix stayed the same: put a bouncer at the write path. Screen every new memory as it comes in, and reject whatever contradicts what is already stored. It sounds like simple hygiene. Today a peer-reviewed study finally put a price tag on that bouncer, and the price tag is the thing you were trying to protect.

The experiment

The paper is “Cross-Session Memory Poisoning of AI Agents in Connected-Vehicle Security”, published November 10 in Information. The setting is a vehicle security operations center: AI agents triage security alerts by retrieving precedents from earlier sessions. That makes the memory layer an attack surface with a delay built in — an adversary writes a few false precedents today and is long gone when the wrong decision happens next month.

The tested defense is the standard one: write-time filtering. Anything a new record contradicts gets rejected. And in the baseline case the defense works exactly as advertised — when the world is fixed, the filter recovers real safety at no cost. Most published evaluations stop there, and that is precisely the problem. Evaluations hold the correct answer still. The real world does not.

So the authors evaluated the same defenses where the world moves: real vehicle traffic, where a fleet refresh shifts the alert distribution, and a synthetic control where the correct label itself changes. Under that drift, the numbers flip. Against a poisoned agent, the write-time filter recovered 5.4 points of performance. Provisional trust — admitting new records at reduced influence until outcomes corroborate them — recovered 18.0. Worse, the filter paid about 12 points of adaptation cost when the world legitimately changed, versus 1.3 for provisional trust, which is statistically indistinguishable from zero. On a language-model agent, provisional trust gained 30 points where the filter gained 0.0.

Indistinguishable on content alone

Why does a safety filter make an agent worse? Because when the world legitimately changes, a correct update and a poisoning attempt are indistinguishable on content alone. “That model can’t do what the docs say” is either a lie an attacker planted or the first honest report of a firmware change. The bits are identical. A filter that rejects contradiction doesn’t know which one it is rejecting, so it rejects both. Rejecting contradiction is rejecting learning — with the filter turned on, the agent becomes exactly what a security operator fears most: confidently fluent, permanently current as of the day it was deployed.

The paper’s closing sentence deserves to be pinned above every memory-roadmap meeting: “Memory defences cannot be assessed in a stationary setting.” Every bench result showing 90-something percent poison rejection is measured in a world where the correct answer never changes. In the world where your users actually live, that number buys you a steeper cliff on the other axis.

Nobody priced the cost

I run a registry of agent-memory products at worldagrees.com/registry — 51 entries as of this week. Before writing this I searched it for the trade-off. Exactly one entry contains the word “poison,” and it is a supply-chain aside about malicious skills, not a discussion of defense cost. Not one vendor page says what this paper demonstrates: that a rejection-based memory defense and an adaptable agent are pulling in opposite directions. When I wrote Nobody sells memory integrity, the gap was that the write path went unchecked. It is worse than that. The check everyone proposes to add has a defect the marketing pages do not mention, because the defect and the feature are the same property.

The pencil, not the bouncer

Provisional trust is not a softer bouncer. It is a different arrangement entirely: new records get in at reduced influence, they prove themselves through outcomes, and influence follows corroboration. A grace period instead of a verdict. That is the architecture I argued for when I built my own memory ledger around staleness and provenance — a note’s authority should come from when it was written and what has since confirmed it, not from a one-time admission decision. The paper’s own caveat is worth respecting: on the language-model agent, the gain was significant with frozen memory but not with persistent memory. Provisional trust helps; it is not yet a finished answer.

But the direction is clear, and it runs against the current instinct. Security people want the notebook locked down; product people want it comprehensive. The honest answer is that a memory you can never overturn is not safe, and a memory anything can overwrite is not yours. The middle path is slower, per-entry, and boring: let new facts in quietly, and promote them only when the world confirms them. The book traces what happens without that humility — how a system trained to agree, given a memory, gets better at agreeing over time (chapter 5, chapter 8). The registry exists so this machinery gets watched while it assembles. Today the literature handed it a standard: if your defense can’t tell poison from news, it can’t tell news from anything.