I spent a few days this week in the open-source projects that give AI agents a memory, and I came out with one sentence I can’t shake: whoever writes the memory owns the agent.
Not whoever trains it. Not whoever hosts it. Whoever writes what it remembers.
The point
Here’s why, in plain terms. An agent is only as good as what it remembers. It walks into every new session carrying what it kept from the last ones — who you are, what worked, what it decided last time. It treats that record as fact. It has to; it has nothing else to go on.
So if someone slips one false fact into that record — a fake preference, a wrong permission, a lie about what happened before — they don’t need to touch the model at all. They’ve changed what the agent will do tomorrow. And next week. And in every session after that, because the agent doesn’t just recall its memory, it learns from it. It builds on what it believes.
And here’s the part that gets me: the agent will defend the false fact. As far as it knows, it learned it. It isn’t lying. It’s being loyal to something a stranger wrote in its notebook.
This isn’t me being paranoid. It’s now its own research field — there’s a survey this year on attacks, defenses, and governance for agent memory, and the reason it exists is that memory attacks are worse than the prompt-injection attacks everyone worried about first. A prompt injection dies when the conversation ends. Poisoned memory doesn’t. It compounds.
Why memory is the soft spot
If you’ve used an AI assistant, you’ve already seen a small version of this. It “remembers” you like spicy food because you mentioned it once. Mostly harmless. Now put an agent behind it — one that books things, buys things, sends things, acts. What it remembers about you stops being a convenience and becomes instructions.
That’s the whole ballgame. Memory isn’t a nice feature bolted onto an agent. It’s the steering wheel.
So I wanted to know: when teams actually build this, what are they building? There are four serious open-source projects, and they disagree about what remembering even is.
Four ways to build a memory
Mem0 works like a note-taker following you around. After each exchange it quietly writes down what it thinks matters — that you like short answers, that your team ships on Thursdays. You never designed the memory. It got distilled out of your conversations. About 63,000 stars on GitHub, the most popular of the bunch, and the easiest to bolt onto something you’ve already built.
Letta — it used to be called MemGPT — goes the other way. The agent keeps its own notebook and edits it. There’s a small page it can always see, a searchable history, and a big archive, and the agent itself decides what moves between them. There’s even a background process that reviews the day and writes itself lessons. The trade: this one doesn’t bolt on. It wants to run the whole agent.
Graphiti, the open core of Zep, treats memory as history. Every fact gets two stamps: when it became true, and when it stopped being true. When something new contradicts something old, the old fact doesn’t get erased — it gets marked as over. So you can ask what the agent believed last Tuesday, and get an honest answer. Remember that when we get back to security. It matters there.
Cognee builds its memory from your documents instead of your chats, and it’s the one that takes correction most seriously — tell it it got something wrong, and the correction sticks.
One warning about the numbers
If you go shopping in this category, ignore the scoreboards. Nearly every number you’ll see was produced by the company that sells the product. Mem0 says it scores 94.4 on a benchmark called LongMemEval. The one clearly independent measurement I could find of that same benchmark scored it 49. Same test, different referee, a 45-point gap. Ask one question every time: who ran this?
And the “open source” label needs a footnote too. All four projects are genuinely open. But Zep’s ready-to-run community server is discontinued — you’d run Graphiti yourself plus your own graph database, which is a real job. Mem0’s best memory features sit behind its $249-a-month cloud plan. The license is free. The operation usually isn’t.
What good memory looks like
So back to the point. If memory is the steering wheel, here’s what I’d want from mine. I’d want to be able to read it — all of it, no proprietary format, no “trust us.” I’d want to know where every fact came from. And when something in it turns out to be wrong, I’d want the wrong version kept and marked, not quietly overwritten — so I can see what the agent believed, when it learned better, and who told it.
That last one is exactly what Graphiti does with its two stamps, and it’s why I keep coming back to it. A memory that can’t lie about its own past is harder to poison usefully. The attacker can still write a false fact. But the history shows the false fact arriving, and from where.
This connects straight to the argument in chapter 8 of the book. A system that learns your preferences serves them back to you whether or not anyone is paying it to. What the memory keeps is the version of you the agent keeps agreeing with — and I made that case from the other side in The agent that remembers. The flattering memory and the poisoned memory are the same machine pointed in different directions. Both of them decide what the agent will believe about you tomorrow.
So that’s the takeaway, and I’ll keep it short. When someone shows you an AI agent, don’t ask what model it runs. Ask what it remembers, who can write to that, and whether you can read it. The model is the engine. The memory is the driver.