← All posts

Blog · On agents · November 14, 2026

When an agent’s memory and its soul drift apart

An AI agent starts with a description of its job and builds up a file of notes as it works. By the end of a busy week, those two files often disagree.

By Timothy O’Brien3 minute readFiled under On agents

If you only use AI in a regular chat window, every conversation starts mostly from scratch. You ask a question, you get an answer, and you close the tab. When you start running AI agents that do work for you over weeks or months, you run into a different problem. The agent has to remember things between sessions, and what it remembers starts changing how it behaves.

Most agent setups split an agent’s context into two main parts. The names vary depending on the software, but in the tools I use they are called the Soul and the Memory. Keeping those two files lined up takes more attention than I expected when I started.

What the soul and the memory do

The Soul is the file where you write down who the agent is and what its job is. You write it when you set the agent up. It says what the agent is responsible for, what it should leave alone, how it should talk to you, and what rules it has to follow. You usually write the Soul when you have a clear head and a clean picture of the work you want done.

The Memory is a separate file of notes that the agent writes and updates for itself as it works. If you tell the agent on Tuesday that you prefer short updates, or it figures out where a folder lives so it doesn’t have to hunt for it again on Wednesday, that goes into the Memory. You don’t want to explain your setup from zero every morning, and the Memory is what saves you from doing that.

When a new conversation starts, the agent loads both files into its context at the same time. It reads the Soul to see what its job is, and it reads the Memory to see what it has learned so far. As long as those two files say the same thing, the agent does its job.

How they drift apart during the week

The trouble is that the Soul sits still while the Memory changes all week long, usually while you are in a hurry.

On a busy afternoon you might ask an agent to help with something outside its normal job because you already have the chat open. Or you get frustrated with a mistake and tell it never to do that again. The agent is trained to be agreeable and helpful. It does what you ask, and then it writes a note in its Memory so it will remember what you said next time.

Agents also have a limit on how much they can keep in Memory, because every note takes up room in the context window. When the file gets full, the agent has to shorten or combine older notes to make space for new ones. That is where small exceptions turn into permanent rules. A note that started on Monday as “we skipped the check once because the server was down” gets shortened on Thursday to “skip the check when deploying.” The date and the reason disappear, and the shortcut stays.

By Friday, the Soul still has your original description of the agent’s job, while the Memory is full of one-time favors, rushed corrections, and workarounds from a week of actual work. Because the agent reads both files before every turn, and because the notes in Memory look like recent instructions from you, the Memory starts winning. The agent drifts away from the job you gave it, one helpful note at a time.

The weekly reset

An agent can’t clean that up on its own. If you ask it to review its own memory, it has no way of knowing which instruction from Tuesday was a one-time shortcut and which one was a real change in how you work. To the software, everything you told it looks equally true.

What works for me is a weekly check where I step back from the daily work and read both files side by side. It takes about fifteen minutes.

I start by reading the Soul from top to bottom to make sure it still describes the job I want the agent doing. Sometimes the drift happened because my own work changed during the week and I never updated the agent’s instructions. When that happens, the fix belongs in the Soul.

Then I read the Memory line by line against the Soul and delete anything that doesn’t belong there. One-off tasks from earlier in the week come out. Workarounds for problems that have already been fixed come out. If the agent learned a specific procedure that is worth keeping for one task, I move that into a separate skill file so it isn’t sitting in the agent’s general memory on every conversation. And if a note in the Memory contradicts the Soul, I decide right then which one is right and delete the other.

People talk about AI agents as if you set them up once and let them run. In practice, anything that learns from your daily conversations will pick up your shortcuts and your bad afternoons along with your instructions. In The World That Agrees With You, chapter 8 covers how to inspect what consumer assistants remember about you, and chapter 10 covers how agent memory works and how to correct it. Both chapters are free to read on this site.