Most agents don't fail loudly.
They fail by losing the plot.
An agent starts strong. Then the workflow branches. A session ends. A fact changes. A summary compresses away a key decision. Retrieval pulls in context that looks relevant but is wrong. Now the agent is still running — still fluent, still confident — but no longer grounded.
This is the failure mode the industry keeps misdiagnosing as a model problem. LongMemEval — a benchmark built specifically for long-term memory — reports that commercial chat assistants and long-context models drop roughly 30% in accuracy across sustained interactions. Not because the models got worse: because the memory underneath them did. Here are the five ways that happens, what each one costs, and the property that prevents it.
Write → manage → read. Every failure below strikes at a stage — and every one is a management problem, not a storage problem.
Summary drift
The agent remembers the meeting. It forgot why you changed the database.
A support agent runs on rolling summaries. Three weeks in, a customer asks about the refund exception they were granted — the one the team agreed to on a Tuesday call. The summary says 'billing discussed.' The exception is gone. It wasn't deleted; it was compressed, one paraphrase at a time, until it stopped seeming important.
Summaries are written by a model deciding what's worth keeping, and each re-summarization summarizes the last summary. Small omissions compound. The detail that survives is the detail that looked generic enough to keep — and the decisions that matter are rarely generic.
The team stops trusting the agent's recollection and starts keeping a parallel record by hand — which was the work the agent was supposed to eliminate.
An append-only raw record underneath every summary. Compress all you want — the original event still exists, and the summary is a view, not the truth.
Retrieval mismatch
The agent found something relevant. Relevant to a different problem.
An engineering agent is asked to 'finish the auth fix from yesterday.' Similarity search returns chunks about authentication: a library comparison from March, a tutorial excerpt, an old incident review. All genuinely about auth. None about yesterday's fix — which lives in a deployment thread that never used the word.
Embedding similarity measures semantic resemblance, not relevance. It has no model of time, causality, or what the agent was actually doing — so 'what changed since yesterday' retrieves nothing, because the answer shares no words with the question.
Context pollution: the window fills with plausible, wrong memory. Answers get confidently specific about the wrong thing, and the failure looks like a model quality problem — so teams tune prompts instead of fixing retrieval.
Planned retrieval: classify the query first — its intent, its time horizon, the memory types it needs — then search across typed stores, not one bag of chunks.
Stale facts
Yesterday's truth is still shaping today's decisions. Silently.
The pricing changed in June. The vector store has both versions — May's and June's — embedded side by side, equally retrievable. Sometimes the agent quotes the old price. Not always. Just often enough that sales stops pasting its answers into customer emails.
Most memory stacks can add, but not update. There's no supersession — no way to say 'this fact replaced that one, as of this date, for this reason' — so stale and current memory coexist, and retrieval picks by similarity, not by currency.
Inconsistency that looks like flakiness. Users experience the agent as unreliable in a way they can't predict, and unpredictable unreliability is worse than a system that fails loudly.
First-class supersession with lineage: facts carry validity intervals and replacement links, so retrieval resolves to the current version — and the trail back to the old one is preserved, not destroyed.
Workflow loss
The session ended. The agent now has to rediscover the world.
A research agent spends four hours mapping a problem space — dead ends ruled out, sources ranked, a working hypothesis forming. A deploy restarts the worker. The new session opens with a clean window and a summary that says 'research in progress.' The dead ends get re-explored. The hypothesis gets re-derived, differently.
The working state of a multi-step task — what's done, what's open, what was tried and rejected — lives in the context window, and the window dies with the session. Summaries capture conclusions, not state. Nothing checkpoints the work itself.
Interrupted work restarts instead of resuming. Long-horizon tasks — the whole point of agents — become the least reliable thing the system does.
Checkpointed working state: the task's live state captured as durable memory, so an interrupted agent resumes from where it was, with its open threads intact.
Context fragmentation
The state exists. It's scattered across five systems, and none of them know it.
The decision was made in chat. The constraint lives in a ticket. The data is in a dashboard screenshot someone pasted. The follow-up happened in email. The agent sees one of these. Each tool has its own slice of the truth, and the team glues the slices together by hand — pasting context into prompts, re-explaining the situation to every new session.
Context fragmentation isn't a retrieval problem; it's an architecture problem. Each surface keeps its own history, with its own format and its own scope, and memory built inside any one of them can only ever see that slice.
Teams rebuild state manually — the human becomes the memory layer, copying context between systems. The agent's answers are only as good as whichever slice it happened to see.
A memory layer that sits outside any single surface: events from every thread, tool, and session appended to one scoped record, so context is assembled across the whole history, not whichever fragment is nearby.
Which failure do you have?
The symptoms are usually visible long before the cause is. Match yours.
| If you're seeing… | It's probably |
|---|---|
| The agent re-asks settled questions after a compaction or a long session | Summary drift |
| It quotes last quarter's price, policy, or config — sometimes, unpredictably | Stale facts |
| It retrieves text about the right topic but the wrong problem | Retrieval mismatch |
| A restart or deploy means starting the task over | Workflow loss |
| The truth lives in five tools and the agent sees whichever one is nearby | Context fragmentation |
Notice what none of these are.
None of the five is a storage failure. The text was stored. The embeddings were indexed. The models were fine. Every one is a management failure — in how memory is written, updated, selected, and trusted over time. Research caught up to this a while ago: agent memory is a write–manage–read loop, and the field's open problems — contradiction handling, write-path filtering, trustworthy reflection, learned forgetting — all live in the manage stage.
Which is why "just add a vector store" keeps not working, and why the interesting question was never how much an agent can remember. It's which memory should be allowed to shape what it does next — and who decided.