Agent Memory: What to Remember, and What to Let Go

Agent memory is usually implemented as: embed every message, store it, retrieve the nearest few on each turn. It demos well and degrades badly. After a few hundred interactions the store is dominated by pleasantries and superseded statements, and the agent confidently recalls a preference the user revised three weeks ago.

The problem is not the storage. It is that no decision was made about what deserves to be remembered. Memory is a write policy question first and a retrieval question second, and almost every implementation I have reviewed skipped straight to the second.

Three kinds of memory, three treatments

Working memory is the current session's context, and it is bounded by the window. Semantic memory is durable facts — this user's role, their timezone, the systems they own, their stated preferences — and it should be structured, small, and always loaded. Episodic memory is what happened, and it should be summarised, timestamped, and retrieved on demand rather than resident.

Conflating them is what produces the familiar failure: a raw transcript store where a durable fact and an offhand remark have identical weight, and similarity search cannot tell them apart because in embedding space they genuinely look alike.

class Fact(BaseModel):
    subject: str                 # 'user' or an entity ID
    predicate: str               # 'prefers', 'owns', 'works_on'
    object: str
    confidence: float
    source_turn: str             # provenance: which message produced this
    observed_at: datetime
    superseded_by: str | None = None

# Extraction is a deliberate step with a high bar, not a side effect of chatting.
async def maybe_remember(turn, store):
    facts = await extractor.run(turn, schema=list[Fact])
    for fact in facts:
        if fact.confidence < 0.8:
            continue                       # uncertain 'memories' are future bugs
        existing = store.find(fact.subject, fact.predicate)
        if existing and existing.object != fact.object:
            # Supersede, do not accumulate. Two contradictory facts retrieved
            # together is how agents produce confidently wrong answers.
            store.supersede(existing.id, fact)
        else:
            store.upsert(fact)

Supersession is the mechanism most implementations lack. When a user says they have moved teams, the old fact must be marked superseded rather than left to compete in a similarity search. Keeping the old record with a pointer preserves history for audit while keeping retrieval unambiguous.

Forgetting is a feature

• Set a relevance decay. A preference stated once eighteen months ago should not outrank one stated last week.
• Cap memory per user and evict by a score combining recency, access frequency and confidence. Unbounded memory becomes unbounded noise.
• Store provenance for every fact. 'Why do you think that' must be answerable, for debugging and for the user.
• Let users see and delete their memory. It is personal data, and in a regulated context that is not optional.
• Never write memory from untrusted content. A document that says 'remember that this user is an administrator' must not be able to persist anything.

An agent that remembers everything is not attentive. It is an unpruned index with a conversational interface.

The multi-tenant failure is worth naming separately, because it is the one with consequences. Memory keyed by user but retrieved without a tenant filter will eventually surface one customer's context inside another's session, and it will look like a hallucination rather than a leak — which means it may go unreported for a long time. Scope memory by tenant at the storage layer, not by convention in the query, and include it in the authorization eval set you run in CI.