Memory in AI Systems
Why an AI product that "remembers" the user is solving a harder problem than it looks like from the outside.
6.7.1Definition
Memory in an AI system is the mechanism by which information from past interactions persists and is retrieved for future ones — distinct from a single conversation's context window (6.2), which resets. It typically works by extracting durable facts from past sessions, storing them (often in a database, 5.1, or vector store, 6.4), and selectively re-injecting relevant ones into future context.
6.7.2Why It Exists
A model has no memory beyond its current context window by default — every new conversation starts from zero unless the surrounding system deliberately carries information forward. Memory systems exist to make an AI product feel continuous across sessions — recalling a user's preferences or past requests — without simply feeding an ever-growing, unbounded transcript into every future context.
6.7.3Design Considerations
- What to remember — durable facts and preferences, not the full transcript; most conversational detail is irrelevant to future sessions and should be discarded rather than stored.
- Selective retrieval — only relevant stored memories are pulled into context for a given request (6.2), not the user's entire history at once.
- User control & correction — the ability to view, correct, or delete what's been remembered, since automatically inferred memory can be wrong or become outdated.
6.7.4Common Mistakes
- Storing every detail of every conversation rather than distilling durable facts, producing a bloated memory store that's expensive to search and mostly irrelevant.
- No user visibility or control over stored memory, leaving people unable to correct a wrong inference or remove something they didn't intend to be remembered.
- Re-injecting stale or contradicted memories without an update mechanism, so an outdated preference keeps resurfacing after the user has explicitly changed it.
- Surfacing sensitive remembered details in contexts where the user hasn't raised them, which can feel invasive rather than helpful.
6.7.5Best Practices
- Extract and store distilled, durable facts rather than raw transcripts.
- Retrieve only what's relevant to the current request, not the user's full memory store indiscriminately.
- Give users a way to view and correct what the system has inferred about them.