AI Agent Memory: What to Keep and What to Forget

Users expect an assistant to remember what they said a few messages ago, and often what they told it last week: their role, their preferences, the account they usually work on. Without memory, every conversation starts from nothing and users repeat themselves. With careless memory, an agent fills its context with stale details, slows down, costs more and may surface information it should have forgotten. AI agent memory is the design of what an agent keeps, for how long, where and how it uses it. This guide explains the main types of memory, how to manage conversation history, when to store long-term facts and how to keep users in control of their data.

Two scopes of AI agent memory

The LangGraph documentation describes memory as a system that remembers information about previous interactions, and divides it into two types by recall scope (LangGraph: Memory overview):

• Short-term memory, or thread-scoped memory, tracks the ongoing conversation by keeping the message history within a session
• Long-term memory retains information across different conversations or sessions, stored in custom namespaces rather than tied to one thread

The terms are from one framework, but the distinction applies to any agent design. Short-term memory is about keeping a single conversation coherent. Long-term memory is about carrying useful knowledge from one conversation to the next.

Manage conversation history carefully

The simplest short-term memory is to send the full message history with every request. That breaks down as conversations grow. The same documentation explains that a full history may not fit in a model's context window, causing an error, and that even when it fits, most models still perform poorly over long contexts: they get distracted by stale or off-topic content while responses become slower and more expensive.

Common techniques to keep history manageable:

• Keep only the most recent messages, plus the original instructions
• Summarise older parts of the conversation into a short note that replaces them
• Remove tool outputs once their key facts have been captured, since raw results can be long
• Start a new thread when the user moves to a clearly different task

Test these choices with real conversations. Trimming too aggressively makes the agent forget what the user just said; trimming too little wastes cost and attention.

Decide what long-term memory is for

Long-term memory is useful when information genuinely helps future conversations. The LangGraph documentation borrows three categories from human memory: semantic memory for facts, episodic memory for experiences and procedural memory for rules. In a business assistant, these might map to:

• Facts: the user's role, team, preferred language or the accounts they manage
• Experiences: examples of past tasks completed well, which can guide similar tasks
• Rules: instructions refined from feedback, such as a preferred report format

Each type has a different shape and update pattern. Facts are often a small profile that is updated in place. Experiences are a growing collection that needs search. Rules are part of the agent's instructions and should change only through review.

Choose when to write memories

The documentation describes two approaches. Writing memories in the hot path means the agent decides what to remember during the conversation, before responding; it notes this can affect latency and make the agent juggle memory creation with its main task. Writing memories in the background means a separate process reviews conversations and creates memories asynchronously.

For most business assistants, a background process is easier to control: it can apply consistent rules about what to store, filter sensitive data and avoid slowing down replies. Hot-path writes suit cases where the user explicitly asks the assistant to remember something.

Retrieve only what is relevant

Stored memories are only useful if the right ones reach the model at the right time. Retrieve a small number of relevant memories for each request, using the user's identity and the topic of the conversation, rather than loading everything. The same retrieval principles used for documents apply; our guide to private RAG covers how to search company knowledge safely, and many of its controls apply to memories too.

Worked example: a hypothetical memory policy

The table below is a hypothetical memory policy for an internal sales assistant. It illustrates the decisions involved, not a recommended configuration.

InformationMemory typeStored whereRetentionUser controlCurrent conversationShort-termThread state, trimmed and summarisedUntil thread closesCan clear conversationUser role and teamLong-term factUser profileWhile employedVisible and editablePreferred report formatLong-term ruleUser profileUntil changedVisible and editablePast deal notesNot stored as memoryRead from CRM when neededCRM rules applyThrough CRM permissionsCustomer personal detailsNot stored as memoryRead from CRM when neededCRM rules applyThrough CRM permissions

The key choice in this example is to leave business records in the systems that own them and retrieve them when needed, rather than copying them into agent memory.

Protect users and their data

Memory creates a new store of information about people, which brings obligations:

• Tell users what the assistant remembers and let them view, correct and delete it
• Do not store sensitive personal data unless there is a clear need and a legal basis
• Separate memories strictly by user and by organisation, and check access on every read
• Apply retention periods, and delete memories when a user leaves or asks
• Treat stored memories as untrusted input when they are fed back to the model, because content written from conversations could contain injected instructions

Test memory behaviour

Memory bugs are subtle. An agent may recall a preference from the wrong user, keep using a fact that the user corrected or forget an instruction after a summary step. Add test cases for these situations to your evaluation set: a correction that must replace an old fact, a request to forget something, two users with different preferences and a long conversation that crosses the summarisation point. Run them whenever you change how memories are written, stored or retrieved.

Keep memory consistent across agents

When several agents work together, decide which one owns each type of memory and how others read it. Shared memory without clear ownership leads to conflicting facts. Our guide to multi-agent workflows discusses how to divide responsibilities between agents, which includes deciding who can read and write shared state.

Frequently Asked Questions

What is AI agent memory?

It is how an agent keeps information from past interactions: short-term memory for the current conversation, and long-term memory for facts, examples or rules that help across conversations.

Why not send the whole conversation every time?

Long histories can exceed the context window, and even when they fit, models tend to perform worse over long contexts while becoming slower and more costly.

What should an agent remember long term?

Information that genuinely helps future tasks, such as role, preferences and refined instructions. Leave business records in the systems that own them and retrieve them when needed.

How do users stay in control of agent memory?

Let them view, correct and delete what is stored, apply retention periods and separate memories strictly by user and organisation.

Design memory that helps, not hoards

We design memory, retrieval and data controls for AI agents that work with business information. See our AI agent orchestration and RAG service, or talk to us about AI agent memory.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.