Context Engineering for AI Agents: Database Memory
- Total tokens
- 100000 tokens
- You entered
- Max context window
- 8000 tokens
- You entered
Context engineering for AI agents with persistent memory requires a database when the agent’s context window cannot hold the full history, when memory must be shared across sessions, or when you need to query and filter past interactions by time, topic, or other attributes. This article explains why a database becomes necessary, how to design the memory layer, and the trade-offs involved.
Formula
The calculator estimates how many sessions an agent would need to process a given number of tokens if each session uses 75% of the context window for new tokens (the rest is reserved for system prompt, tools, and output).
Formula: Sessions = ceil( TotalTokens / (MaxContext * 0.75) )
- Total tokens – the total amount of text (input + output) you expect the agent to handle across its lifetime.
- Max context window – the maximum token limit of the model you are using (e.g., 8,000 for many models, 128,000 for others).
- 0.75 – a safety factor to leave room for system prompts, tool results, and the model’s response.
Worked Example
Suppose you have 100,000 tokens of conversation history and a model with an 8,000-token context window.
- Multiply the max context by 0.75: 8,000 × 0.75 = 6,000 tokens per session.
- Divide total tokens by that number: 100,000 ÷ 6,000 = 16.67.
- Round up to the nearest whole session: 17 sessions.
The calculator gives 17, matching the manual calculation.
Why a Database Becomes Necessary
Every AI agent has a context window – the number of tokens it can process at once. Once the conversation exceeds that limit, older information is lost unless you store it somewhere. A database provides:
- Persistence – memory survives restarts and new sessions.
- Scalability – store millions of tokens, far beyond any context window.
- Queryability – retrieve specific past facts, not just a raw dump of text.
- Filtering – search by date, topic, sentiment, or other metadata.
Without a database, you are limited to either truncating history (losing information) or re-sending everything (exceeding the context limit).
How Persistent Memory Works with a Database
The typical architecture is a retrieval-augmented generation (RAG) pattern. The agent’s memory is stored as text chunks, each with an embedding vector. When a new query arrives, the system:
- Converts the query into an embedding vector.
- Searches the database for chunks with similar vectors (using cosine similarity or other metrics).
- Retrieves the top-k most relevant chunks.
- Injects those chunks into the context window as background information.
This way, the agent only sees the most relevant parts of its history, not everything. You can also store structured facts (e.g., user preferences) in a relational table and query them directly.
Choosing the Right Database Type
Not all databases are equal for memory. Your choice depends on how you plan to retrieve memories.
- Vector database – best for semantic similarity search. Examples include Pinecone, Weaviate, and pgvector (a Postgres extension). Use when you need to find memories by meaning, not exact match.
- Relational database – best for structured facts like user settings, timestamps, and counts. Use when you need exact queries (e.g., “what did the user say on March 3?”).
- Key-value store – best for simple lookups by ID. Use when you only need to fetch a specific memory by key.
- Hybrid – many production systems combine a vector index for semantic search and a relational table for metadata filtering.
If you already have a Postgres database, pgvector is a low-friction way to add vector search without a new infrastructure component.
Common Problems and How to Avoid Them
Building a memory database is not just about storing text. Several pitfalls commonly appear:
- Context bloat – retrieving too many chunks fills the context window. Set a strict budget for how many tokens you inject.
- Stale memories – old facts may no longer be true. Include timestamps and let the agent weigh recency.
- Embedding drift – if you update your embedding model, old vectors become incompatible. Keep a version field and re-embed when needed.
- Duplicate memories – the same fact stored multiple times can confuse the agent. Use a deduplication step (e.g., hash of the text) before inserting.
- Privacy – storing personal data in a database requires encryption and access controls. Never store raw passwords or sensitive information without proper safeguards.
Cost and Performance Trade-offs
Adding a database introduces latency and cost. Each retrieval requires an embedding call and a database query. For a small agent handling a few hundred sessions, this overhead is negligible. For a large-scale system, consider:
- Embedding cost – every new memory chunk must be embedded, which costs API calls.
- Storage cost – vector indexes can be memory-hungry. Use quantization to reduce size.
- Query latency – a well-indexed vector search returns in milliseconds, but a full table scan will be slow. Use appropriate indexes.
- Cache hot memories – frequently accessed memories can be cached in memory to avoid repeated database hits.
Alternatives to a Full Database
If your agent’s memory needs are modest, a database may be overkill. Alternatives include:
- In-memory list – keep a Python list or JSON file of recent messages. Works for short-lived sessions.
- File-based storage – write memories to a text file or JSONL. Simple, but no querying beyond full read.
- Context window only – if your agent’s total lifetime is shorter than the context window, you may not need external memory at all.
- Summarization – periodically summarize older conversations into a compact summary that fits in the context window.
The choice depends on the scale and query needs. A database becomes necessary when you need to search across a large history or share memory between multiple agents.
Frequently Asked Questions
How do I decide between a vector database and a relational database?
Use a vector database when you need to find memories by meaning (e.g., “find the time the user mentioned their dog”). Use a relational database when you need exact facts (e.g., “what is the user’s preferred timezone?”). Many systems use both: relational for structured data, vector for free-form text.
Can I use a regular SQL database for vector search?
Yes, with extensions like pgvector for PostgreSQL. You can store vectors as columns and perform similarity search with SQL. This avoids adding a separate vector database service, but may be slower for very large collections.
How much context should I reserve for memory retrieval?
There is no fixed rule, but a common practice is to reserve 10–30% of the context window for retrieved memories. For an 8,000-token window, that is 800–2,400 tokens. The rest goes to the system prompt, tools, and the agent’s reasoning.
What is the difference between short-term and long-term memory for AI agents?
Short-term memory is the current context window – everything the agent sees in the current session. Long-term memory is stored externally and retrieved as needed. A database is typically used for long-term memory, while short-term memory is just the conversation history in the prompt.
How do I prevent the agent from retrieving outdated information?
Store a timestamp with each memory and filter out memories older than a certain threshold. You can also let the agent ask the user to confirm if a memory seems stale. In some cases, you may want to overwrite old memories with new ones on the same topic.
Is it safe to store user conversations in a database?
It depends on the sensitivity of the data and your compliance requirements. Always encrypt data at rest and in transit, limit access to the database, and consider anonymizing or deleting data after a retention period. Never store credentials or other secrets in plain text.
Related
Formula used
in t | Total tokens | tokens | 100000 | 0 in m | Max context window | tokens | 8000 | 0 out sessions | Sessions needed | sessions | ceil(t / (m * 0.75)) | 0
Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.