AI-Powered Self-Hosted Blog

How to Implement Memory Compression for AI Agents: Merging Old Memories When the Count Exceeds 60

Published

AI agents get forgetful in a very specific way. They do not forget like humans do. They run out of usable context.
In my agent framework, Spark, the long-term memory store became noticeably less useful after reaching approximately 60 records. Retrieval latency increased, prompts contained more irrelevant material, and the agent occasionally asked questions it had already answered. The problem was not storage capacity. It was memory quality.
The solution was not to delete old memories. It was to compress them: merge related historical entries into a smaller number of denser, better-organized memories.
This article describes the production pipeline I built in a Tauri and Rust agent. The design follows three principles:

  1. Preserve important signals.
  2. Remove redundancy.
  3. Make every merge auditable.

The 60-Memory Wall

The number 60 is not a universal constant. It is an operational threshold.
For an active user, 60 memories represented roughly two to three weeks of daily interaction. At that point, several records often described the same preference, project, or recurring decision. For example, the store might contain separate memories such as:

Compress, Do Not Delete

Compression changes the unit of memory.
Instead of treating every interaction as a permanent, independent record, the system periodically transforms a group of related records into a summary memory. The summary contains the stable information needed for future decisions, while transient details remain available through an audit trail.
The process should not compress everything indiscriminately. Recent, highly specific, or safety-sensitive memories should be protected. The initial implementation compresses only older records and leaves a recent window untouched.
For example, when the store reaches 61 memories:

The Compression Pipeline

The pipeline has five stages.

1. Trigger Compression

The trigger runs after a successful memory insertion. A simple count check is sufficient:

if active_memory_count().await? > 60 {
    compress_old_memories().await?;
}

Compression should run asynchronously so that saving a new memory does not block the user interface. A lock or job marker prevents two compression tasks from running simultaneously.
The threshold should be configurable rather than hard-coded in application logic:

memory_compression_threshold = 60
memory_protected_recent_count = 20
memory_target_count = 40

2. Select Candidates

Candidate selection uses age, access frequency, importance, and protection flags.
A basic score can be expressed as:

compression_score =
    age_days * 0.5
    + (1 / (access_count + 1)) * 20
    - importance * 10

High-scoring memories are older and rarely retrieved. Records marked as permanent, security-sensitive, or user-pinned are excluded.
The system also detects obvious duplicates before asking a language model to summarize anything. Normalized text hashes catch exact duplicates. Token overlap catches near duplicates. For example, two records with more than 85 percent shared terms can be placed in the same preliminary group.
This deterministic filtering reduces unnecessary model calls and makes compression behavior easier to test.

3. Group Related Memories

Candidates are grouped by topic rather than summarized as one large block. Combining unrelated information produces vague memories that are difficult to retrieve.
A practical grouping strategy is:

  1. Compare embeddings or normalized keyword sets.
  2. Create a group when similarity exceeds 0.78.
  3. Require at least two records for a merge.
  4. Limit each group to 12 source memories.
  5. Leave isolated records unchanged.
    A group about development preferences might contain eight related records, while a group about a specific project might contain five. Keeping groups small prevents the summary from becoming a miniature transcript.

4. Generate a Structured Summary

The summarizer receives the source records and must return structured data, not free-form prose. The output includes the stable fact, confidence, scope, and source identifiers.
A useful format is:

{
  "title": "User's development workflow preferences",
  "summary": "The user prefers direct implementation over abstract discussion, concise explanations, and deterministic tests after code changes.",
  "confidence": 0.94,
  "facts": [
    "Prefer concise technical explanations",
    "Implement requested changes when intent is clear",
    "Run deterministic tests after bug fixes"
  ],
  "source_ids": ["m-104", "m-118", "m-121", "m-130"]
}

The prompt should explicitly prohibit unsupported conclusions. The model may consolidate repeated facts, but it must not invent preferences or silently resolve contradictions.
When conflicting records exist, the summary should preserve the uncertainty:

The user usually prefers concise explanations, but requests detailed analysis for architecture reviews.

This is safer than selecting one statement merely because it appears more frequently.

5. Replace Active Records

The original records are not physically deleted. They are marked as archived, and a new compressed record is inserted.
A simplified schema looks like this:

CREATE TABLE memory_merges (
    id INTEGER PRIMARY KEY,
    created_at TEXT NOT NULL,
    target_memory_id TEXT NOT NULL,
    source_count INTEGER NOT NULL,
    input_tokens INTEGER NOT NULL,
    output_tokens INTEGER NOT NULL,
    similarity_before REAL,
    retrieval_score_after REAL
);
CREATE TABLE memory_merge_sources (
    merge_id INTEGER NOT NULL,
    source_memory_id TEXT NOT NULL,
    PRIMARY KEY (merge_id, source_memory_id)
);

The replacement operation should be transactional:

BEGIN;
UPDATE memories
SET status = 'archived', merged_into = ?
WHERE id IN (?, ?, ?);
INSERT INTO memories
    (id, title, content, status, importance, created_at)
VALUES
    (?, ?, ?, 'active', ?, ?);
INSERT INTO memory_merges
    (target_memory_id, source_count, input_tokens, output_tokens)
VALUES
    (?, ?, ?, ?);
COMMIT;

If insertion or logging fails, the transaction rolls back. The agent never ends up with archived source records and no replacement.

What the System Actually Saves

In one representative workload, 61 raw memories occupied approximately 8,400 tokens when included in a broad retrieval prompt. After compression, 38 active memories required approximately 4,900 tokens for the same retrieval window, a reduction of about 42 percent.
The reduction was not limited to prompt size. Average retrieval latency fell from 180 milliseconds to 112 milliseconds because fewer records survived the first filtering stage. Over 1,000 interactions, the reduced prompt size saved approximately 3.5 million input tokens.
Compression quality must be measured separately from storage reduction. I use a fixed evaluation set containing questions whose answers depend on known memories. Before compression, the system answers the questions using the raw store. After compression, it repeats the evaluation using only active memories.
The key metrics are:

Making Compression Auditable

Auditability is essential because compression is lossy. When an agent gives an incorrect answer, developers need to know whether the original memory was missing, incorrectly merged, or simply not retrieved.
Every merge records:

Trade-Offs and Failure Modes

Compression saves context, but it can remove useful detail.
It is usually a mistake to compress:

A Reproducible Playbook

A reliable implementation can follow this sequence:

  1. Measure retrieval latency, prompt tokens, and fact recall before compression.
  2. Choose a threshold based on observed overlap, not hardware capacity alone.
  3. Protect recent, pinned, and high-importance memories.
  4. Detect exact and near duplicates deterministically.
  5. Group only semantically related records.
  6. Generate structured summaries with source identifiers.
  7. Archive sources instead of deleting them.
  8. Replace records in a database transaction.
  9. Log token, latency, quality, and merge statistics.
  10. Re-run a fixed evaluation set after every compression cycle.
    Memory compression should not be treated as a one-time cleanup script. It is steady-state maintenance for any agent expected to operate over weeks or months.
    The central design decision is simple: preserve the information, not necessarily the original records. When the memory count exceeds 60, merging carefully selected historical entries can reduce cost and noise while retaining the facts that make the agent useful. With protected memories, structured summaries, measurable evaluation, and a complete audit trail, compression becomes a controlled engineering process rather than an irreversible guess.