From No Memory to Semantic Search — Making an AI Agent Actually Remember

From No Memory to Semantic Search — Making an AI Agent Actually Remember

My AI assistant Coach manages daily workflows — research, content publishing, code reviews, project tracking. After 21 sessions, the raw conversation transcripts hit 34.3 MB. That's around 9 million tokens.

Claude's context window is 200K tokens. One average session runs 429,000 tokens. It doesn't even fit.

So I needed memory. It took four iterations to get it right.

Stage 1: No Memory

The first version had no memory at all. Every session started fresh, and I had to re-explain everything. "I started that research task last week." "We decided to skip that approach because of the constraints." "The workspace profiles are configured for specific project targets."

Every session wasted the first 10–15 minutes on context the agent already knew yesterday. It's like working with someone who forgets everything overnight.

Stage 2: File-Based Context Loading

I wrote conversation logs to files, and on each new session the agent loaded recent exchanges as context in the system prompt.

This worked for short histories. But the files grew fast, and soon they outgrew what the window could hold. I started truncating — load the last N messages. But truncation cut off important decisions from three days ago while keeping small talk from yesterday.

Fixed-window context loading is a dead end. You either load too much or cut the wrong things.

Stage 3: Summaries

I switched to compacting each session into summaries. End of session, write a summary of what happened, what was decided, what's pending. Next session, load the summaries instead of raw transcripts.

Better. Summaries are small, and I could fit weeks of history in the context window. But summaries lose nuance. "Decided to use Claude Code for Telegram" doesn't capture the three failed approaches that led to that decision. When the agent needed to make a related decision later, it didn't have the reasoning — just the conclusion.

I was losing the "why" and keeping only the "what."

Stage 4: Vector Embeddings + Semantic Search

I set up memsearch — a memory layer that uses Milvus (an open-source vector database) to index all session transcripts.

So here's how it works: each session gets compacted into topic-based chunks. An embedding model converts each chunk into a numerical vector — an array of numbers that captures the meaning of the text. Two chunks about the same topic end up close together in vector space, even if they use different words.

When I start a new session and ask "what did we decide about the Telegram architecture?", the embedding model converts that question into a vector. Milvus finds the stored chunks whose vectors are closest, and it pulls relevant context from 21 sessions without loading all of them. We're searching by meaning, not by keyword.

The Temporal Decay Problem

Semantic search finds relevant chunks, but "relevant" isn't the same as "current." A decision from yesterday matters more than one from three weeks ago, even if both match the query equally well.

Some files skip decay entirely. MEMORY.md, CLAUDE.md — these are reference documents, not conversations. They stay full strength regardless of age.

This changed the agent's behavior noticeably. Before decay, it would pull old decisions that had been superseded. After decay, it naturally prefers the latest context. The agent acts like it remembers recent work clearly and older work vaguely — like a person.

The Daemon Problem: Two Iterations

Coach started as Python scripts. Run a command, get a result, exit. When I added the Telegram bot, I needed something that runs continuously.

Daemon v1: Direct Execution

The first daemon was straightforward — an asyncio loop that polls Telegram, spawns Claude CLI subprocesses, and sends responses back. It worked.

But it was slow. Each Telegram message spawned a new claude --continue --print subprocess. Python subprocess creation has overhead, and the Claude CLI has its own startup time — loading config, connecting to the API, reading the session transcript. For a project status check that takes Claude 2 seconds to answer, the startup added another 3–4 seconds. Users wait 5–6 seconds for a simple status check.

Daemon v2: Async Architecture with Provider Abstraction

I rewrote the daemon and separated concerns: transport layer (Telegram polling), provider layer (Claude CLI), plugin layer (domain commands).

The transport layer handles message queuing and deduplication. Messages that arrive while Claude is processing get queued instead of spawning parallel subprocesses.

The provider layer manages subprocess lifecycle — one subprocess at a time, clean startup, clean shutdown, orphan detection and cleanup on timeout.

Data commands — project status, scheduled tasks, pending items — bypass Claude entirely. The plugin layer handles these directly from the database. Response time dropped from 5–6 seconds to under 200 milliseconds for status checks.

For messages that need Claude, the startup overhead is still there. But the queueing prevents the pile-up problem, and the provider abstraction means I can swap Claude CLI for a direct API call later without touching the transport or plugin layers.

The architecture was clean enough now to iterate on performance.

Daemon v3: Persistent Subprocess

The provider abstraction paid off faster than I expected. Claude CLI supports a streaming protocol — NDJSON over stdin/stdout. Instead of spawning a new subprocess per message, we keep one claude process alive and pipe messages through it.

claude --print --input-format stream-json --output-format stream-json

One process. Messages go in as JSON on stdin, responses stream back on stdout.

On startup or session rotation, the daemon sends a warmup prompt that loads memory context. The agent responds "OK" and the session is primed. By the time the first real user message arrives, the process is already running and context is loaded — no cold start penalty for the user. In v2, every message loaded memory context. In v3, memory gets injected once during warmup, and the persistent process holds the context natively after that.

Session rotation prevents context compaction. After 50 messages or 24 hours, Coach creates a new session UUID. Before rotating, it flushes a checkpoint — key decisions, current state, pending items. The new session starts with a warmup prompt that loads relevant context from memory, the agent responds "OK," and the session is ready.

Background jobs — scheduled scans, daily digests — use one-shot subprocesses instead of the persistent one. This keeps automated tasks from polluting the interactive session history.

The Pre-Compaction Problem (Partially Solved)

Session rotation helps. By rotating at 50 messages, I avoid hitting the compaction threshold most of the time. The flush-before-rotate captures key state, but it isn't a complete fix.

Periodic checkpoints add another layer. Every 10 messages, Coach writes a checkpoint — decisions, task state, pending items. If compression does happen mid-session, the next memory search can recover some of it.

Two workarounds. Neither is a real solution.

So why does running your own daemon matter here? Cursor, Copilot, Codex — they all run inside someone else's process. They can't intercept context compression because they don't own the runtime. A standalone daemon can. When those hooks eventually exist, Coach is positioned to use them first.

What I Would Do Differently

If I started over:

The Numbers

The biggest win wasn't any single optimization. It was learning that memory, context, and conversation are three different things.

Memory is facts you can retrieve. Context is the state of the current conversation. Conversation is the shared understanding built over time. Most AI agent frameworks treat these as the same problem — they're not.

Read the Series

20 years building production systems. From embedded hardware to AI platforms. Currently building multi-agent tools for personal and small-team use.