The Orchestrator That Ate Itself
I was building a multi-agent system and decided to put an LLM in charge of routing. The classifier — I called it a neural command processor — would look at each request and decide which agent should handle it. I set confidence thresholds at 0.90, added keyword detection for multi-agent requests, and put in some escalation logic. Looked good to me.
Then I ran it. The same message — “Check the auth module” — got routed to three different agents across three separate runs. First it was code review intent, then security check intent, then implementation intent. Same input every time. The temperature, context window position, even the system load affected the result. The classifier just wasn’t reliable.
So I made it worse. I added auto-route — speculative execution for free-text messages. The LLM would classify a message, build a DAG, and dispatch to agents. Those agents would produce output. Then the router would look at that output and decide it needed coordination too. So it built a DAG from the DAG. Recursive coordination. The system kept burning tokens to plan plans of plans.
Auto-route created a feedback loop: the LLM planned a DAG, agents executed it, and the router classified that output as needing further coordination — triggering another plan. The system was burning tokens planning plans of plans. A speculative execution feature with no exit condition is an infinite regress waiting to happen.
I shipped auto-route with enabled: false and left a comment: “prior misclassification issues.” Then I deleted the classifier — 211 lines gone. The neural command processor went too — another 1,040 lines. All dead code.
The system still worked, but only for workflow execution and command-based plugin calls. The routing decision was already made before the LLM ever saw the request. Free-text multi-agent coordination was gone.
The Core Problem
If you put an LLM at the routing decision point, you get a non-deterministic dispatch layer. Prompt tuning doesn’t fix that. The problem isn’t the prompt — it’s that the LLM is making the decision at all.
The LLM routing classifier wasn’t misconfigured — it was the wrong tool for the job. Temperature, context window position, and system load all affect classification. Prompt tuning treats symptoms; removing the decision point treats the cause.
So how do you get multi-agent coordination for free-text requests without an LLM in the routing loop?
The Mechanism: Intercept, Record, Replay
The answer I landed on: intercept the LLM’s own tool calls and learn from them.
The mechanism has three distinct phases: Intercept (capture what the LLM does naturally), Record (build a DAG + embed the goal), Replay (match by semantic similarity, execute deterministically). The LLM only participates in phase one — once per pattern.
1. Intercept During Streaming
When the LLM gets a free-text request through MCP (Model Context Protocol, the standard for tool-calling), it figures out which tools to call on its own. Say I send “Resume training on the equip decks.” The LLM scans the available MCP tools, finds the right agent, calls it. Done.
The key part: while that response is streaming, a session worker watches every MCP tool call. When the LLM calls agents — say, agent_invoke(trainer, "load equip deck") followed by agent_invoke(trainer, "next card") — the worker captures that whole sequence and sends it to a background process.
The LLM is already doing the planning work. We just record what it did.
2. Build a DAG and Embed the Goal
The background process — I called it a FeedbackWorker — takes that captured sequence and does two things:
- Builds a DAG (directed acyclic graph) from the invocation chain — preserving the dependency order
- Embeds the user’s original goal as a 384-dimensional vector using all-MiniLM-L6-v2, a lightweight local model that runs in about 5ms on CPU
The DAG and its embedding go into a local registry. The embedding captures what the user meant, not the exact words. “Resume training” and “continue my equip session” and “let’s do more equip cards” all land close to each other in vector space.
3. Match and Replay
The next time a request comes in, we embed the new goal and compute cosine similarity against all the cached plan embeddings. Score of 0.75 or above? We replay the cached DAG directly.
No LLM call. No token cost. Same agents, same dependency order, same result every time.
First time: User goal → LLM discovers tools → agents execute → success
↓
Session worker captures tool sequence
↓
FeedbackWorker builds DAG + embeds goal → cached plan
Second time: User goal → embed → cosine match (0.87) → replay cached DAG
Zero LLM tokens. Deterministic.The LLM taught the system what to do. The system remembers. The LLM never has to teach it again.
Why Semantic Matching, Not Keywords
My first version used verb-set matching. Extract verbs from the goal (“review”, “test”), match against cached plans by verb set. Works fine for exact repeats, but breaks when someone says the same thing differently:
"review and test the auth module" → verbs: [review, test] → match
"review and test the payment service" → verbs: [review, test] → match
"examine and validate the codebase" → verbs: [examine, validate] → MISS“Examine” and “validate” aren’t in the verb dictionary. The user said the same thing using different words, and the system missed it.
With embeddings, “examine” and “review” land close together in vector space. Cosine similarity between “review and test the auth module” and “examine and validate the codebase” comes out around 0.86 — well above the 0.75 threshold. Match. Replay.
I kept verb-set matching as a fallback for plans cached before embeddings were added. But semantic matching is what handles real usage, where people don’t stick to a fixed vocabulary.
Self-Correction
Cached plans go stale. Agents change, requirements change.
When a replayed plan fails, it increments a failure counter. Three consecutive failures and the plan gets marked stale — skipped permanently. The next matching request falls through to the LLM, which builds a fresh plan. If that succeeds, it gets cached as the new pattern.
Good plans get used more and stay. Bad plans get marked stale and disappear. No manual cleanup needed.
Plans don’t need manual expiration. Three consecutive failures → marked stale → skipped permanently. The LLM generates a replacement on the next match, which gets cached if it succeeds. The registry self-heals without operator intervention.
The Numbers
After a few weeks running this, the pattern is clear:
LLM cost for repeatable tasks converges toward zero.
What Changed
The system went from fixed workflows and plugin commands back to handling free-text requests — but this time without an LLM making the routing decisions. It got back what was lost when I removed the classifier.
No classifier. No routing LLM. No recursive coordination loops. The system watches the LLM handle new requests, learns from what it does, and replays what worked for anything it has seen before.
The planning work happens once. Everything after that is replay.
Free-text multi-agent coordination — without an LLM in the routing loop. The classifier removal was a loss. The intercept-record-replay mechanism turned it into a gain: same capability, deterministic execution, converging token cost.
This is part of a series about building a multi-agent AI system. Previous posts: Deterministic Routing Beats LLM Routing, Five Stages to One Brain.
