Focused AI routing study
Agent Routing Lab
Two AI routes, one preserved trace—and an evaluation error you can inspect.
Your first visit
Read the evaluation review below, then inspect the original answers and scores in the demo.
Review note / 10 September 2026
The evaluator missed an error.
The recorded evaluator preferred Route B and awarded it 5/5 for groundedness. But B claims that system-message or memory features do not count toward token limits. Moving text into a system message does not remove its tokens; memory costs depend on what is retrieved into context.
The conclusion: the score records a model’s preference, not a verified quality win. The supplied context also does not establish B’s claim that the full conversation history is sent on every call. Both require scrutiny before accepting the diagnosis.
The original answers, scores, costs, and timings remain intact. This study demonstrates traceability and an evaluator failure; it does not establish a superior route.
What a stronger evaluation would test
Check each factual claim against the supplied evidence and provider documentation. Add repeated cases and an independent review rubric. Measure context-related slowdown separately, holding the model and other settings fixed while changing context size. Those are next experiments, not results from this capture.
01 / The idea
AI decisions you can inspect.
An agent result alone cannot explain why a route was chosen. Builders need to see the context, latency, cost, quality judgment, and what happened when a request failed.
02 / The approach
The choices behind it.
Give both routes the same bounded MCP context. Record the stages, export the evidence, and make the public version key-free so visitors can inspect it without spending tokens.
- Get controlled context
- Compare two routes
- Evaluate the answers
- Inspect the trace
03 / Where it stands
What works today.
A working reference implementation with a preserved API run, recorded failure recovery, and documented tests. The follow-up review identifies a factual error missed by its evaluator.
The original model scores are preserved as evidence; they do not establish a quality winner or a model benchmark.
Another side of the work
LedgerLens
Explore the next project ↗