How it actually
compares — with numbers.

LTP calls the LLM once to compile the plan, not once per step, so its cost stays flat (~2.5k tokens) as the task deepens — as long as the added depth is deterministic execution (loops, code, tool calls), not extra LLM steps or a re-compile. ReAct re-sends the system prompt and tool schemas every turn, so its cost scales with turns. Hybrid is the only mode at 100% — but it is not the cheapest: pure LTP is. Hybrid spends a little more to convert LTP's fragile tail into reliability.

Hybrid is 100% at every tier — the only mode that is. The model shapes the rest: a weak model makes ReAct loop (9.3k tokens), so LTP's token edge is largest; a mid model (Sonnet) lifts everyone to 100% — the sweet spot; a strong model (Opus) answers in ~3 turns "in its head", which is cheaper but drops ReAct to 89.3% on over-confidence (it skipped tools and miscomputed). Hybrid absorbs both failure modes.

Two honest findings. (1) Cost is the robust win: LTP routes the same tools at roughly half ReAct's tokens on every model. (2) Hybrid is the most reliable mode at every tier (100% / 85.7% / 78.6%), because its fallback rescues each pure mode's distinct misses — LTP can't finish working-mem, ReAct short-circuits file-words, and hybrid takes whichever one works. Six of seven tasks are now solid across all models. The decline on stronger models comes down to one stubborn task: file-words, where Sonnet and Opus return the raw file object instead of counting even when told to reply with only the number — the same over-confidence that drops their ReAct math score. We keep it in rather than quietly drop it.

Per-task receipts (avg tokens over 2 reps; ✗ = both reps failed, ½ = one of two):

But raw tokens aren't the bill — cost is, and cost flips by model. Each cell is $/task / tokens:

On Sonnet — a production model — kernelmcp LTP/Hybrid is ~3× cheaper than LangGraph and faster (2.2s vs 4.2s), despite sending more raw tokens. The reason is prompt caching: kernelmcp's large static prefix exceeds the cache minimum and is billed at ~10% on a hit (effective ~$0.78/M vs LangGraph's ~$3.9/M); LangGraph's minimal prompt is too small to cache, so it pays full price. On Haiku nothing caches below its larger floor, so there the lean agent stays ~17% cheaper.

Fairness, openly: this is a bare LangGraph agent vs kernelmcp's full loop on arithmetic; LangGraph's tool is a plain exec vs our security sandbox; and the integrated workflows kernelmcp is built for (memory, 120+ tools, planning, scheduling) aren't portable to a stock agent, so they're not in this comparison. Pitch: cost-at-scale, latency, and the suite — not raw token count.

Two honest reads. Reliability: CrewAI and kernelmcp Hybrid are the only systems at 100% on both models; LangGraph is close (92.9%); pure LTP is cheapest but unreliable on stateful chains. kernelmcp does not dominate reliability — CrewAI matches it. Cost: on Sonnet, kernelmcp Hybrid is the cheapest 100%-reliable system at $0.0075 — ~2.4× under LangGraph ($0.0181) and ~3× under CrewAI ($0.0231) (prompt caching). On Haiku (no caching) Hybrid is pricier than the lean agents. So the kernel's edge is real but conditional: cheapest-at-100% on a caching-capable production model, not on the cheapest tier.

Integrity note: CrewAI's built-in token counter undercounts on its litellm path (2,182 reported vs 8,502 actual on a Sonnet task), so its cost here is measured by intercepting the Anthropic SDK directly — every real API call counted. Caveats: 7 memory tasks only (file tasks excluded — tool names differ across the standalone servers); LangGraph/CrewAI latency includes a fresh memorymcp subprocess spawned per run, so latency isn't comparable.

Depth tips the balance. On deep chains, kernelmcp Hybrid is the cheapest 100%-reliable system on both models — 1.4× under LangGraph and 2.1× under CrewAI on Haiku, widening to 2.6× / 3.1× on Sonnet — and faster (Sonnet 9.5s vs ~30s). The reason is structural: LTP compiles the whole chain in one call (~2.4k tokens) then executes it deterministically — so the LLM is called once, not once per step, and cost stays flat as long as the added depth is deterministic execution rather than extra LLM steps or a re-compile. LangGraph and CrewAI re-invoke the LLM at every step, so their cost grows with chain length. This is the opposite of the single-tool result — the compile-once design pays off exactly as agentic depth grows.

Honest qualifier: LTP alone is unreliable on stateful chains (80% / 60% — it stumbles on a working-memory task), so this is a win for Hybrid (LTP-first + verified fallback), not LTP by itself. Reliability is a four-way tie at 100%; kernelmcp wins on cost at equal reliability, not on reliability. Small sample (5 tasks).

The core retriever is at exact parity. Force one chunk per document on both sides — chunking removed as a variable — and every metric is identical (recall@1 0.875, MRR 0.93): same embedder, same cosine, same ranking. ragmcp's default-config edge comes entirely from its finer default chunk (500 chars vs LlamaIndex's 1024 tokens), not a smarter algorithm. ragmcp ingests ~1.7× faster in bulk; LlamaIndex answers ~1.3× faster per query. Verdict: competitive on retrieval, opposite performance tradeoffs — not a knockout.

Recall is close and split: memorymcp ranks the target higher (recall@1, MRR), Mem0 catches a couple more in the tail (recall@5). The clear, defensible gap is cost: memorymcp's deterministic PatternFactExtractor ingests free and fast, where Mem0's intended LLM extraction costs ~$0.009/fact. Mem0 answers ~3.5× faster per query (memorymcp pays for a richer pipeline + persistent store).

Caveat stated against us: this dataset is pure paraphrase with low lexical overlap, which under-uses Mem0's BM25 keyword layer — on keyword-heavy queries Mem0 would close the gap. It's a semantic-retrieval result, not a universal one.

The process backend (socket shim + OS limits) stops egress and resource attacks, but has no kernel isolation — host filesystem reads/writes leak. The Docker backend (network_mode=none, dropped capabilities, read-only rootfs, Docker’s default seccomp, memory & PID caps, throwaway container) contains all five — 5/5, verified on the host fs. Use the process backend for trusted code with guardrails; use Docker to contain code you don’t trust.

Under the hood — static-validator coverage. A separate binary test of 20 payloads (a token prints only if the unsafe action ran) shows what the hardened validator catches before any backend runs — and, honestly, what it doesn’t:

A static scanner is bypassable by design: os.popen, pathlib and urllib evade the patterns. The validator is a first filter, not containment — which is exactly why the host-impact above is run in Docker. (On Docker those bypasses still execute, but inside a network-less, read-only, throwaway container, so nothing reaches the host.)

We lose the index contest, structurally (3/10 vs 6/10 on surfacing an authoritative source) — you can’t out-crawl a crawler. But on the actual deliverable, the cited answer, it’s a tie: 5/6 each (the one miss is an Everest height both phrase as “8,848”, which the strict substring grader counts against both). The revealing detail — our top results were still SEO pages, yet the answer was right, because the LLM extracts the fact from any decent fetched page (Tavily’s own top-3 sometimes included quora/instagram too). The answer layer compensates for mediocre sources, on both sides. The price we pay for having no index is latency.

Parity with DeepEval on judge-vs-human agreement — but the first run said 0.42 accuracy, F1 0.00: evalmcp’s LLM judge marked everything wrong, worse than the no-LLM baseline. Two bugs (a stray {{ }} in an f-string, a bare json.loads that failed on fenced replies and silently scored 0) — fixed, 0.42→1.00, with regression tests. Honest caveats: small, clear-cut dataset (both good judges hit the ceiling — parity, not a knockout), evalmcp grades correctness not Ragas-style faithfulness, and Ragas itself couldn’t run (a broken transitive dependency in the test env), so I’m not faking a number for it.

Each benchmark is one reproducible script with raw JSON in its repo: ragmcp/benchmarks/, memorymcp/benchmarks/, sandboxmcp/benchmarks/, websearchmcp/benchmarks/, evalmcp/benchmarks/. Every one of these runs surfaced or confirmed a real bug — all fixed.

Read it honestly: LTP's column is strikingly flat (~2.5k) — it compiles once instead of calling the LLM per step, so cost stays flat as deterministic depth grows — while ReAct swings from 2.6k to 25.7k (it can spiral on a trivial task like seq). LTP's one miss is iran-capital (knowledge); ReAct's is reverse (it answered without the sandbox). Hybrid is the only column with no ✗ — its two spikes (vowels, others) are verified fallbacks: the price of catching the misses. Full raw JSON for every run, model, and suite lives in the repo.