Harness Swap Lifts Same Model from 46.7% to 66.7% Success

On August 11, Pi Harness founder Mario Zechner shared some data: developer 0xEvan used Pi to call DeepSeek V4 Flash, processed nearly 1 billion input tokens, achieved a cache hit rate of 99.93%, and ended up spending just $2.65. Without caching, the same usage would have cost an estimated $132.

That same day, another developer, Shantanu Goel, said that DeepSeek V4 Flash typically maintains a cache hit rate of 94% to 97% in other harnesses, but in Pi it consistently stays above 99%. Mario commented: "This is especially useful when using local models."

There are plenty more cases like this in the community.

This also brings to mind what Mario said in May this year about the Pi + DeepSeek V4 combination: "pi + ds4 == sovereign AI enterprise ready clearly." In plain Chinese, something like: "Pi + DeepSeek 4—enterprise-grade AI, done."

At the time, it was just a joke he made after seeing a developer run a terminal Tetris with Pi and DeepSeek V4. Three months later, a public head-to-head test unexpectedly supplied the data to back it up.

Composio is a company that builds tools for AI agents. It recently ran a public comparative test. It took the same model—DeepSeek V4 Flash—and ran it in 8 different agent harnesses across 30 challenging agent tasks. These tasks required the agent to take actions, use tools, and complete the entire job independently.

Pi Agent came in first: it passed 20 of 30 tasks, a success rate of 66.7%. Oh My Pi passed 17, taking second. Claude Code, Codex, and Deep Agents each passed 16. Prime Agent passed 15, with 6 runs not scored—2 because the scorer timed out processing them, and 4 because they left no records. Hermes Agent also passed 15. OpenCode passed 14, ranking last.

The same model, with only the outer harness swapped, saw its success rate jump from 46.7% to 66.7%—a full 20 percentage points.

The cost gap is even more striking. Pi spent an average of $0.028 per successful task, while Claude Code needed $0.195, nearly seven times as much.

Pi's median time to complete a task was 132.2 seconds, slightly slower than Claude Code's 122.7 and OpenCode's 129.7. But when you weigh success rate, speed, and cost together, Pi turned in the most impressive performance in this round.

The test demonstrates what's called the "harness multiplier effect": the tooling built around an AI amplifies or weakens how the model actually performs. Choose the right harness, and the same model becomes more reliable and more efficient at the same time. Choose the wrong one, and even if the underlying model's intelligence is identical, task success rates and runtime efficiency can drop noticeably.

Composio stresses, therefore, that models should not be evaluated in isolation. If an agent leaderboard only names the model without saying which harness was used, that score is incomplete.

There's another detail in Composio's test that's worth noting: Pi had almost no extra configuration. It used a fresh, unmodified default installation, and only had the MCP server plugins needed for the test plugged in. Beyond that, Pi had no custom settings, no tuning, no special configuration. This near-out-of-the-box setup ended up passing the most tasks.

Now look at Prime Agent. It produced the largest sessions of the eight harnesses, with some sessions consuming as many as 3.5 million tokens and making 33 tool calls. Imagine an agent that, before actually starting to work, writes itself a task list as long as a phone book.

Those sessions became so huge that the scorer itself timed out while handling them. Two runs couldn't be scored, and four left no records, so six runs were excluded from the results. Even looking only at valid runs, Prime passed about as many tasks as Hermes, but took nearly twice as long as Pi.

These numbers show a stark contrast: Prime, with the most elaborate setup and the largest sessions, was dragged down by its own operational overhead, while the lighter Pi passed the most tasks at lower cost. In this test at least, adding more layers did not buy better outcomes.

This also challenges the old assumption that more configuration means better results. People typically reach for the largest model, then pile on every plugin, every extension, and every layer of complex features, assuming that more capabilities make a stronger agent. This test offers a different approach: pick a fast, low-cost model, put it inside a clean, lightweight harness, and let real tasks validate the pairing.

DeepSeek V4 Flash, as the name suggests, is a Flash model. Its positioning favors speed and operational efficiency, not winning model intelligence contests. The reason the lightweight Pi came out ahead is also simple. Every added layer is another place the agent can get lost. Every added tool is another decision it has to make. Every large instruction file is more noise it has to read before acting.

So a clean harness gives the model a short path from receiving a task to completing it, while a bloated harness makes it wander. That's why a default installation can beat heavyweight configurations: the path is shorter, and there are fewer chances to go in the wrong direction.

Pi itself is not a harness designed specifically for DeepSeek. It's more of an open agent foundation for developers: Pi lets developers modify the system prompt through extensions, filter conversation history, customize context compression, and dynamically add, remove, enable, or disable tools. Before a request goes to the model, developers can even inspect and rewrite the final payload directly. This programmability leaves a lot of room for optimizing DeepSeek's cache.

The DeepSeek API caches the prefix of the prompt in a request. If the token sequence at the start of the next request is exactly the same as the last one, the server reads those tokens directly from cache and bills them at a far lower price than normal input tokens. A cache hit costs much less than a cache miss.

The key point is that this is a prefix cache: matching must start from the very first token. If the front of the context changes, many tokens downstream may no longer hit the existing cache. The earlier the prefix changes, the more tokens get dragged down along with it.

A typical agent request includes the system prompt, tool definitions, conversation history, and new content for the current turn. On every step, the agent has to carry along a large chunk of context that has already appeared before. The longer the session, the more repeated content there is, and the better suited it is to caching in theory. But if the harness reorganizes this content on every turn—adding new timestamps, changing tool order, or rewriting the history summary—even a very long context is hard to reuse reliably.

This has spawned a wave of harness projects dedicated to optimizing DeepSeek's cache. The open-source Reasonix is a terminal coding agent designed around DeepSeek's prefix cache, and its cache performance has drawn a lot of attention from developers. One developer who prefers Pi even built DeepPi specifically to port Reasonix's cache optimization methods over to Pi. He says that when calling the DeepSeek API, DeepPi's cache hit rate stays consistently between 99.7% and 99.9%.

"I tried to port some of Reasonix's performance advantages into a Deepseek package optimized for Pi. It only activates when you use the Deepseek API, but once activated, my cache hit rate stabilized between 99.7% and 99.9%."

Reasonix's core design principle is to keep the front of the context stable, use appending rather than modifying, and minimize the cost of change.

In practice, Reasonix injects a concise, stable environment summary at startup instead of regenerating it every turn. Outdated tool output is truncated and cleaned before it triggers summary compression, so the massive result of a cat command twenty turns ago doesn't keep sitting in the prompt prefix. Reasonix also documents the schema contracts of its built-in tools and does regression reviews when they change, because if tool definitions are silently reordered, it invalidates the cache in ways that aren't visible on the surface.

In dual-model mode, the execution model and the planning model run in separate, cache-stable sessions instead of being interleaved in the same context. This is the project's most clever design. The simplest way to add a planning model would be to insert planning turns directly into the same conversation, but that breaks cache stability for both roles. Keeping them in separate sessions lets each prompt prefix stay unchanged.

The Pi ecosystem has also seen plenty of similar third-party extensions. Take pi-deepseek-cache: it also makes DeepSeek cache optimization its core goal, and its design shares a lot in common with Reasonix.

Reasonix emphasizes "injecting a stable environment summary at startup." pi-deepseek-cache's P0 layer does exactly the same thing—it freezes the date and current working directory when the agent starts, eliminating at the root the cache invalidation caused by dynamic content in Pi's default system prompt, such as Current date: YYYY-MM-DD and Current working directory:.

Reasonix emphasizes "pruning and cleaning outdated tool output before compression." pi-deepseek-cache's P3 layer achieves this through cache-friendly compression: when the conversation history gets too long and needs summarizing, it uses deepseek-v4-flash at temperature 0 for deterministic summarization, and hashes the summary result for caching. This ensures the same history input always reuses a byte-identical summary, avoiding a broken prefix caused by fluctuating summary wording.

Reasonix also mentions "documenting tool Schema contracts and running regression reviews on changes." pi-deepseek-cache's P2 layer offers a similar safeguard: it uses SHA-256 hashing to diagnose the prefix and track when the prefix changes, so developers can quickly find the root cause of cache invalidation.

The cost reduction from this extension is easy to see: with deepseek-v4-flash, the input token cost drops from $0.14 to $0.003 per million tokens, a 98% decrease; with deepseek-v4-pro, it drops from $3.00 to $0.025, a 99% decrease.

Interestingly, DeepSeek has not yet released its own harness. Just yesterday (August 11), the WeChat official account "DeepSeek Harness Team" was registered, widely read as a strong signal that a Harness product is about to launch. On top of that, internal beta testing has already begun, so an official release shouldn't be far off.

What special value could an official Harness bring? At its core, "native adaptation." Third-party harnesses can only reverse-optimize through public APIs, but an official team can coordinate back-to-back with the model training team. They can optimize the model for the Harness's calling patterns, and the Harness can use non-public information inside the model. That kind of deep integration is something no third party can replicate.

Reference links:

https://x.com/badlogicgames/status/2086877202239353285

https://www.reddit.com/r/DeepSeek/comments/1vhhxvy/deeppi_reasonixlevel_cache_performance_in_pi/

https://dev.to/arshtechpro/reasonix-deepseek-a-terminal-coding-agent-built-around-the-thing-everyone-else-ignores-3l21

https://pi.dev/packages/@rohaquinlop/pi-deepseek-cache

https://www.youtube.com/watch?v=j3c35v386T0