Fancy Harnesses Increase Costs for Coding Agents Without Enhancing Performance
1 min read AI for Software Engineering (Copilots, SDLC, Testing) -/5
In short
  • A recent controlled study reveals that utilizing various coding-agent harnesses, such as Claude Code, OpenCode, or mini-SWE-agent, yields minimal improvement in pass rates compared to merely
  • Notably, the cost per task can escalate significantly, potentially tripling due to the necessity of resending the system prompt at each step.
  • This finding raises important questions about the cost-effectiveness of adopting these advanced harnesses.
-/5 (0)
A recent controlled study reveals that utilizing various coding-agent harnesses, such as Claude Code, OpenCode, or mini-SWE-agent, yields minimal improvement in pass rates compared to merely rerunning the same setup. Notably, the cost per task can escalate significantly, potentially tripling due to the necessity of resending the system prompt at each step. This finding raises important questions about the cost-effectiveness of adopting these advanced harnesses. While they may appear to offer enhanced capabilities, the evidence suggests that they do not translate into better outcomes. Therefore, stakeholders must carefully weigh the financial implications against the actual performance benefits when considering the integration of these tools into their operations.