The last few years of research has resulted in relentless improvement in the capabilities of LLMs. This is likely to continue. Coding agents built on these models have allowed forward-looking software teams to move into agentic engineering, where 100% of the code is written by agents. While this has resulted in large productivity gains, token costs at the organization level have often grown faster than expected¹.

In this blog post we argue that a new frontier focused on cost deserves attention. We present the results of an experiment that gives a quantitative illustration of the possibilities at this frontier. We make two principal claims:

1. It is possible to achieve frontier capability in coding agents at a fraction of the cost.

2. The companies building the most capable models (OpenAI, Anthropic, Google) do not have the incentive to pursue the cost frontier.

Motivation

Software development tasks range widely in their essential complexity. On the low end the task might be a simple refactor, like renaming a function across many usages. At the high end it might be to design and implement a brand new feature in the product.

Despite this, most of us use our coding agents in a fixed configuration; often it’s an expensive frontier model set to a high or extra high reasoning level. This makes sense. Frequent manual model switching is not practical and it’s not possible to predict the complexity of the task in advance.

This is a little absurd, though, because in many cases we’re doing the equivalent of asking a research mathematician to compute the sales tax on a grocery bill. We’re paying the cost² of a frontier model even when the task does not require its intelligence.

What if there was a way to know in advance the cheapest model that could do a given task? How much could we save by doing that perfect assignment?

We spent $12,000 and did the experiment to find out.

Experiment

Our goal was to figure out the lowest cost we could achieve on a set of coding tasks if we could perfectly route each task to the cheapest model that could do it. Of course since we don’t have perfect foresight, we used the next best thing, hindsight. We chose a set of models with a range of token costs, and we gave each task to all the models, and computed the total cost by adding up the cost of either the cheapest model that completed it or the cost of the most expensive model in case of failure.

We chose the publicly available portion of SWE-Bench Pro as our representative set of coding tasks. There are 731 tasks in this benchmark and they’ve been chosen to capture the diversity of real-world software challenges and minimize the chances of the solution being in the training set of the models.

We chose thirteen model configurations for our experiment: five models and various reasoning settings across Anthropic and OpenAI, summarized in the table below.

Provider	Model	Reasoning level	Relative input cost	Relative output cost
Anthropic	Fable 5	high, xhigh	13.3	11.1
Anthropic	Opus 4.8	high, xhigh	6.7	5.6
OpenAI	GPT-5.4	medium, high, xhigh	3.3	3.3
OpenAI	GPT-5.5	medium, high, xhigh	6.7	6.7
OpenAI	GPT-5.4-mini	medium, high, xhigh	1.0	1.0

The relative cost columns show the multiples for each model, using the cheapest model, GPT-5.4-mini, as the baseline.

While SWE-Bench Pro provides its own agent (SWE-agent), we used each provider’s own harness (Claude Code and Codex CLI) for runs involving their model, because we’re interested in comparing the real-world cost of using agents, as opposed to comparing raw model capabilities inside a single harness.

Thus our experimental protocol was:

1. Run the entire SWE-Bench Pro suite once in all thirteen model configurations.

2. For each task:

   • Mark it as completed if any of the model configurations were successful.

   • If any model solved the task, consider the cost of the task to be that of the cheapest model that could do it. If all models were unsuccessful, consider the cost to be that of the most expensive model in the mix.

The total cost of running this experiment was approximately $12,000. We present our results in the graph above.

The most interesting point on the graph is that of peak productivity. This is the point at which money spent on tokens yields the greatest organizational benefit without sacrificing capability.

For instance, at the peak productivity point, frontier accuracy can be achieved at 58% of the cost of GPT-5.5 (xhigh), 52% of the cost of Opus 4.8 (xhigh) and 38% of the cost of Fable (xhigh). The exact results are presented in the table below.

Run	Accuracy	Total cost
Claude Fable 5, xhigh	88.6%	$2,171
Claude Fable 5, high	86.6%	$1,593
Claude Opus 4.8, xhigh	68.8%	$1,556
Claude Opus 4.8, high	64.3%	$847
GPT-5.4, xhigh	51.7%	$834
GPT-5.5, xhigh	51.2%	$1,413
GPT-5.5, high	50.3%	$1,066
GPT-5.4, high	48.6%	$490
GPT-5.4-mini, xhigh	48.3%	$496
GPT-5.5, medium	46.6%	$756
GPT-5.4, medium	46.1%	$372
GPT-5.4-mini, high	43.6%	$261
GPT-5.4-mini, medium	42.3%	$201
Peak productivity	93.6%	$814

In this experiment the routing informed by hindsight achieved accuracy that’s greater than any individual model, as a side effect of being able to take advantage of the particular strengths of each model.

The rest of the points on the frontier represent trading off accuracy for lower cost. Each of the thirteen points represents a version of the experiment where only a cheaper subset of the model configurations was made available, from just the cheapest one (GPT 5.4 mini medium) at the bottom, to all thirteen configurations at the top. Restricting the available models in this way caps the spend but reduces accuracy because many more cases fail.

The opportunity

The entire cost frontier represents the opportunity to build a new class of products. These products can only be built by routing across multiple providers (and maybe adding open-weights models into the mix) because that’s the only way to achieve the lowest cost without compromising on accuracy. Moreover, no single provider has the incentive to optimize for this frontier. They are focused on building the most capable models in the world regardless of cost.

The practical path to building products at the new frontier is to replace hindsight with policies. As an example, consider a trivial policy that runs every task on GPT-5.4-mini (medium) at first, and reruns with Claude Fable 5 (xhigh) only upon failure. This policy achieves success on 654 tasks (89.5% accuracy) while costing $1,623.

We can imagine policies that are more intricate. For a start, model switching does not have to happen at the granularity of a task. It can happen at each turn, or a single turn might use a blend of models through the use of subagents. A policy could in theory be tuned to the specific characteristics of a repository (e.g., always use the strongest model when touching “risky” parts of the code), or the preferences of an organization. Finally, if we frame this as an optimization problem, the policy can itself be learned by a model custom built for that purpose.

The cost frontier of agentic engineering remains mostly unexplored. As the capabilities and token costs of the models keep increasing, delivering cost-effective productivity gains for software teams will be an urgent technical problem worthy of a team willing to obsess over it.

At Journal we are building for the future of agentic software engineering. If this research piqued your curiosity, or if you’re engaged in similar work and would like to share, we would love to hear from you. Please get in touch.

Footnotes

1. Uber Caps Usage of Tools Like Claude Code to Manage Costs ↩

2. We will ignore subsidized plans and only consider API token costs from each provider. ↩