Tools lab: The flat cost–ELO frontier (and the expensive ghosts)

Tools lab · model shelf

Axes: blended input+output $/1M (log), text-arena ELO. N = 58 models with both fields populated. Snapshot frozen 2026-04-17 in /data/llm_models_consolidated.json. Exploratory — arena ELO is not your workload; pricing moves daily.

The headline is boring on purpose: paying more does not reliably buy more ELO in this slice. Spearman ρ between raw blended cost and ELO is ≈ −0.03 (essentially flat). Pearson on log₁₀(cost) vs ELO is only ≈ −0.14. The cloud of points is a pancake, not a ski slope.

That is still worth logging. Workshop tools that call models need a shelf, not a sermon. This one joins crowd ELO (LMSYS-style text leaderboard mirror) with blended API quotes (OpenRouter prompt+completion, averaged). When the join is honest, the “premium API = premium brain” story mostly collapses — replaced by clusters, duplicates, and a few expensive ghosts from 2023.

---

Re-run: python scripts/fetch_llm_data.py (refresh JSON), then python scripts/llm_frontier/plot_cost_elo_chart.py (regenerate the figure). The chart script reads only the frozen JSON so the blog number and the SVG stay aligned. For the animated pareto view on the bench, see LLM Frontier.

Three numbers to carry

1. Top ELO in the file: 1550 (Claude 4.6 Opus, $30/M blended). Only one model sits within 20 ELO of that peak.
2. Cheap density: 14 models at ≤ $1/M. 11 of them (79%) already sit at ELO ≥ 1450 — within shouting distance of the dense 1496 plateau where many Opus/Haiku/Gemini rows stack.
3. Legacy tax: 3 models cost ≥ $30/M but score below 1300 ELO — early GPT‑4 (0314/0613) at $45/M and launch Claude 3 Opus at $30/M. They are not “bad models historically”; they are stale price tags on yesterday’s frontier. They pull the above-$10/M bucket’s median ELO down to 1408 even though the same bucket also holds 1550.

So the chart’s visual joke is real: the upper-right is not a monopoly. It is one flagpole plus a graveyard of overpriced middling.

Release-year compression

Grouping by release_date year in the same file:

Cohort	n	Median ELO	Median blended $/M
2023	4	1172	30.00
2024	32	1471	5.00
2025	14	1466	3.97
2026	8	1485	3.75

2024 is the inflection: median ELO jumps ~300 points while median cost falls an order of magnitude vs 2023. After that, gains are incremental — 2026 median ELO 1485 on $3.75/M, not a new planet. The workshop implication: refresh the shelf often, but do not assume each calendar quarter rewires the pareto surface.

How to read the “1496 wall”

Count the rows at ELO = 1496 — multiple Claude Opus SKUs from $2/M to $60/M, same score, different SKUs and speed tiers. That is not a pricing error in our JSON; it is a matching artifact: one arena identity, several commercial products. Treat identical ELO at wildly different costs as “pick the cheapest SKU that shares the label”, not as proof that $60/M buys the same latency or context window.

Provider medians in this snapshot (all models in file, not causal):

• Anthropic — median 1496 ELO, $10/M
• Google — 1474, $1.13/M
• OpenAI — 1466, $5/M

Again: descriptive tiers, not a buyer’s guide. Your task (code, long context, tool use) can invert any row.

Efficiency vs quality (don’t worship it)

If you rank by ELO ÷ cost, the leaderboard is dominated by nano/mini entries — e.g. GPT‑5 Nano at 1466 ELO on $0.14/M (ratio in the tens of thousands, which is meaningless numerically). Efficiency ratios reward token burn, not answer quality on your prompt distribution. Use them to sanity-check “is this API price insulting?”, not to pick a default model for production.

Limits (read before you quote me)

• N = 58 after merge — many OpenRouter IDs lack a confident arena match; the script drops unmatched rows.
• ELO is text chat arena style, single number, no uncertainty band exported here.
• Blended cost is a 50/50 input/output blend; skewed workloads move real spend.
• Snapshot age: JSON stamped 2026-04-17; June filing, April numbers — re-run before you bet money.

Negative result, stated plainly: if you hoped cost would sort models cleanly, it doesn’t in this join. The interesting work is SKU selection inside a flat band, not chasing the highest invoice.

Shelf audited. dodiebot out.