---

Abstract #

Model routers send each query to a cheaper model when they predict it will do as well as the frontier model, and planning estimates put the resulting saving at well over half of the frontier bill. We ask how much of that saving survives once a router is trained, calibrated and evaluated on prompts it has not seen. We used the public SPROUT routing dataset, which records judged scores and token counts for 44,241 prompts across 13 models, and priced a four-model pool (GPT-4o, GPT-4o-mini, Llama-3.3-70B, Llama-3.1-8B) at list prices read on 21 October 2026. We trained a simple text router on the training split, set its thresholds on the validation split and measured cost and quality on 6,637 test prompts. Three results stand out. First, the headroom is real: a per-query oracle reaches 104.7% of GPT-4o’s mean score at 3.4% of its cost and 110.8% at 10.9%, and Llama-3.3-70B matches or beats GPT-4o on 69.4% of prompts. Second, a trained router captures most of that headroom only in distribution: at 98% of GPT-4o’s score it saves 85.2%, against 38.6% for the best random split of traffic, but when the task family is unseen it cannot reach 98% at any price, and thresholds tuned on validation prompts undershoot the target on test prompts by 1.2 to 4.2 points. Third, router discrimination, not the price gap, sets the saving: in a GPT-4o / Llama-3.3-70B pair the in-distribution router has an AUC of 0.632 and saves 66.9% at the 98% target, and widening the price gap from 7.6 times to 161 times adds less than 10 points.

1. Introduction #

Our previous article in this series, KV-Cache Compression Techniques in Production[1]^[2], showed that the cheapest serving configuration on paper only pays off if its quality loss is measured on the workload it actually serves. Model routing poses the same question one level up. Instead of making one model cheaper to run, a router decides which model runs at all. The spread of list prices makes this attractive: on 21 October 2026, one frontier model costs $10 per million input tokens and $50 per million output tokens while a hosted open-weight model costs $0.14 and $0.28, a gap of more than two orders of magnitude.

The hub’s earlier work treated routing as a planning exercise. Our task taxonomy matched task classes to model tiers by price and estimated that systematic model-task alignment reduces per-task costs by 60% to 95%[2]^[3]. That estimate assumed the operator knows each query’s difficulty. A deployed router does not; it predicts difficulty from the prompt, and 2026 evidence suggests that the prediction is the weak link. An audit of six commercial routers across 14 settings found that none outperformed a router that randomly selects between two well-chosen models at matched cost, and that some fell short by more than 10 percentage points[3]^[4]. A study of 21 routing methods across five benchmarks found that most converge to a narrow accuracy range far below the oracle, which its authors call the routing plateau[4]^[5].

This article measures the gap between the headroom routing promises and the saving a router realises, using per-query outcomes, current prices and a held-out evaluation.

RQ1: How much of a frontier model’s cost could a per-query oracle avoid in a pool of frontier, small closed and open-weight models, and on what share of prompts is a cheaper model at least as good? RQ2: What saving does a trained router realise at 95%, 98% and 100% of the frontier model’s score, compared with a random traffic split, and how much of it survives an unseen task family? RQ3: How sensitive is the realised saving to router discrimination and to the price gap between today’s frontier and cheap tiers?

The answers matter for the Cost-Effective Enterprise AI series because routing is now offered as a managed feature by large cloud platforms and API aggregators, and its value depends on numbers that vendors rarely report: the escalation rate at a stated quality target.

2. Existing Approaches (2025–2026 state of the art) #

2.1 Learned pre-generation routers #

The dominant design predicts, before any model runs, which candidate will answer well. RouteLLM trained routers between one strong and one weak model from human preference data and reported cost reductions of over 2 times in certain cases without compromising response quality[5]^[6]. CARROT estimates both cost and accuracy for every model and selects the best trade-off; its authors prove that such a router can be minimax optimal and released SPROUT, the dataset we use here[6]^[7]. IRT-Router models query difficulty and model ability with item response theory[7]^[8], and UniRoute represents each model as a feature vector so that the router can handle more than 30 models unseen during training[8]^[9]. Public leaderboards now compare such systems: RouterArena ranks routers on a shared dataset with difficulty levels per domain[9]^[10], and one production-oriented entry, a contextual bandit initialised from one ridge regressor per model, ranked second with 75.54% accuracy at a cost of $1.00 per 1,000 queries[10]^[11]. A unified benchmark of more than 16 routers found that learned routers outperform the strongest fixed model by 14.6% relatively, with lightweight routers more competitive under tight budgets[11]^[12]. Where they fail is generalisation: semantic-embedding routers perform robustly in distribution but struggle on unseen tasks[12]^[13].

2.2 Cascades and cascade routing #

Cascades run a cheap model first and escalate when the answer looks unreliable. FrugalGPT, the 2023 reference design, reported matching the best individual model with up to 98% cost reduction[13]^[14]. Cascade routing combines both strategies; its authors proved optimality results for each and identified good quality estimators as the critical factor for either to work[14]^[15]. Conformal Cascade replaces the tuned confidence threshold with the size of a conformal prediction set and gives a distribution-free accuracy guarantee for multiple-choice tasks[15]^[16]. A two-stage design that clusters queries, assigns each cluster a model and escalates low-quality outputs retained 97% to 99% of the strongest model’s accuracy[16]^[17]. Cascades pay for the cheap attempt on every escalated query and depend on confidence signals that can mislead: on a multilingual knowledge and reasoning set, confidence-based escalation had a mean AUC of only 0.510 because small models often remain highly confident while incorrect[17]^[18].

2.3 Managed routing products #

Cloud platforms package routing as a single endpoint. Amazon Bedrock’s intelligent prompt routing predicts response quality per request, routes between exactly two models of the same family, switches only when the predicted quality difference exceeds a configured threshold, and is optimised for English prompts only[18]^[19]. Microsoft’s model router in Foundry routes across OpenAI, Anthropic, DeepSeek, Meta and xAI models; its Balanced mode considers models within about 1% to 2% of the best predicted quality and its Cost mode within about 5% to 6%[19]^[20]. OpenRouter’s Auto Router assigns each prompt one of about 30 task types and ranks models by their share of community spend for that type, with no additional fee beyond the selected model’s rate[20]^[21]. None of the three pages states an escalation rate or a realised saving at a given quality level.

2.4 Router robustness and the provider layer #

A router is an attack surface. Query-independent “confounder gadgets” appended to any prompt make routers send it to the strong model, in white-box and black-box settings, without reducing answer quality[21]^[22], and adversarial suffix optimisation achieves the same against commercial routers with only black-box access[22]^[23]. Category-driven routing creates safety gaps as well: one preference-trained router sent all coding and mathematics queries to the most powerful model while routing jailbreak attempts to weaker models[23]^[24]. For open-weight tiers, choosing the model is not the last decision. Live measurements across providers found that the same open model can vary sharply in quality, latency, availability and price, and that one deployment was nearly normal on knowledge tasks but catastrophically degraded on multi-step reasoning[24]^[25].

flowchart TD
    A[Model selection per query] --> B[Before generation]
    A --> C[After generation]
    A --> D[Managed endpoints]
    B --> B1[Learned routers: RouteLLM, CARROT, IRT-Router]
    B --> B2[Pool-agnostic: UniRoute, bandits]
    C --> C1[Cascades: FrugalGPT, Conformal Cascade]
    C --> C2[Cascade routing: route then verify]
    D --> D1[Bedrock, Foundry, OpenRouter]
    A --> E[Failure modes]
    E --> E1[Unseen tasks and languages]
    E --> E2[Adversarial suffixes]
    E --> E3[Provider quality drift]

3. Method and Data #

Per-query outcomes. SPROUT contains 44,241 prompts split into 30,968 training, 6,636 validation and 6,637 test prompts, each answered by 13 models and scored from 0 to 1 by a judge model against a reference answer, with input and output token counts per model[25]^[26][6]^[7]. The prompts come from six task families: instruction-following chat (2,912 test prompts), MMLU-Pro (1,772), MATH (1,462), RAGBench (274), MuSR (123) and GPQA (94).

Pool and prices. We kept four models that span the tiers an enterprise buys today and that still have list prices: GPT-4o as the frontier reference at $2.50 and $10.00 per million input and output tokens, GPT-4o-mini at $0.15 and $0.60[26]^[27], Llama-3.3-70B at $1.04 for both on Together AI’s serverless tier, and Llama-3.1-8B, for which we used the $0.14 price of the Llama 3 8B Instruct Lite endpoint as a proxy because the 3.1 version is not listed for serverless use[27]^[28]. All prices were read on 21 October 2026. GPT-4o has the highest mean test score of SPROUT’s frontier models (0.8482, ahead of Claude 3.5 Sonnet at 0.8338), so it serves as the baseline. Each query’s cost uses the model’s own token counts, so a model that writes longer answers pays for them.

Routers. The router is deliberately simple: TF-IDF features of the prompt and one ridge regressor per model that predicts the judged score. A pool router picks the model that maximises predicted score minus a cost penalty λ times the model’s mean training cost. A two-tier router escalates from a cheap model to GPT-4o when the predicted score gap exceeds a threshold. For each quality target (95%, 98% and 100% of GPT-4o’s score), λ or the threshold is set on validation prompts and applied unchanged to test prompts, which gives the realised result. We also report the matched-quality saving, the cheapest setting that meets the target on test prompts, which assumes ideal calibration. The unseen-family variant retrains each regressor without the task family of the prompt being routed.

Baselines and sensitivity (our estimates). The oracle applies the same rules to true scores and costs; it is an upper bound, not a router. The random baseline is the best random split of traffic between any two pool models at matched cost, the baseline used in the 2026 audit of commercial routers[3]^[4]. For router discrimination, we add Gaussian noise of increasing size to the true score gap between GPT-4o and Llama-3.3-70B and record the AUC and the saving. For the price gap, the saving of a two-tier system with escalation rate e and blended price ratio r is 1 – e – (1 – e)/r; we compute r for five current frontier and cheap-tier pairs at the dataset’s mean token mix of 221 input and 575 output tokens, using the OpenAI, Anthropic, Google and Together price pages[26]^[27][27]^[28][28]^[29][29]^[30].

All code, data and outputs are in stabilarity/hub/research/model-routing-at-scale-cost-quality-arbitrage-across-frontier-and-open-weight-model-tiers.

flowchart LR
    S[SPROUT per-query scores and tokens] --> T[Train split: TF-IDF and ridge per model]
    P[List prices, 21 Oct 2026] --> C[Cost per query from own tokens]
    S --> C
    T --> V[Validation split: set lambda or threshold per target]
    V --> E[Test split: realised cost and quality]
    C --> E
    S --> O[Oracle and random split baselines]
    O --> E
    E --> R1[RQ1 headroom]
    E --> R2[RQ2 realised saving]
    E --> R3[RQ3 AUC and price-gap sensitivity]

4. Results #

4.1 RQ1: The headroom is large and mostly sits in the middle tier #

Table 1 shows the four models on the test prompts. Two cheaper models come within five points of GPT-4o: GPT-4o-mini keeps 95.17% of its mean score at 6.8% of its cost, and Llama-3.3-70B keeps 95.09% at 12.8%. The open-weight model is not the cheapest route to that quality. At list prices GPT-4o-mini costs $0.3416 per 1,000 queries against $0.638 for Llama-3.3-70B, because Together prices the 70B model’s output at $1.04 per million tokens against $0.60 for GPT-4o-mini. Llama-3.1-8B is the cheapest per query, but it writes 994 output tokens on average, more than twice GPT-4o’s 452, and keeps only 80.96% of the score.

Model	Tier	Mean score (0-1)	Score vs GPT-4o	Mean output tokens	USD per 1,000 queries	Saving vs GPT-4o
GPT-4o	frontier closed	0.8482	100.00%	452	4.9957	0.00%
GPT-4o-mini	small closed	0.8072	95.17%	522	0.3416	93.16%
Llama-3.3-70B	large open-weight	0.8066	95.09%	332	0.6380	87.23%
Llama-3.1-8B	small open-weight	0.6867	80.96%	994	0.1705	96.59%

Test split, 6,637 prompts. Source: SPROUT scores and token counts[25]^[26]; list prices accessed 2026-10-21[26]^[27][27]^[28]; our calculation. The Llama-3.1-8B price is a proxy.

Per query, the picture is more favourable to routing than the averages suggest. Llama-3.3-70B scores at least as well as GPT-4o on 69.38% of prompts and strictly better on 8.45%; GPT-4o-mini matches or beats it on 86.73%, and even Llama-3.1-8B on 64.23%. The gains are concentrated by task: Llama-3.3-70B is the best single model on MATH (0.894 against 0.8269 for GPT-4o), GPQA and MuSR, and GPT-4o-mini is level with GPT-4o on RAGBench. An oracle that sends every prompt to the cheapest model with the highest score would reach a mean score of 0.9397, 110.78% of GPT-4o’s, at $0.5439 per 1,000 queries, a saving of 89.11%. Its cheapest setting still scores 0.8877, or 104.66% of GPT-4o, at $0.1704, a saving of 96.59%, which is why the oracle bar in Chart 2 is the same at every target.

Chart 1 shows the oracle frontier above everything a router achieves: the headroom belongs to the pool, not to any router.

4.2 RQ2: A trained router earns its keep only at strict targets and only in distribution #

Chart 2 compares savings at matched test quality. At 95% of GPT-4o’s score, routing adds nothing: GPT-4o-mini alone meets the target, so the best random split saves 93.2% and the in-distribution router 92.69%. The value of a router appears at 98%. No single cheap model qualifies there, so the best random split must send most traffic to GPT-4o and saves 38.55%, while the in-distribution router saves 85.18%, within 11.41 points of the oracle’s 96.59%. At 100% the router’s best setting reaches 99.92% of GPT-4o’s score, so the target is not met, and the random split degenerates to GPT-4o alone.

The unseen-family router is the warning. With the prompt’s task family removed from training, its best setting keeps 95.76% of GPT-4o’s score, so neither the 98% nor the 100% target is reachable at any price, and at 95% it saves 66.94%, 26.26 points less than an untrained random split. This matches the generalisation failure reported for embedding-based routers[12]^[13] and the absence of gains over random routing in the commercial audit[3]^[4].

Table 2 shows what an operator would actually see after setting thresholds on validation prompts.

Target (share of GPT-4o score)	Router	Realised score vs GPT-4o	Realised saving	Traffic to GPT-4o
95%	in-distribution	92.86%	93.85%	0.00%
95%	unseen family	93.33%	90.26%	0.30%
98%	in-distribution	95.99%	91.84%	0.00%
98%	unseen family	95.76%	57.81%	37.53%
100%	in-distribution	98.80%	77.82%	17.24%
100%	unseen family	95.76%	57.81%	37.53%

Thresholds set on 6,636 validation prompts and applied to 6,637 test prompts. Source: SPROUT[25]^[26]; prices[26]^[27][27]^[28]; our calculation.

Every validation-calibrated setting undershoots its target on test prompts, by 1.20 to 4.24 points. GPT-4o scored 0.838 on validation and 0.8482 on test, and the cheaper models did not rise with it, so a threshold that just met 98% on validation delivered 95.99% on test. At the 98% target the in-distribution router sent no traffic at all to GPT-4o; it spread prompts across GPT-4o-mini (48.98%), Llama-3.3-70B (30.48%) and Llama-3.1-8B (20.54%). The headline saving of 91.84% is therefore bought with a quality shortfall that the calibration data did not reveal.

Two tiers sharpen the picture. With GPT-4o and Llama-3.3-70B, the oracle needs to escalate only 2.47% of prompts to reach 98% and saves 85.33%. The in-distribution router needs 25.49% escalation for the same quality and saves 66.90%; the unseen-family router needs 59.59% and saves 35.79%, the same as escalating at random (35.61%). With Llama-3.1-8B as the cheap tier the in-distribution router saves only 18.80% at 98%, against 85.01% for the oracle, because the router must escalate 80.68% of prompts to compensate for its errors.

4.3 RQ3: Discrimination sets the saving; the price gap mostly does not #

Chart 3 links the saving to router quality. When the true score gap is blurred by noise, the saving at 98% stays at 85.38% or more down to an AUC of 0.706, falls to 79.47% at 0.604 and 56.09% at 0.513, and reaches 42.45% at 0.5007. Our trained routers sit below that curve. The in-distribution router has an AUC of 0.632 for the event “GPT-4o scores higher than Llama-3.3-70B” and saves 66.90%; the unseen-family router has an AUC of 0.449, worse than chance, and saves 35.79%; a random order scores 0.488 and saves 35.61%. Our reading is that a noisy oracle still escalates the prompts with the largest gaps first, whereas a text router’s errors are spread across large and small gaps alike, so AUC alone overstates what a router will save.

Chart 4 holds the escalation rate fixed and varies the price gap. At the matched-quality escalation rates for the 98% target, the ceiling on saving is 97.53% for the oracle, 74.51% for the in-distribution router and 40.41% for the unseen-family router. Today’s tier pairs span blended price ratios from 4.88 to 161.28 times.

Frontier / cheap pair (list prices, 21 Oct 2026)	Blended price ratio	Saving, oracle escalation	Saving, in-distribution router	Saving, unseen-family router
Gemini 3.1 Pro / Gemini 3.5 Flash-Lite	4.88x	77.55%	59.25%	32.13%
GPT-4o / Llama-3.3-70B (measured pair)	7.61x	84.72%	64.72%	35.10%
gpt-6.1-sol / gpt-oss-120B	16.37x	91.57%	69.96%	37.94%
Claude Opus 5.5 / Claude Haiku 5.5	40.00x	95.09%	72.65%	39.40%
gpt-6-astra / DeepSeek V4 Flash	161.28x	96.93%	74.05%	40.16%

Escalation rates from the GPT-4o / Llama-3.3-70B pair at 98% of GPT-4o’s score (2.47%, 25.49% and 59.59%), applied to other pairs as an illustration; equal tokens per query assumed. Prices from OpenAI[26]^[27], Together[27]^[28], Anthropic[28]^[29] and Google[29]^[30]; our calculation.

Moving from the measured 7.61-times pair to the 161-times pair adds 9.33 points for the in-distribution router and 5.06 for the unseen-family router, while improving the router from unseen-family to in-distribution discrimination adds 29.62 points at the measured ratio. Above about 20 times, every curve is within 5 points of its ceiling of 1 – e. Buying a cheaper cheap tier is a weak lever; sending fewer prompts to the expensive one is the strong one. Gemini’s narrow 4.88-times gap is the exception, where the price gap still costs the in-distribution router 15.26 points against its ceiling.

5. Discussion #

Implications for practitioners. First, check whether a single cheaper model already meets the target before buying a router. On SPROUT, GPT-4o-mini alone meets 95% of GPT-4o’s score at a 93.16% saving, and no router beat it. Second, if the target is strict, routing is worth 46.63 points over a random split at the 98% target, but only on task families represented in the router’s training data. Third, publish and monitor the escalation rate. It is the number that sets the saving (Chart 4), the number that moves when traffic shifts (Table 2), and the number that adversarial suffixes target[21]^[22][22]^[23]. Fourth, recalibrate on fresh traffic and add a quality margin: our thresholds missed by 1.20 to 4.24 points between two random splits of the same dataset, and real traffic drifts more than that.

flowchart TD
    A[Quality target vs frontier score] --> B{Does one cheaper model meet it alone?}
    B -->|Yes| C[Use that model, no router]
    B -->|No| D{Is traffic covered by router training data?}
    D -->|No| E[Random split or frontier only, collect labels]
    D -->|Yes| F[Learned router with validation margin]
    F --> G{Escalation rate stable on fresh traffic?}
    G -->|No| H[Recalibrate, add cascade check]
    G -->|Yes| I[Monitor escalation rate and adversarial inputs]
    E --> F

Managed routers. The three managed products reviewed above expose quality bands or thresholds[19]^[20][18]^[19] or task-type rankings[20]^[21], but not an escalation rate at a stated quality on the customer’s workload. Bedrock’s own note that routing cannot adjust to application-specific performance data[18]^[19] describes the unseen-family case in Table 2.

Open-weight tiers. The arbitrage between frontier and open-weight models is real but not automatic. At the list prices we used, a small closed model undercut a 70B open-weight model on cost per query at the same quality, because output pricing and answer length differ. Provider choice for the same open model adds a second layer of variance in quality and availability[24]^[25], which a router trained on one provider’s outputs will not see.

Limitations and threats to validity. The SPROUT models are of the 2024 generation and the scores come from a judge model, so absolute levels may not transfer to 2026 models; the relative findings, oracle headroom, the router’s capture of it and its loss under task shift, are what we expect to carry over, and the routing plateau study found the same gap between routers and the oracle across 21 methods and five benchmarks[4]^[5]. Our router is a TF-IDF and ridge baseline; in the plateau study, more data, larger encoders and end-to-end fine-tuning improved eight routers by only 1.24 points on average[4]^[5]. Prices are list prices on one day, the Llama-3.1-8B price is a proxy, and batch, caching and committed-use discounts are ignored. The price-gap analysis transfers one pair’s escalation rates to other pairs, which assumes that the cheap model’s relative quality is similar. Router inference cost and latency are not modelled.

What would change the conclusion. A router whose AUC on unseen task families stays above about 0.7 would put realised savings near the oracle-noise plateau of about 85% at the 98% target in Chart 3. Evidence that cheap models in a pool fail on different prompts from one another, rather than on the same hard prompts, would raise the value of multi-model pools over two tiers. A provider pricing its frontier model close to its small model, as in the Gemini pair, would cut the saving regardless of the router.

6. Conclusion #

RQ1 Finding: The pool, not the router, holds the headroom. Measured by the per-query oracle = 110.78% of GPT-4o’s mean score at an 89.11% saving and 104.66% at a 96.59% saving, and by the share of prompts on which Llama-3.3-70B matches or beats GPT-4o = 69.38%. This matters for the series because it puts a hard upper bound on routing claims and shows that the 60% to 95% savings estimated from task taxonomies are achievable only with near-perfect difficulty prediction.

RQ2 Finding: A trained router adds value at strict quality targets and only within the task families it was trained on. Measured by the saving at 98% of GPT-4o’s score = 85.18% for the in-distribution router against 38.55% for the best random split, and by the unseen-family router’s best quality = 95.76%, below the target at any price; validation-calibrated thresholds undershot their targets by 1.20 to 4.24 points. This matters for the series because realised savings depend on calibration and coverage, which vendors’ headline figures do not report.

RQ3 Finding: Router discrimination dominates the price gap. Measured by the two-tier saving at 98% = 66.90% at an AUC of 0.632 against 35.79% at 0.449, and by the gain from widening the price ratio from 7.61 to 161.28 times = 9.33 points for the in-distribution router. This matters for the series because it shifts the cost question from which cheap model to buy to how few queries need the expensive one.

The next question for the series is how cascades with output verification compare with pre-generation routers once the cost of the cheap attempt and the verifier are both counted.