Frontier pricing isn't your problem — 5 cost-competitive models, ranked

Operational cost visibility — log per-query cost, catch the p99 tail, build the eval — is one half of cost discipline. The other half: which model, which format, which architecture.

The thesis: for the 80% of tasks that don’t need frontier reasoning, five cost-competitive models deliver at 3x to 45x lower cost than GPT-5.5 or Claude Opus 4.8. The remaining 20% — long-horizon agentic, frontier reasoning, multimodal — still goes to the leading proprietary models. The mistake I have watched most often is the converse: spending frontier proprietary money on the 80%, then being unable to afford the frontier proprietary spend that the 20% actually requires.

1. The model roster — capability composite, not single-number ranking

I have spent the last 6 months re-pricing the five models I now reach for before the frontier APIs. The capability composite is from AA Intelligence Index v4.1 (2026-06-18). The cost column is per-vendor verified 2026-06-17, with MiMo-V2.5-Pro the exception (AA blended pricing is the only public source). Order is by AA Index. The two at 44 are listed by recency + agentic-strength: MiniMax M3 first as the 2026 forward-looking model, then DeepSeek V4-Pro as the cost leader (named separately in the result note).

The cheapest model is not the most capable model, and the most capable is not the cheapest. The decision rule is: pick by workload, not by index number.

2. Format — the smallest lever

Prompt format gets discussed more than it delivers. TOON (Token-Oriented Object Notation) and GCF (Graph Compact Format) cut prompt tokens 45-48% on uniform-array payloads, in a 30-trial test on gpt-5-mini high. End-to-end cost saving is smaller — 11.9% in the best case (matrix best cell). Other workloads may differ.

The gap between prompt-token saving and end-to-end saving is reasoning tokens. The 45-48% prompt saving assumes the model still produces the same number of output tokens, including any chain-of-thought. On reasoning-heavy workloads, the prompt saving is partially offset by the reasoning cost. Reasoning tokens are priced as output (8x the input rate for gpt-5-mini), and the model spent extra reasoning parsing the unfamiliar TOON/GCF section header and delimiter. The 10-row × 30-trial × 7-cell sample is too small to power an accuracy claim either way (hit@10 = 50% on every cell, including JSON).

Format saves tokens, not dollars. The 11.9% is a snapshot for gpt-5-mini high; future models with more pretraining exposure to TOON/GCF will likely save more. The bigger savings are model + architecture.

3. Architecture — four levers that compound

Four architecture levers, applied together, decide whether the system holds up at 1K+ queries/day.

The four levers compound. 1M context lets you fit more; hybrid + rerank makes what you fit more relevant; prompt caching makes the repeated parts cheaper; routing matches the right model to the right task. The cheapest lever is still the RAG-side one — a 10-line chunker change, not a 10-week migration.

4. Operational sub-levers

Two patterns that operate on conversations (not single queries) or on vendor-managed infrastructure (not user code). Both are cheap to enable, both have sharp failure modes.

Summarize + tier

For multi-turn workflows, conversation history grows. The naive approach sends the full transcript on every call. A two-step pattern cuts that: a cheap model (gpt-5.4-mini or haiku 4.5) summarizes the conversation history when it crosses a threshold (2,000-32,000 tokens is the common range), the summary is prepended to the strong model’s context, and the last 2 turns are kept verbatim for short-term coherence.

On a 10-turn conversation with gpt-5.5 strong + gpt-5.4-mini cheap for summarization, the reduction is 30-50% on multi-turn costs. The cheap model handles the summarization at 5-10% of the strong-model rate; the strong model only sees the summary plus the last few turns. The OpenAI cookbook pattern (context_summarization_with_realtime_api, May 2025) is the reference implementation; the cascade structure follows FrugalGPT (Chen et al., 2023).

Failure modes:

• Lost in the middle. The summary may drop a key detail from turn 5.
• Reference failure. The user says “what did you say about X?” — turn 5 is no longer in the context.
• Summary cost compounds. Re-summarization runs every time the threshold is crossed. Cheap model or not, the cost adds up.
• Hallucination. The cheap model invents a fact the user never said.

Use when: long agentic loops, multi-turn research workflows, conversational RAG with follow-up questions. Don’t use when: single-turn queries, retrieval-heavy workflows where the answer is in the retrieved documents.

Server-side compaction

OpenAI’s /responses/compact endpoint auto-prunes old context when the conversation crosses a threshold. The vendor returns an encrypted opaque compaction item that replaces the old turns; the developer never sees the raw text. Zero Data Retention-friendly when store=false — OpenAI does not persist the compacted context. For previous_response_id chaining, pass only the new user message each turn; the compaction happens server-side. For stateless input-array chaining, append output items (including compaction items) on each turn; drop items before the latest compaction to control latency.

No equivalent on Anthropic, DeepSeek, Moonshot, or Z.ai as of 2026-06-20. Vendor-specific infrastructure — useful for OpenAI customers, irrelevant elsewhere.

Use when: OpenAI is the strong model, conversation history is volatile, ZDR compliance is required. Don’t use when: cross-vendor flexibility matters, you need to audit the compacted context yourself.

The two sub-levers compound with the four architecture levers in §3. Summarize + tier is per-conversation; server-side compaction is per-infrastructure. Both sit on top of the per-query levers (1M context, hybrid + rerank, prompt caching, route by difficulty) — they don’t replace them, they reduce the surface area those levers operate on.

The pattern

Four categories of cost discipline. The first is operational: log per-query cost, catch the p99 tail, build the eval. The second is model: pick the cheapest model that meets the capability bar, ordered by AA Index, deployed on cloud-provider paths for data governance. The third is architecture: 1M context + hybrid + rerank + prompt caching + routing by query difficulty, in that order. The fourth is operational sub-levers: summarize + tier for multi-turn conversations, server-side compaction for OpenAI customers.

The era of “as much AI as you can” is over. The era of “as much AI as cost-justified” is here. The model is rarely the part that decides whether the system ships. The procurement, the architecture, and the operational discipline are.

Pick the model that fits the task, not the model with the highest marketing budget.

Methodology note

• 5 cost-competitive models evaluated against frontier proprietary. The “open-weight” framing applies to 4 of 5 (GLM 5.2, DeepSeek V4-Pro, Kimi K2.6, MiMo-V2.5-Pro); MiniMax M3 uses a restrictive minimax-community license, not an OSI-approved open-source license.
• Capability ranking: AA Intelligence Index v4.1 (artificialanalysis.ai, 2026-06-18) — a weighted composite of nine benchmarks per AA v4.1 methodology: GDPval-AA v2, τ³-Bench Banking, Terminal-Bench 2.1, SciCode, AA-LCR, AA-Omniscience, Humanity’s Last Exam, GPQA Diamond, and CritPt. Ranks the 5 cost-competitive models in this deck; cost-per-task measurements come from the AA cost-per-task panel (DeepSeek V4-Pro at $0.04/task, Claude Opus 4.8 at $1.78/task — 44x gap, the 40-45x figure in the result note).
• Cost data: verified 2026-06-17 from each vendor’s official pricing page, with the exception of MiMo-V2.5-Pro where AA blended pricing ($0.18/M tokens) is the only public source.
• 49-67% retrieval-failure reduction: Anthropic’s “Contextual Retrieval” announcement (Sep 2024) — hybrid 49%, +rerank 67%.
• 0.1x cache-hit pricing: standard Anthropic prompt-caching tier (8x the cache-miss input rate).
• 40-45x DeepSeek cost ratio: from Artificial Analysis’s Intelligence Index v4.1 cost-per-task measurements.
• 11.9% end-to-end format saving: from a 30-trial × 7-cell × 10-row canary on gpt-5-mini with reasoning effort high; the matrix best cell, hit@10 = 50% on every cell (including JSON), so the sample is too small to power an accuracy claim.
• 30-50% multi-turn cost reduction from summarize + tier: a worked-example characteristic of the conversation-summarization pattern (OpenAI cookbook, context_summarization_with_realtime_api, May 2025) combined with a cheap-model cascade (FrugalGPT, Chen et al. 2023), measured on a 10-turn gpt-5.5 + gpt-5.4-mini pattern.
• Server-side compaction: OpenAI-specific (/responses/compact, platform.openai.com/docs); no equivalent on Anthropic, DeepSeek, Moonshot, or Z.ai as of 2026-06-20.
• Models referenced throughout: GLM 5.2, MiniMax M3, DeepSeek V4-Pro, Kimi K2.6, MiMo-V2.5-Pro — ranked by AA Intelligence Index v4.1, see artificialanalysis.ai/methodology.

Related: Why 70% of AI pilots never reach production — and the 3 RAG fixes that worked