Industry Research, AI Inference

The Inference Flywheels

Performance and cost loops in AI inference, and what they mean for frontier labs, compute economics and competition
14 October 2026. Library filings cited by company and event; web marks facts from outside sources/; Interpretation marks analysis.

How to read this report

Scope. This report covers the inference side of AI: what happens after a model is trained, when it is served to users and agents. It looks at two self-reinforcing loops. In the performance flywheel, more inference makes models better. In the cost flywheel, more inference makes each token cheaper. It traces what the two loops mean for frontier labs, for the economics of AI compute, and for the competitive landscape.

Evidence. Company facts come from the earnings calls, conference transcripts and filings in the Research library’s Equity/*/sources/ folders, and each one is tagged with the company and event, for example (NVDA Q2 FY27 call). Facts from outside the library are tagged web and listed separately at the end. Most of these are secondhand (Epoch AI, SemiAnalysis, The Information, OpenRouter data as reported by others), and they are flagged where they conflict. Statements that are my own reasoning are labelled Interpretation.

What this is not. It is not a valuation and not a buy/sell view. Management claims about their own price-performance (“lowest cost per token”, “half the cost of a GPU”) are vendor claims and are reported as such.

Bottom line

  1. The cost flywheel is real and fast. The price of reaching a fixed level of capability is falling by roughly 10x to 40x a year, depending on the benchmark and period web: Epoch AI. Companies report the drivers directly: NVIDIA says GB300’s cost per token fell 60% in six months from software alone (NVDA Q1 FY27 call), Google cut Gemini serving unit costs 78% over 2025 (GOOG Q4 FY25 call), and AWS quadrupled Trainium2 token throughput in 2025 (AMZN Q1 FY26 call).
  2. Cheaper tokens have raised demand for compute, not lowered it. H100 rental prices rose about 20% in the first five months of 2026 (NVDA Q1 FY27 call). Microsoft, Amazon and Google all say demand still exceeds supply (MSFT Q4 FY26, AMZN Q2 FY26, GOOG Q4 FY25 calls). Google’s API token volume more than doubled in two quarters, from 10 to 22 billion tokens a minute (GOOG Q1 and Q2 FY26 calls). Lower cost per token is being spent on more tokens per task (reasoning and agents) and on new uses.
  3. The performance flywheel is real but leaky. Spending more inference makes answers better (reasoning, test-time compute). Reinforcement learning, which is mostly inference, makes models better, and product usage creates training signal. But capability gains spread quickly through distillation and open-weight models, so a performance lead decays unless the lab keeps reinvesting.
  4. Together the loops form a treadmill for frontier labs, not a moat. Cost declines turn each capability tier into a commodity within quarters, while the frontier price point holds roughly steady. Labs earn premium margins only on the newest tier, so they must keep buying compute to stay there. Their binding constraint is compute and the financing for it. NVIDIA now invests in labs and arranges financing for them (NVDA Q2 FY27 call), and Broadcom describes helping two labs “bridge the gap” between cash flow and infrastructure needs (AVGO Q3 FY26 call).
  5. In compute economics, power is the denominator. A data center is capped in megawatts, so tokens per watt sets the revenue ceiling of the site. This lets the newest hardware command a premium even when it costs more per chip, and it is NVIDIA’s central argument for its margins (NVDA GTC 2026 keynote and analyst Q&A). The open question is depreciation. Six-year useful lives meet 30x-a-generation efficiency gains, and only tight supply has so far kept older GPUs valuable.
  6. Inference shifts the competitive balance toward vertical integration and specialisation. Inference workloads are stable and known in advance, so they can be served by custom chips (TPU, Trainium, Maia, MTIA, OpenAI’s chip with Broadcom) and by speed specialists (Cerebras, and Groq inside NVIDIA). Google, which owns the model, the chip and the distribution, has the most complete cost flywheel. NVIDIA’s counter is fungibility (one platform for every model and phase), co-design speed, and financing.
  7. Where value accrues today: the owners of scarce, efficient compute (NVIDIA, the custom-silicon partners, the hyperscalers with their own chips) and the owners of distribution (the hyperscalers, plus the labs with products people pay for, such as coding agents). Pure model access without distribution or a cost edge is where the cost flywheel bites hardest. Chinese open-weight models carry a large share of tokens but a small share of dollars web.

The two flywheels in one picture

Performance flywheel Cost flywheel More inference per taskreasoning, agents, speed Better answerstest-time compute Better modelsRL, feedback, distillation More usage andtoken volume Scale and utilisationbatching, fungible fleets Efficiency leverschips, software, ASICs Lower cost per token~10x to 40x a year cheaper tokens buy more tokens per task (Jevons) Cost savings are spent on more inference per task, which drives performance, which drives volume.

The loops connect at two points. Savings from the cost flywheel are spent on more inference per task, which drives the performance flywheel. Better performance drives the volume that feeds the cost flywheel. Interpretation: this coupling explains how prices per token can collapse while spending on compute keeps rising.

The performance flywheel

The performance flywheel is any mechanism by which more inference makes the system more capable. Five mechanisms matter.

1. Test-time compute: thinking longer gives better answers

Reasoning models spend more tokens per answer, either through longer chains of thought, several sampled answers checked against each other, or search. Within limits, accuracy rises with that spend web: test-time scaling literature. Commercial models expose this as an “effort” or “thinking budget” setting web.

Evidence in the library. NVIDIA’s CEO describes the economy as a set of token tiers, with today’s typical price around $6 per million tokens and demand for a $50-per-million tier of “super large models, super fast” for which highly paid engineers would pay (NVDA GTC Financial Analyst Q&A, March 2026). He also argued that “token cost is gonna keep on coming down” while “the smartness per token” keeps rising (same). Google said Gemini 3 Pro processed three times as many daily tokens as 2.5 Pro after launch (GOOG Q4 FY25 call). A better model drew more usage, and probably more tokens per request.

Why it matters. Test-time compute turns inference from a fixed cost of serving into a lever for quality. The customer chooses how much intelligence to buy per task. That makes demand for inference much more elastic to price than a classic “serve the same answer cheaper” model.

2. Reinforcement learning is mostly inference

Post-training with reinforcement learning (RL) works by having the model generate many attempts (rollouts), scoring them, and updating the model. The generation step is inference. NVIDIA now describes the AI lifecycle in four phases (data preparation, pre-training, post-training, agentic inference) and argues that one fungible system should serve all of them (NVDA Q2 FY27 call). Amazon says post-training RL and agent tool use run mostly on CPUs, which is driving CPU demand alongside accelerators (AMZN Q1 and Q2 FY26 calls).

Interpretation. The line between training compute and inference compute is blurring. A lab that expands its inference fleet can also run more RL, and RL is where most recent capability gains have come from (reasoning, coding, tool use). This is the strongest internal link in the performance flywheel.

3. Usage produces training signal

Products in use generate feedback such as accepted or rejected edits, follow-up complaints and agent traces, and that feedback can become RL reward. Cursor describes a flywheel of real-world user feedback, private evaluations, RL environments and better coding models, and trains its Composer models with RL in the same harness that serves users web: Cursor technical report and talks. Third-party write-ups claim measurable gains from real-time RL on user sessions, but these are unverified web.

Inside the library, Meta describes the same logic in ads. Its adaptive ranking model routes requests to more compute-intensive inference models when a conversion is more likely, which “improves our inference ROI” (META Q1 FY26 call). Microsoft says customers will judge AI by “evals” and “where they see the value of tokens” (MSFT Q3 FY26 call).

Caveat. The data flywheel is strongest where outcomes are verifiable (code that compiles and persists, ads that convert) and where the lab owns the product surface. For a general chat API, the provider sees prompts but not outcomes, so the loop is weak. Interpretation: this is why labs are pushing into owned products such as coding agents, enterprise agents and consumer apps rather than staying API wholesalers.

4. Distillation: big models teach small ones

A frontier model’s outputs can train a smaller, cheaper model that keeps much of the capability. Meta says it does not serve its largest ads model (GEM) directly because that would be too costly. Instead it transfers the larger model’s knowledge into lightweight runtime models (META Q1 FY26 call). Microsoft reports an 89% reduction in GPU costs in Dynamics 365 and up to 84% in PowerPoint by using its own smaller models (MAI-Voice-2-Flash, MAI-Image-2.5), and says MAI models match GPT-5.6 quality on the most common tasks at significantly lower cost (MSFT Q4 FY26 call). Google releases “cost-efficient Flash models” alongside each Pro generation (GOOG Q1 and Q2 FY26 calls).

Interpretation. Distillation is the bridge between the two flywheels: performance created at the frontier becomes cost reduction one tier down. It also leaks. Anyone with access to a frontier model’s outputs can distil from it, which is one reason open-weight models trail the frontier by only months.

5. Speed is itself performance

For agents, latency compounds: an agent that makes 50 sequential model calls feels each delay 50 times. Cerebras argues that “speed is not a benchmark item. Speed changes user engagement, it changes agentic performance, and it changes AI productivity” (CBRS Q2 FY26 call). It says it serves OpenAI’s largest model at 10x the speed of alternatives (company claim). NVIDIA licensed Groq’s inference technology and hired its leaders in a $20 billion deal in December 2025 web, and now sells Groq 3 LPX racks for “super high interactivity” tiers that carry higher cost per token but “high ASP services” (NVDA Q2 FY27 call).

Interpretation. Speed creates a premium tier inside inference. The cost flywheel compresses the price of ordinary tokens, while fast tokens on frontier models keep a price premium. This is the market segmentation NVIDIA describes in its tier framework.

Where the performance flywheel breaks

  • Leakage. Distillation and open weights spread capability gains. Chinese open-weight models are reported at roughly 30% to 46% of US organisations’ tokens on OpenRouter in 2026 and around 60% to 90% cheaper than leading closed models web; figures vary by definition.
  • Benchmarks overstate gains. Epoch AI notes the risk that models are tuned to benchmarks, so measured gains may exceed real-world gains web.
  • Weak loop for undifferentiated APIs. Without product ownership, usage gives little proprietary signal.

The cost flywheel

The cost flywheel is any mechanism by which more volume, or simply more time, lowers the cost of producing a token of given quality. The library’s filings point to six levers.

Lever 1: Hardware generations, measured per watt

NVIDIA frames everything per watt because “every data center, every single factory, by definition, is power-constrained. A 1 GW factory will never become 2 GW” (NVDA GTC 2026 keynote). Its claimed gains:

Step Claimed gain Source
Grace Blackwell NVL72 vs H200 35x perf/watt claimed; SemiAnalysis said ~50x NVDA GTC 2026 keynote
GB300, six months of software 2.7x throughput, 60% lower cost per token NVDA Q1 FY27 call
Vera Rubin vs GB300 30x throughput per MW, 35x lower token cost NVDA Q2 FY27 call
Inference provider software update, same hardware ~700 to ~5,000 tokens/s NVDA GTC 2026 keynote

All are NVIDIA claims about its own products; benchmark conditions vary.

Lever 2: Software on installed hardware

Hyperscalers report large gains with no new chips:

Company Gain Source
Google Gemini serving unit cost down 78% over 2025 GOOG Q4 FY25 call
Google Cost of core AI responses in Search down >30% after Gemini 3 upgrade; AI Mode cost at its lowest since launch GOOG Q1 and Q2 FY26 calls
Microsoft +50% throughput on OpenAI inference for Copilots MSFT Q2 FY26 call
Microsoft +40% inference throughput for most-used Copilot models MSFT Q3 FY26 call
Amazon 4x Trainium2 token throughput in 2025 AMZN Q1 FY26 call

Microsoft adds a point that matters for depreciation: the fleet runs newer models over time, so “as you go through the useful life, actually, you get more and more and more efficient at its delivery,” and margins on contracted GPUs “actually improve with time” (MSFT Q2 FY26 call).

Lever 3: Model efficiency and routing

Smaller distilled models, mixture-of-experts designs and lower-precision maths cut the compute per token (general technique; the library’s evidence is the Flash, MAI and Meta examples above). Routing sends easy requests to cheap models and hard ones to expensive models. Meta says that “if you’re serving billions of people… you want the ability to serve the vast majority of prompts from simpler and more efficient models” (META Q2 FY26 call).

Lever 4: Custom silicon

Inference workloads are stable enough to justify chips designed for them. Each claim below is the company’s own:

Chip Claimed advantage Source
AWS Trainium2 / Trainium3 ~30% to 40% better price-performance than comparable GPUs; Trn3 30% to 40% better again; “several hundred basis points of operating margin advantage” for inference; >$225B of Trainium revenue commitments AMZN Q4 FY25, Q1 FY26 calls
Google TPU “80% better price performance for inference”; TPU business “more than twice” the next-largest hyperscaler’s GOOG Goldman Sachs conference, Sept 2026
Microsoft Maia 200 >30% better tokens per dollar than latest silicon in its fleet MSFT Q3 FY26 call
Broadcom XPUs (Google, Anthropic, OpenAI, Meta) “Half the cost” of a GPU for the customer’s own workloads AVGO Q3 FY26 call

Amazon says Trainium will save it “tens of billions of dollars of CapEx each year” (AMZN Q1 FY26 call). Bedrock runs most of its inference on Trainium (same).

Lever 5: Disaggregation and specialisation

As inference matures, the two phases of serving a request split onto different hardware: prefill (reading the prompt) and decode (writing the answer). Cerebras and AMD combine AMD Helios racks with Cerebras systems and claim 5x throughput while keeping Cerebras’s speed (CBRS Q2 FY26 call). Cerebras distinguishes speed, measured in tokens per second per user and critical for experience, from throughput, measured in total tokens across users and critical for economics. GPUs deliver high throughput only at low per-user speed, it says (same).

Lever 6: Scale and utilisation

Large fleets serve many workloads at once, so they batch requests and keep hardware busy. Microsoft calls its fleet “fungible” and reallocates capacity to wherever demand is, monetising efficiency gains “in the quarter” because demand exceeds supply (MSFT Q2 and Q4 FY26 calls). Amazon notes that customers want inference close to their data, and more of that data sits in AWS than anywhere else (AMZN Q1 and Q2 FY26 calls).

The result: collapsing prices at fixed capability

  • Epoch AI (March 2025) found the price of reaching GPT-4-level performance on PhD-level science questions fell ~40x a year, with declines ranging from 9x to 900x a year across benchmarks web.
  • A September 2026 Epoch report, as described secondhand, estimates the cost of a fixed performance level fell ~47% a quarter (~13x a year) since 2023. It also finds the decline is fastest just after a capability first appears, around 66% a quarter at first and roughly halving after two years web; original not checked.
  • An independent study puts the decline at ~5x to 10x a year for frontier models, with ~3x a year attributable to algorithmic progress once hardware and competition are removed web.

Interpretation. Whatever the exact rate, a capability that costs $X today will cost a small fraction of $X in a year. That is the central fact for anyone selling or financing inference.

How the loops interlock: the Jevons effect

If token costs fall 10x or more a year, why are GPU rents rising?

  • NVIDIA: “The price of renting an H100 has risen 20% year to date” (NVDA Q1 FY27 call, May 2026). SemiAnalysis’s index shows one-year H100 contracts rising ~40% between October 2025 and March 2026 web. A TrendForce report puts the rise in China at 20% to 30% web.
  • Microsoft: “demand continues to exceed available supply… you can even see it… in some of the pricing that’s occurring in the spot market” (MSFT Q4 FY26 call).
  • Amazon: even at roughly $200 billion of 2026 capex, “we will still not have enough capacity to meet all the demand we have in 2026”, likely in 2027 too, and “the demand we already have for 2028 is striking” (AMZN Q2 FY26 call).

Usage is compounding:

Metric Data Source
Google first-party model API tokens/min 10B (Q4 FY25) to 16B (Q1 FY26) to ~22B (Q2 FY26) GOOG calls
Google Cloud customers >1T tokens in 12 months 330 (Q1 FY26) to ~500 (Q2 FY26) GOOG calls
Amazon Bedrock Processed more tokens in Q1 2026 than all prior years combined; customer spend +170% q/q AMZN Q1 FY26 call
Microsoft Foundry customers at 1T-token annualised run rate Up 4x year over year MSFT Q4 FY26 call
NVIDIA-served inference providers (e.g. Fireworks, Together) “Growing… 100 times in the last year” NVDA GTC 2026 keynote

The mechanism (Interpretation). Lower cost per token is spent three ways:

  1. More tokens per task. Reasoning and agentic workflows use many times more tokens per job than a chat answer, so cheaper tokens let the same budget buy more thinking.
  2. New tasks become economic. Amazon expects the “lion’s share” of AI demand is “still yet to come… as inference continues to get less expensive” (AMZN Q4 FY25 call).
  3. The frontier price point holds roughly steady. Mid-2026 list prices for flagship models clustered around $5 per million input tokens, and one flagship was priced above its predecessor web; secondhand pricing sites, verify on official pages. Customers buy the newest capability at a similar price, not the old capability at a lower one.

So the cost flywheel does not deflate total spend. It deflates the price of any given capability, while spend migrates to the frontier and to more volume. That holds as long as capability gains keep opening new uses. The key risk to the whole system is a plateau in useful capability, at which point the cost flywheel would turn into straightforward price deflation.

Implications for frontier labs

The treadmill economics

Interpretation, supported by the evidence above. A frontier lab earns its best margins on its newest model, while it alone offers that capability. Within roughly a year, falling costs and distillation bring cheaper substitutes, often open-weight. To keep its margin, the lab must keep releasing new frontier models, which means more compute for training and RL, and more inference capacity for heavier usage. The performance flywheel is a race to stay ahead of the cost flywheel’s commoditisation.

Margins: improving, but far from software-like, and the figures conflict

All lab figures below are web, secondhand and unaudited:

Lab Reported metric Note
OpenAI Gross margin ~33% in 2025 (40% in 2024), as inference costs grew ~4x Includes free users (Sacra; forum citing reports)
OpenAI “Compute margin” on paying users ~70% in Oct 2025, up from 52% at end-2024 The Information; excludes free-tier costs; OpenAI did not confirm
OpenAI Projected ~$278B cumulative negative free cash flow 2026 to 2030 FT as cited by Sacra
Anthropic Gross margin estimates range from ~40% to ~60% in 2026; one report says inference ran 23% over plan Definitions differ (revenue share to cloud partners, training cost)
Anthropic Run-rate revenue reported at $14B (Feb 2026) rising to tens of billions by mid-2026 Run-rate, not booked revenue

What the numbers say together. Unit economics on paid usage are improving fast as the cost flywheel works, which shows in the rising compute margin. But total economics are dominated by the next round of compute, for training, RL and capacity ahead of demand, so free cash flow stays deeply negative. The cost flywheel helps gross margin. The performance race consumes it.

Compute and financing are the binding constraint

NVIDIA stated it plainly: frontier labs “are growing faster than what their balance sheets and credit profiles can support… their growth is not limited by their technology or customer demand. It is limited by compute” (NVDA Q2 FY27 call). In response, NVIDIA:

  • has invested “nearly $50 billion” in frontier labs;
  • set up financing platforms with Apollo, BlackRock, Blackstone, Brookfield, Goldman Sachs and KKR to raise over $500 billion of third-party capital;
  • secured a 4.25 GW site to be used by OpenAI, whose commitments it puts at ~12 GW of NVIDIA compute;
  • provides “selective credit enhancement” for ~2 GW for another lab, and acknowledged “some will call this circular financing” (all NVDA Q2 FY27 call).

Broadcom describes the same thing from the custom-chip side: “empowering two of our most strategic customers, the leading AI labs, to bridge the gap between their current cash flow and the significant upfront investments” (AVGO Q3 FY26 call). Meta’s capital raising shows that even a cash-rich hyperscaler is adding debt and partnerships such as BlackRock to fund the build-out (META Q2 FY26 call).

Interpretation. The suppliers of compute are underwriting the demand for compute. This is the clearest fragility in the system. If lab revenue growth slows while compute obligations are fixed, the stress lands on the lenders and on the suppliers’ receivables and investments, not only on the labs.

Strategic responses visible in the evidence

  • Own the product, not just the model. Coding agents, enterprise agent platforms and consumer apps give a proprietary feedback loop (performance flywheel) and pricing power. Cerebras notes that “no one says, ‘I’m happy with slow tokens’ when coding” (CBRS Q2 FY26 call).
  • Get your own chip. OpenAI’s “Jalapeño” with Broadcom (1.3 GW planned for 2027, over 5 GW in 2028), and Anthropic set to become Broadcom’s largest XPU customer in 2027 (AVGO Q3 FY26 call). Anthropic and OpenAI have also made multi-gigawatt Trainium commitments (AMZN Q1 and Q2 FY26 calls).
  • Multi-source compute. OpenAI runs on NVIDIA (~12 GW committed), Broadcom XPUs, Cerebras (fast inference), and AWS (NVDA Q2 FY27, AVGO Q3 FY26, CBRS Q2 FY26, AMZN Q1 FY26 calls). The labs are using the cost flywheel’s competition among suppliers to bargain.
  • Distribute through every cloud. OpenAI models are now on AWS Bedrock (AMZN Q1 FY26 call), and OpenAI and Anthropic models are on Microsoft Foundry (MSFT Q4 FY26 call).

The structurally advantaged lab: the vertically integrated one

Google is the only player with the model, the chip, the cloud and mass distribution (Search, YouTube, Android). It reports the deepest serving-cost declines (78% in 2025), claims “2x better than anybody else in the market on inference per dollar” for Gemini Flash (GOOG Goldman Sachs conference, Sept 2026), and now sells TPUs into other companies’ data centres (GOOG Q1 and Q2 FY26 calls). Google says its first priority for TPU allocation is “what we need to compete at the frontier” (GOOG Q2 FY26 call). Interpretation: when compute is scarce, owning the chip supply is a competitive weapon, not just a cost saving.

Implications for AI compute economics

The token factory equation

Interpretation, using NVIDIA’s own framing:

Factory revenue = power (MW) x tokens per watt x utilisation x price per token

With power fixed by the site, the only levers are tokens per watt, utilisation and price per token. NVIDIA’s argument follows: a 1 GW factory costs about $40 billion even before the computers, amortised over 15 years (NVDA GTC 2026 keynote). So the computer that produces the most tokens per watt maximises revenue even if its sticker price is higher, and “customers would prefer to buy our next generation product at a higher price than our current generation product at a lower price” (NVDA GTC 2026 Analyst Q&A). This underpins NVIDIA’s gross margins.

The counterpoint. Custom silicon competes on tokens per dollar of total cost of ownership, not per watt alone. Where power is less binding, or the workload is narrow and known, a cheaper chip can win. Microsoft states its metric as “tokens per watt per dollar” (MSFT Q2 FY26 call), which blends both.

Capex at unprecedented scale

Company 2026 capex guide Source
Alphabet $195B to $205B, raised from $180B to $190B; 2027 to “increase significantly” GOOG Q2 FY26 call
Microsoft ~$190B calendar 2026 (incl. ~$25B higher component pricing); ~$175B after lease reclassification MSFT Q3 and Q4 FY26 calls
Amazon Above the earlier ~$200B estimate, partly due to memory prices AMZN Q2 FY26 call
Top-5 hyperscalers ~$800B in 2026, ~$1.3T in 2027 (NVIDIA citing industry estimates) NVDA Q2 FY27 call

Payback, useful lives and the obsolescence question

What management says:

  • Amazon: servers and networking break even in “a little less than three years”; useful life 5 to 6 years; most AI capacity contracted for at least five years; data centres last 30+ years and host 5 to 6 generations of servers (AMZN Q2 FY26 call).
  • Microsoft: most GPUs bought are “already contracted for most of their useful life”; servers depreciated over six years (MSFT Q2 FY26 call). From FY27 it extended data-centre and office building lives from 15 to 25 years, which also shifts future leases from finance to operating leases and lowers reported capex (MSFT Q4 FY26 call).

The tension (Interpretation). If Vera Rubin really produces tokens at 1/35th the cost of GB300, which is NVIDIA’s claim, then a GB300 bought today faces a competitor-produced token price far below its own cost well within six years. Two things have so far kept older hardware valuable:

  1. Scarcity. While demand exceeds supply, every working GPU earns. Rising H100 rents prove this.
  2. Contracts. Capacity sold for its useful life moves the obsolescence risk to the buyer, often a lab, and so back into the financing chain described above.

If supply catches up with demand, the cost flywheel’s other face appears: older fleets reprice sharply, and depreciation schedules set in a scarce market may prove too long. This is the single most important thing to watch in the compute economy.

Rising input costs work against the flywheel. Microsoft attributes ~$25 billion of 2026 capex to higher component pricing (MSFT Q3 FY26 call), Meta raised its capex forecast mostly due to memory pricing (META Q1 FY26 call), and Broadcom says rising memory content is diluting its gross margin (AVGO Q3 FY26 call). HBM and memory are where the cost flywheel’s hardware savings partly leak to suppliers (see the library’s Memory Semiconductors work for orientation).

Implications for the competitive landscape

Where each player sits on the two flywheels

Player type Examples Cost flywheel Performance flywheel Main risk
Merchant GPU platform NVIDIA Drives it through co-design and annual cadence; captures value through tokens per watt pricing Indirect, by enabling RL and agentic inference on one fungible platform Custom silicon share gains; financing exposure to labs
Custom silicon designers Broadcom (TPU, MTIA, OpenAI, Anthropic XPUs), MediaTek Sell “half the cost” economics to the largest buyers None directly Customer concentration; lower gross margin from memory content
Vertically integrated hyperscaler Google Strongest: own chip, model, cloud and distribution Strong: owned products (Search, Gemini app, Workspace) give feedback Capex scale; Search disruption
Hyperscaler with own chips, multi-model Amazon, Microsoft Strong: Trainium/Maia plus scale and utilisation Moderate: Copilot and own models (MAI); depends on partner labs Partner labs’ bargaining power; capex payback
Hyperscaler, internal use Meta Strong internally (MTIA, distillation, routing) Strong in ads (adaptive ranking) Consumer AI monetisation still early
Frontier labs OpenAI, Anthropic Benefit as buyers; building own chips Strongest where they own products (coding agents) Treadmill economics; financing; distillation leakage
Speed specialists Cerebras; Groq (inside NVIDIA) Own a premium tier (fast tokens) Speed improves agent performance Customer concentration (OpenAI); incumbents adding speed tiers
Open-weight model makers DeepSeek, Qwen, Kimi, others Push commodity prices down Benefit from distillation Thin monetisation

Five landscape shifts the flywheels are causing

  1. From one market to tiers. Jensen Huang’s “token segments” (NVDA GTC Analyst Q&A) and the Groq and Cerebras speed tiers show inference splitting into a commodity tier (cheap, high throughput, often open-weight), a frontier tier (newest models) and a speed tier (fast frontier tokens at a premium). Pricing power exists only in the last two.
  2. From general-purpose to specialised hardware. Cerebras: “as inference grows and matures, opportunities for specialization emerge” (CBRS Q2 FY26 call). Broadcom’s six XPU customers and the three-generation MTIA roadmap (AVGO Q3 FY26 call) show the biggest buyers designing around known workloads. NVIDIA’s answer has been to absorb specialisation, with Groq LPX for latency and Vera CPUs for agents, and to argue that fungibility across all phases is worth more than point efficiency (NVDA Q2 FY27 call).
  3. From “GPUs” to full systems. Rack-scale design, networking (Broadcom’s Tomahawk Ultra moving scale-up onto Ethernet), memory and power delivery now drive tokens per watt as much as the chip does (AVGO Q3 FY26 call; NVDA GTC 2026 keynote).
  4. From token volume to token value. Open models win volume while closed models keep the dollars. One gateway reports open models handle nearly a third of tokens but less than 4% of spend web. A token share number on its own says little about who earns the money.
  5. Suppliers becoming financiers. NVIDIA’s equity and credit support and Broadcom’s lab financing turn chip vendors into part-lenders to their own customers (NVDA Q2 FY27 call; AVGO Q3 FY26 call). This tightens the link between lab success and supplier results.

Read-across to companies covered in the library

Orientation only; see each company’s own files.

  • NVDA: The cost flywheel is NVIDIA’s product roadmap, and the Jevons effect is its demand engine. The risks are custom silicon at the largest buyers and its growing balance-sheet exposure to labs.
  • AVGO, MRVL: Direct beneficiaries of inference specialisation. Watch customer concentration and memory-driven margin dilution.
  • GOOG: Most complete flywheel owner. TPU external sales are a new way to monetise the cost advantage.
  • AMZN, MSFT: Trainium and Maia are margin levers. The multi-model clouds (Bedrock, Foundry) monetise volume regardless of which lab wins.
  • META: An inference-ROI story inside ads, and a large MTIA buyer.
  • CBRS: A pure play on the speed tier, with heavy OpenAI concentration.
  • TSM, MU, SKHY, SNDK, ANET, CRDO, ALAB, COHR, LITE, CIEN: Suppliers whose volumes ride the Jevons effect. Memory suppliers are capturing part of the cost flywheel’s savings.
  • NET, AKAM: Edge inference is a possible beneficiary as smaller distilled models allow inference closer to users. Not examined here.

Signals to monitor

Signal Why it matters Where to find it
GPU rental prices (H100, B200) Rising means Jevons dominates; falling sharply means supply has caught up and older fleets reprice NVDA and MSFT commentary; SemiAnalysis index web
Disclosed token volumes Tests whether volume growth keeps outrunning price declines GOOG, MSFT, AMZN calls
Frontier list prices per million tokens Flat means value migrates to the frontier; falling means commoditisation is reaching the top tier Lab pricing pages web
Lab gross and compute margins Shows whether cost gains reach labs or are competed away Press reports; IPO filings if they come
Custom silicon share of hyperscaler capex Shift of value from NVIDIA to Broadcom and in-house chips AVGO, GOOG, AMZN, MSFT, META calls
Depreciation policy changes A shortening of server lives would signal real obsolescence 10-K accounting notes
HBM and memory pricing How much of the cost flywheel leaks to memory suppliers MU, SKHY calls; hyperscaler capex commentary
Vendor financing of labs Size and terms of credit support; any stress NVDA, AVGO filings and calls
Open-weight share of tokens and of spend Pressure on commodity-tier pricing OpenRouter and gateway data web
Capability progress per dollar A plateau would break the Jevons loop Epoch AI and benchmarks web

Open questions and uncertainties

  • How long does Jevons hold? The demand evidence is strong for 2026 to 2028 (Amazon, Microsoft, Google), but it rests on capability continuing to unlock new uses. There is no filing evidence on what happens at a plateau.
  • The real margin structure of the labs. All figures are secondhand and inconsistent in definition. No audited financials exist in the library.
  • Inference share of compute. Estimates for 2026 range from ~60% (AMD CEO, as reported) to ~two-thirds (Deloitte, as reported) web. The training/inference line is blurring because of RL.
  • Vendor price-performance claims. NVIDIA, Google, Amazon, Microsoft and Broadcom each claim large advantages, often against different baselines. They cannot all be best on the same workload. Independent benchmarks (SemiAnalysis InferenceX, MLPerf) are cited by vendors but were not read directly here.
  • Persistence of the data flywheel. Evidence that user interaction data materially improves frontier models is strongest in coding and ads, and weak elsewhere.

Sources

Library filings and transcripts (source of truth)

All paths are under Equity/.

  • NVIDIA: NVDA_NVIDIA_CORP/sources/Transcripts/ files 2026-03-04 (Morgan Stanley TMT), 2026-03-16 (GTC 2026 keynote), 2026-03-17 (GTC Financial Analyst Q&A), 2026-05-20 (Q1 FY27 call), 2026-06-01 (GTC Taipei keynote), 2026-06-24 (AGM), 2026-08-26 (Q2 FY27 call)
  • Alphabet: GOOG_Alphabet_Inc/sources/Transcripts/ files 2026-02-04 (Q4 FY25 call), 2026-04-29 (Q1 FY26 call), 2026-07-22 (Q2 FY26 call), 2026-09-08 (Goldman Sachs Communacopia)
  • Microsoft: MSFT_MICROSOFT_CORP/sources/Transcripts/ files 2026-01-28 (Q2 FY26 call), 2026-04-29 (Q3 FY26 call), 2026-07-29 (Q4 FY26 call)
  • Amazon: AMZN_AMAZON_COM_INC/sources/Transcripts/ files 2026-02-05 (Q4 FY25 call), 2026-04-29 (Q1 FY26 call), 2026-07-30 (Q2 FY26 call)
  • Meta: META_Meta_Platforms_Inc/sources/Transcripts/ files 2026-04-29 (Q1 FY26 call), 2026-07-29 (Q2 FY26 call)
  • Broadcom: AVGO_Broadcom_Inc./sources/Transcripts/2026-09-02_AVGO_Q3_FY2026_Earnings_Call_Transcript.md
  • Cerebras: CBRS_Cerebras_Systems_Inc./sources/Transcripts/2026-08-12_CBRS_Q2_FY2026_Earnings_Call_Transcript.md

Outside sources/ (web, labelled web in the text)

Prior work in Industry/ (Memory Semiconductors, Analog Semiconductors, Cybersecurity) was used for orientation only and is not cited as evidence.