Semiconductor AI Chips: The Inference Cliff

The thesis the market has yet to price

The center of gravity in semiconductor AI chips will move from training to inference by 2027. This reads as opinion. It is arithmetic.

Training a frontier model happens once per generation. Inference happens billions of times per day, every day, for the product's whole run. The recurring cost dwarfs the one-time cost.

Consensus prices the training moment. The recurring workload sets the real economics, and that workload favors a different kind of chip.

Why the consensus has the frame wrong

Analysts track training FLOPs and quarterly data-center revenue at the dominant GPU vendor. Those numbers describe the present with precision.

The metric that predicts the next regime is tokens served per day. That volume compounds as applications ship, and it rewards silicon tuned for throughput and memory bandwidth rather than peak training performance.

The 90% of analysts who read today correctly read the pace of change wrong. Training demand is lumpy and episodic. Inference demand is continuous, and continuous demand invites custom silicon built to crush cost per token.

The mechanism: why serving favors custom silicon

Training rewards raw flexibility. A model in development changes shape often, so general-purpose hardware earns its premium.

Serving rewards the opposite. A shipped model is fixed, its operations are known, and a chip designed for those exact operations wins on power and price.

This is the causal engine, no correlation dressed as a thesis. When workloads freeze, specialized silicon beats general silicon on cost per output. Every prior compute wave, from video encoding to Bitcoin mining, walked the path from CPU to GPU to ASIC. AI inference follows the same road, and the destination is dedicated accelerators sitting close to fast memory.

The cost curve nobody disputes

Cost curves in silicon obey a pattern. Volume drives learning, learning drives price down, and price collapse pulls demand forward.

Solar offers the reference trajectory: by IEA accounts, module cost fell roughly 90% across the decade from 2010 to 2020. Compute follows the same logic, compounded by architecture gains.

By broad industry accounts, the cost to serve a fixed AI model output dropped by more than an order of magnitude across 2023 to 2025. Three forces drove it: cheaper silicon, smarter model compression, and chips built for the serving job. This is a slope with three data points, so I will call it a trajectory.

The cliff event and its date

Adoption rarely climbs a smooth ramp. It sits flat, then jumps when a cost threshold breaks.

Cliff event: inference spend overtakes training spend inside data-center AI budgets by 2027. At that crossover, the buyer's question changes from "who trains fastest" to "who serves cheapest per token."

That single question reroutes tens of billions in capital. Buyers optimizing for cost per token will favor custom accelerators and high-bandwidth memory over general-purpose training hardware. The vendor that owns training loses its pricing lock the moment serving becomes the dominant line item.

Three categories that change shape by 2028

Here is where the money moves. Three groups reorganize around inference economics.

• Hyperscalers with in-house silicon: Google TPU, Amazon Trainium and Inferentia, Microsoft Maia. Each design cuts dependence on merchant GPUs for serving.
• Memory makers: SK Hynix, Samsung, and Micron ride high-bandwidth memory demand, since inference is bandwidth-bound as much as compute-bound.
• Edge inference: phones, cars, and humanoid robots run models locally, pushing a new class of low-power accelerators into billions of devices.

The dominant GPU vendor keeps the training crown for years. The serving crown is up for grabs, and serving is where the recurring revenue lives.

The common thread: value migrates toward whoever serves a token cheapest, wherever that token gets served. I mapped the parallel migration in models in my note on foundation model commoditization.

My position, and what would change it

My stance is explicit: the dominant GPU vendor's share of AI compute spend will decline as inference commoditizes, even as its absolute revenue keeps rising.

Absolute growth and share erosion coexist. A market can triple in size while its leader's slice shrinks. Consensus confuses the two, reading record revenue as a durable moat.

What would change my mind: a serving architecture where merchant GPUs beat custom ASICs on cost per token at scale, sustained across two product generations. Should that hold, the moat survives and my thesis fails. I watch that ratio each quarter.

The prediction

Here is the falsifiable call. By the end of 2026, at least one major hyperscaler will report that in-house silicon serves the majority of its internal AI inference workload.

Confidence: Medium. This is a market-timing claim, so I hold it below my hardware-trajectory calls. The direction is inevitable. The exact date carries risk.

Kill signal: should the top three cloud providers each disclose that merchant GPUs handle the bulk of inference through 2026, with in-house share flat or falling, the thesis is dead. Watch the disclosure, watch the ratio.

What each decision-maker should do now

For the CTO: reassess any stack that assumes training-class GPUs for serving. Inference-optimized silicon changes your unit economics. Model your cost per token, then compare merchant GPU against dedicated accelerator across a full year.

For venture and growth investors: the impossible-looking bet is custom inference silicon and the software that schedules it. The data supports it before the market does.

For the chief strategy officer: a three-year plan anchored to today's chip hierarchy assumes a world that dissolves at the crossover. For procurement: pause any long GPU contract that locks you into training economics for an inference future. Read my related take on disruption timing before you sign.