Nvidia has quietly doubled down on a long‑predicted pivot in the AI hardware market. The company paid roughly $20 billion for the intellectual property of Groq, a startup that built a lean, inference‑first architecture, and absorbed much of its core team. The deal underlines a strategic judgement: the era of vast, costly training clusters is giving way to mass deployment of inference engines that must be far cheaper and more efficient to run at scale.
Chinese chipmakers are racing to mirror that logic. Sunrise (曦望), a spin‑out from SenseTime, has unveiled a GPGPU chip — the Qiwang S3 — specifically tuned for large‑model inference. The company says S3 cuts per‑token inference costs by about 90% versus its predecessor and has launched a “one million tokens for one fen” (0.01 RMB) cost target in partnership with several AI firms. Such claims, if realised, would materially lower the operational cost of running conversational agents, embodied robots and other always‑on AI services.
Industry voices in China and the United States are converging on a similar forecast: inference demand is set to explode. Executives interviewed by a Chinese financial daily predict that by 2030 inference will account for as much as 80% of total AI compute. Their argument is simple: once models are trained, they must be run constantly across billions of endpoints, and next‑generation “intelligent agents” will consume many times the compute of a single large language model instance.
Groq and a handful of other Western startups — including Etched.ai — have staked their strategy on what they call “extreme inference”: purpose‑built silicon that jettisons the flexibility required for training in favour of low‑latency, high‑throughput, power‑efficient execution. Groq’s founders, veterans of Google’s TPU effort, claim LPU (language processing unit) performance far exceeding that of mainstream GPUs for inference, and at a much lower cost per operation. Nvidia’s move to licence that technology and recruit the team suggests it sees a threat — or an opportunity — too big to ignore.
The changing economics matter because inference is where many commercial AI use cases live. Consumer apps, customer service bots, and increasingly robots and other physical devices need cheap, low‑power inference to scale. Chinese internet giants are already preparing for mass demand: promotional campaigns during the Lunar New Year, including cash incentives, are aimed at pushing consumer adoption of AI services and accelerating usage metrics that justify continued investment in compute and chip capacity.
This transition opens a window for domestic chipmakers. Training GPUs are a closed game with high engineering and capital barriers, which has helped Nvidia dominate that end of the market. By contrast, the inference market prizes energy efficiency, latency and price‑performance — attributes where specialist designs and regional partnerships can carve out lucrative niches. Chinese executives envision multiple homegrown GPU firms each worth hundreds of billions of RMB serving local cloud, telco and enterprise customers, even if none individually reaches Nvidia’s colossal market cap.
But the shift is not riskless. Some analysts warn of overheated capital markets and valuation froth around AI hardware. Historical bubbles in tech finance are invoked as a reminder that excitement can outpace actual revenue and user engagement. Moreover, trade‑offs between specialization and software ecosystem compatibility are real: the fastest inference silicon matters less if software stacks, toolchains and customer integrations lag behind or lock buyers into fragmented islands of hardware.
Still, the balance of incentives has changed. Training remains vital, but the commercial prize — and the persistent, global demand for compute — increasingly resides in inference. Nvidia’s purchase is both a defensive hedging of its training franchise and an offensive move to own the stack across the lifecycle of models. For governments, cloud operators and enterprises, the new battleground will be energy‑efficient scale, software portability and the economics of token pricing as AI becomes ever more embedded in daily life.