Google splits its eighth-generation TPU in two, with a training chip and an inference chip

Google has introduced the eighth generation of its custom Tensor Processing Unit at Google Cloud Next, and this generation comes with two purpose-built architectures: TPU 8t for training and TPU 8i for inference. Google says both will be generally available later this year, and the pitch is simple: more AI performance per dollar and per watt.

This is a big, gilded, two-part bet on where the money in AI is going. Training builds the models. Inference is the bill that arrives every time somebody uses one. Google wants a purpose-built chip for each.

Two chips, two jobs

Google says both were designed in partnership with Google DeepMind. Both run on its own Axion Arm-based CPU hosts, a first for the TPU line, which Google says lets it optimize the full system and not just the chip.

The training chip, TPU 8t, is the heavyweight. According to Google, a single superpod scales to 9,600 chips and two petabytes of shared high-bandwidth memory, delivering 121 ExaFlops of compute. Pair it with the new Virgo Network, JAX and Google's Pathways software, and the company says TPU 8t can scale up to a million chips in a single logical cluster. A million chips. In one cluster. Breathe.

Google also says TPU 8t is engineered to target over 97% goodput, its term for useful, productive compute time, through reliability features such as real-time telemetry, automatic rerouting around faulty links and optical circuit switching. The Google Cloud technical deep dive lists 216 GB of HBM, 128 MB of on-chip SRAM and 12.6 peak FP4 petaflops per chip.

The inference chip, TPU 8i, is built around memory. Google says it pairs 288 GB of high-bandwidth memory with 384 MB of on-chip SRAM, three times the previous generation, to keep a model's active working set on the chip. The deep dive lists 10.1 peak FP4 petaflops per chip.

The numbers Google is leading with

The headline for the money crowd is price-performance. Google says TPU 8i delivers 80% better performance-per-dollar than the previous generation, which it describes as enabling businesses to serve nearly twice the customer volume at the same cost. For training, the technical deep dive claims up to 2.7x performance-per-dollar over Ironwood, the seventh-generation TPU. Both chips, Google says, offer up to two times better performance-per-watt than Ironwood.

One wrinkle: CNBC reported a slightly different training figure, saying Google described 2.8 times the performance of Ironwood for the same price. The company's own deep dive says 2.7x. Same ballpark, different scorecards.

The inference engineering is where the details get fun:

• TPU 8i doubles interconnect bandwidth to 19.2 Tb/s, aimed at Mixture of Experts models.
• A new Boardfly architecture cuts maximum network diameter by more than 50%. In a 1,024-chip pod, Google says it drops from 16 hops to seven, a 56% reduction.
• An on-chip Collectives Acceleration Engine reduces on-chip latency by up to 5x, according to Google.

Alphabet CEO Sundar Pichai framed the inference chip in agent terms. The architecture is designed "to deliver the massive throughput and low latency needed to concurrently run millions of agents cost-effectively," he wrote in a blog post.

The Nvidia of it all

Now the power politics. Google's TPU push isn't stopping it from offering Nvidia hardware. Google announced A5X bare metal instances powered by Nvidia Vera Rubin NVL72 at the same event, and says its Virgo Network will support up to 80,000 Nvidia GPUs in a single data center and up to 960,000 across multiple sites. Google is selling the guest chips and the house chips at the same party.

CNBC put the competitive picture bluntly: "None of the tech giants are displacing Nvidia, and Google isn't even comparing the performance of its new chips with those from the AI chip leader." Google's materials compare the new TPUs with its own Ironwood, not with Nvidia's accelerators.

On customers, Google names Citadel Securities as an organization choosing TPUs for its AI workloads. The sources give no other external customer for the new generation.

What to watch

Start with pricing. The announcements give no price structure or detailed availability schedule for Google Cloud customers deploying TPU 8t and TPU 8i. Efficiency claims are lovely. Invoices are where chips get judged.

Then there's the Nvidia comparison absent from Google's announcements: how TPU 8t and TPU 8i stack up head-to-head against Nvidia's current and upcoming accelerators, including Vera Rubin.

Software matters too. Google says native PyTorch support for TPUs is now in preview, which is the kind of door that has to open if developers are going to walk through it.

And keep an ear on the user mood. One Reddit user complained that despite Google's claims of cost-reducing TPUs, Gemini quotas for paid users keep getting lower. One post is one post, but if cheaper silicon is the story, customers will want to feel it.

Two chips, a million-chip ambition and a very large rival standing in the doorway. Gorgeous.