Google’s New TPUs: Two Chips for Two Very Different Jobs

Everyone’s buying Nvidia. Microsoft, Meta, OpenAI — they’re all fighting over the same H100s and B200s like it’s Black Friday for server racks. Google? It’s been doing its own thing for years now with Tensor Processing Units, and the latest generation shows they’re not just iterating — they’re rethinking the whole approach.

Instead of one chip that tries to do everything, Google split the eighth-gen TPU into two distinct flavors. The TPU8t is built for training, the TPU8i for inference. That’s not just a marketing gimmick. The workloads are fundamentally different. Training is about shoving massive datasets through a model over weeks, optimizing weights and biases. Inference is about responding in real-time, often with latency measured in milliseconds.

Google’s argument is that the “agent era” — where AI doesn’t just answer questions but takes actions, runs workflows, and interacts with tools — demands hardware that’s purpose-built for each phase. I’ve seen this pattern before. Specialization usually wins when the volume is high enough. And Google’s volume is definitely high enough.

The TPU8t targets that brutal training grind. Google claims it can cut training time for frontier models from months to weeks. That’s a bold claim, but they’ve got the engineering chops to back it up. The seventh-gen Ironwood was already impressive, and the jump to the eighth gen seems more architectural than just a clock speed bump.

The TPU8i, on the other hand, is all about serving models efficiently. Lower power, higher throughput per watt, optimized for the kind of real-time inference that agentic systems need. If you’re running a fleet of AI agents that need to respond instantly, you don’t want a training monster sipping power in idle. You want lean, mean inference silicon.

What I find interesting is that Google is pushing this narrative of the “agent era” to justify the split. It’s a bit of marketing spin, sure, but it’s also true that the way we use AI is shifting. Simple chatbots are giving way to autonomous agents that browse the web, execute code, and manage workflows. Those agents have different hardware demands than a model that just generates text.

Of course, the big question is whether this matters to anyone outside of Google Cloud. Most developers will never touch a TPU directly. They’ll use Vertex AI or some other managed service. But for Google, this is about keeping their infrastructure competitive with Nvidia’s dominance. And frankly, having your own custom silicon gives you options that Nvidia customers don’t have.

Will the TPU8t and TPU8i actually deliver on the promises? We’ll see when benchmarks drop. But the strategy is sound. Specialized hardware for specialized workloads. It’s worked for Google before.