Key Takeaways
| What you’ll learn in this guide Eighth generation TPUs (Ironwood) are Google’s first chip designed primarily for inference at planetary scale. A single Ironwood pod connects 9,216 chips through optical circuit switching for trillion-parameter models. Performance per watt roughly doubles compared with the previous Trillium generation. Liquid cooling and Google’s 24/7 carbon-free energy goals make it one of the greenest AI accelerators on the market.Available on Google Cloud through familiar APIs Vertex AI, GKE, and JAX/PyTorch frameworks. Early adopters in healthcare, finance, robotics and media report 30–50% lower serving cost per token. |
Why Eighth Generation TPUs Matter Right Now
From being an experimental technique in academia to the type of computing task behind the operations of today’s data centers, generative AI relies on hundreds-of-billion-parameter models answering customers’ questions, drafting contracts, diagnosing medical images, and helping robots move around in warehouses. Doing all of that efficiently requires silicon that is purpose-built for this task, which turns out to be exactly what Google’s hardware engineering team had in mind while designing its eighth generation TPUs.
Whereas previous generations of TPUs focused primarily on the matrix multiplications involved in deep learning model training, the new chip named Ironwood after one of the world’s most resilient types of trees is optimized for inference first. It assumes that thousands of users will connect to the same model, that latency needs to be counted in milliseconds, and that every saved watt of energy is potentially multiplied by a factor of a million. The result is a product that, effectively, makes large language models, multi-modal agents, and embodied AI applications affordable.
This guide provides detailed insight into the architecture, performance benchmarks, and commercial strategies around these next-generation TPUs on Google Cloud. Sustainability considerations are also covered along with customer examples. By the time
Figure 2 Conceptual layout of an Ironwood die: matrix cores, vector units, SparseCore, and side-mounted HBM3e stacks.
Inside the Ironwood Architecture
To fully understand how the new Ironwood design works, it is good to recall some things that characterize all TPUs. First, there is a large number of matrix multiply units called MXU by Google. This unit processes dense tensors from the neural network algorithms at significantly lower power consumption than any regular GPU. These are supported by vector processing units and other hardware.
Compute fabric and SparseCore
- Ironwood boosts each of those blocks as well. Its matrix engines are broader, its vector engine is better at dealing with the activation functions now common in contemporary transformers, and there’s a new SparseCore block that speeds up embeddings in recommender systems and retrieval-augmented generation tasks. Collectively, these blocks enable a single chip to take on tasks that otherwise would have needed several accelerators connected with hacks to the software.
- Wider matrix blocks optimized for bfloat16, FP8, and INT8 inference pipelines.
- Improved vector engine for activations like SwiGLU, GELU, and RMSNorm.
- New SparseCore engine for billion-entry embedding tables in ads, recommendations, and RAG.
- Hundreds of gigabytes of per-chip HBM3e capacity – enough for most production
Memory bandwidth that matches the math
The typical constraint on inference isn’t floating point operations per second but rather the rate at which weights and activations can be delivered to the processing engines. Ironwood couples its matrix engine with HBM3e memory stacks capable of providing bandwidth in terabytes per second per processor. This results in a 70 billion parameter model fitting comfortably within high-speed memory without the need for costly PCIe trips, allowing users to enjoy their expected low-latency experiences when generating the first token.
Figure 3 Indicative generational throughput. Ironwood delivers a step-change over Trillium and earlier TPUs.
Performance: A Generational Leap, Not a Polish
The figures released by Google put the performance of Ironwood around five times better than its predecessor Trillium, with energy efficiency doubling as well. This has been confirmed through independent tests conducted by their partners like Anthropic and Salesforce.
From independent sources like Forbes to engineering teams at key providers of models, it is evident that there has been a quantum leap in economics of tokens per cost.
What changes for ML engineers
- Throughput per unit cost: less chips required to reach the same QPS goal.
- Reduced tail latency due to increased on-chip SRAM capacity and improved caching techniques.
- Increased window sizes can now be used as opposed to merely being shown.
- The model scale-up is smooth since routing is done on-chip, rather than network-based.
Figure 4 A single Ironwood pod links 9,216 chips with optical circuit switches, behaving like one accelerator to the software layer.
Pod-Scale Computing: 9,216 Chips Behaving Like One
The world’s most powerful accelerators face their own limits when they require too much memory for a single silicon die. That is where Ironwood comes in, using Google’s third-generation optical circuit switching fabric to connect up to 9,216 chips together in one pod, delivering very low latency with high consistency between any pair of endpoints. This is useful because the software layer sees a pod as a giant machine – models are distributed on thousands of chips using just a few lines of code via JAX, PyTorch or TensorFlow.
This is important since models at the forefront are evolving even faster than our hardware capabilities. The trillion-parameter model, multimodal agents capable of reasoning over hours of video, or reinforcement learning agents for robotics can take advantage of treating one pod of accelerators as a single powerful device.
Why optical circuit switching is a quiet revolution
- Changes the physical topology in milliseconds, so that dead nodes don’t kill off jobs.
- Gets rid of the network as a bottleneck for operations such as all-reduce and all-gather.
- Delivers bandwidth that scales linearly with the pod, not with the slowest link.
- Reduces the energy required to shift data between chips – the biggest silent cost of AI training.
Figure 5 Liquid cooling, recycled water and 24/7 carbon-free energy give Ironwood a leading sustainability profile.
Sustainability Built Into the Silicon
Nowadays, the amount of electricity consumed by AI is something that needs the attention of corporate boards. Google has committed to becoming completely dependent on carbon-free power around the clock by 2030, and the eighth generation TPUs is critical in achieving that goal. The doubling of inference performance per watt would enable the completion of twice the number of inferences using the same energy used by the previous generation’s chips.
The efficiency of liquid cooling systems ensures that warehouses consume electricity as efficiently as possible, with PUE values being closer to the minimum level of 1.1. Additionally, many Google data centers are capable of using non-drinking or recycled water to cool their servers.
Sustainability checklist for AI buyers
- Make sure that the performance-per-watt of the chip is what it says it is.
- Request the carbon intensity from the cloud provider in the region where you have deployed the system.
- Opt for regions which use hourly carbon free energy balancing.
- Do not oversize your server fleet.
Figure 6 — Industries deploying eighth generation TPUs in production.
Real-World Use Cases Already in Production
Hardware only matters in proportion to the value it unlocks. The teams that have moved earliest to Ironwood point to four families of workload where the chip changes the conversation.
Healthcare and life sciences
Prediction of protein structures, triage of medical images, and summarization of clinical notes each involve large transformer models that were earlier forced to trade-off accuracy and latency. With additional memory available per chip, healthcare facilities are now able to use one accurate model for radiology tasks instead of several low-accuracy models.
Financial services
Fraud scoring and risk modeling in real-time should take no more than 100 milliseconds for response time, even during periods of market stress. The benefit of using pod connectivity together with HBM3e capability at the chip level allows banks to build larger ensemble models per rack.
Robotics and embodied AI
Training for the vision-language-action systems driving warehouse and surgical robots involves simulation rollouts predominantly. Training for Ironwood pods benefits greatly from such an approach, reducing training time from weeks to days. Further information on this area can be found in our accompanying article on physical AI in our internal resource portal.
Generative media pipelines
The studios that create their content via AI-powered video and audio generation require predictable costs per minute. The increased efficiency per watt leads to higher profitability of subscription-based creative applications.
Authority, Trust, and the Companies Behind the Numbers
Google Cloud – the division delivering eighth-gen TPUs – operates at hyperscale with operations in over 40 cloud regions and serves over 9 million developers globally and has been a Leader in the Gartner Magic Quadrant for Strategic Cloud Platform Services for many years running. Google Cloud’s customer support and reliability have received top ratings from independent third-party sources like G2, where the Google Cloud Platform earned 4.5 stars out of 5 based on more than 800 reviews.
For an authoritative technical resource, the official Google Cloud TPU documentation covers configuration options and limitations along with framework support. The industry experts at MLCommons, which publishes MLPerf benchmark tests, provide an unbiased comparison between Ironwood and competing accelerators. More information is available here.
Company snapshot
- Company: Google LLC (parent company – Alphabet Inc.)
- Cloud division: Google Cloud (cloud.google.com)
- X/Twitter account: @googlecloud (over 1.1 million followers)
- LinkedIn profile: Google Cloud (over 4.7 million followers)
- Customer rating: 4.5/5 stars based on over 80
How to Pilot Eighth Generation TPUs on Google Cloud
Adopting new silicon does not have to be a forklift exercise. Most teams can stand up a meaningful pilot in a single sprint by following a simple sequence.
A pragmatic four-step pilot
- Step 1 – Choose an application where inference costs or latencies are currently a limiting factor.
- Step 2 – Model containerization using either JAX or PyTorch/XLA; both platforms receive native support from Ironwood.
- Step 3 – Submit a request for allocating a smaller part of Ironwood via the Google Cloud Platform Console and run benchmarks compared to your current infrastructure.
- Step 4 – Evaluate performance based on costs, p95 latencies, and energy used per 1,000 tokens and make an informed decision.
Tooling that smooths the migration
- Model registry, auto-scaling, and online endpoints are all built into Vertex AI.
- GKE, along with the JobSet operator, manages the multi-pod training jobs.
- MaxText and MaxDiffusion provide implementation examples for you to use as a starting point.
- Cloud Monitoring provides you with utilization statistics so that you can properly scale out.
Eighth Generation TPUs vs Other AI Accelerators
Choosing an accelerator is rarely about raw flops alone. The table below distills the trade-offs that matter most when you are sizing a production fleet.
| Dimension | Ironwood (TPU v8) | Trillium (TPU v6e) | Leading GPU |
| Primary workload | Inference at scale | Training & inference | Training & inference |
| Pod size | Up to 9,216 chips | Up to 256 chips | Up to ~576 GPUs |
| Interconnect | Optical circuit switching | ICI mesh | NVLink + InfiniBand |
| Cooling | Liquid | Liquid | Air or liquid |
| Perf/watt vs prior gen | ~2× | ~1.7× | ~1.4× |
| Pricing model | On-demand & committed | On-demand & committed | On-demand, reserved, spot |
Common Misconceptions About TPU v8
“TPUs only work with TensorFlow”
It was true ten years back but not anymore. Currently, JAX is the most used framework for TPU, and with PyTorch XLA, PyTorch users can get close to native access. Practically all models on Hugging Face are supported either with JAX or PyTorch implementation.
“You need a special degree to deploy them”
If you can deploy a containerized model on Kubernetes, you can deploy it on a TPU pod. Vertex AI further abstracts the cluster work for teams that prefer a managed endpoint.
“They are only for hyperscalers”
Single-host TPU VMs are available by the hour, with no minimum commitment. Startups commonly prototype on a single chip and scale up only after the model proves out.
Frequently Asked Questions
What are eighth-generation TPUs?
Ironwood TPUs are Google’s most recent custom application-specific integrated circuits dedicated to artificial intelligence applications. They are designed specifically for inference tasks, which entail deploying trained models for use by customers. However, they can be used for training purposes too. They succeed the Trillium seventh-generation TPUs.
How does Ironwood perform compared to the previous TPU?
The chip has demonstrated roughly five times more throughput in inference tasks while consuming two times less power compared to its predecessor Trillium. These figures might vary depending on many factors such as the architecture of a model being served.
Is Ironwood rental available?
Sure, you can rent an eighth-generation TPU from Google Cloud by the hour in single-host configuration or enjoy committed-use discounts for consistent deployments on virtual machine configurations.
Which frameworks do eighth-generation TPUs support?
All major artificial intelligence frameworks are officially supported, including JAX; PyTorch using the PyTorch/XLA extension; and TensorFlow. All other most common libraries such as Flax, Hugging Face Transformers, vLLM forks, and JAX-based stacks operate seamlessly with minimum modifications.
Eighth generation TPUs are environmentally friendly compared to GPUs, aren’t they?
From the environmental point of view, they perform better compared to
Conclusion: A Defining Moment for AI Infrastructure
The eighth-generation TPUs aren’t simply an incremental update. They’re a bet on a world where the future of AI isn’t about building increasingly large models but about serving increasingly powerful models to billions of users in a way that sustains itself. Ironwood’s architecture, its pod-scale fabric, and its power characteristics are all bets made toward achieving that vision — and early results in healthcare, financial services, robotics, and media would seem to indicate that it’s paying off.
Is generative AI on your roadmap, a retrieval-enhanced assistant, or some other task where serving costs are now a boardroom-level issue? Then you’ll want to put eighth generation TPUs on your list of potential solutions. Start small, measure truthfully, and let the cost-per-token speak for itself.
Want to evaluate Ironwood yourself? Create a single-host TPU VM in Google Cloud, migrate one endpoint, and see what happens. You can’t beat the cost-per-token performance of your own data.
