NVIDIA and Google Unveil Cost-Effective AI Inference Infrastructure
70At the recent Google Cloud Next conference, Google and NVIDIA revealed their latest hardware plans designed to make AI inference more affordable at large scales. They showcased new A5X bare-metal instances built on NVIDIA Vera Rubin NVL72 rack-scale systems. This joint effort combines hardware and software design to slash inference costs significantly while boosting processing speed and efficiency.
New Hardware Boosts AI Inference Efficiency
The A5X instances are engineered to deliver up to ten times lower inference costs per token compared to previous systems. At the same time, they provide ten times higher token throughput per megawatt of energy used. This means faster AI processing with less power, which is crucial for large-scale AI applications.
Handling thousands of processors requires massive bandwidth to keep everything running smoothly. To address this challenge, Google and NVIDIA paired NVIDIA ConnectX-9 SuperNICs with Google’s Virgo networking technology. This setup enables the system to connect up to 80,000 NVIDIA Rubin GPUs within a single site and as many as 960,000 GPUs across multiple sites. Such scale demands precise workload management to prevent delays and idle time across nearly a million processors.
Scaling AI with Advanced Networking and Management
Managing this enormous infrastructure involves sophisticated techniques to route data efficiently across the processors. Google’s VP and GM of AI and Computing Infrastructure, Mark Lohmeyer, explained that the future of AI depends on running demanding workloads on integrated, AI-optimized systems. Combining Google Cloud’s infrastructure with NVIDIA’s hardware and software is meant to give users the flexibility to train, optimize, and deploy a wide range of AI models—from open-source and frontier models to more complex physical and agentic AI tasks.
This collaboration aims to improve how AI workloads are handled in terms of performance, cost, and sustainability. By optimizing the hardware and network architecture, the companies are setting the stage for more accessible and scalable AI deployments in enterprise environments.
Addressing Data Privacy and Regulatory Challenges
Beyond raw processing power, data governance remains a key issue for businesses, especially in sectors like finance and healthcare. These industries face strict rules about data sovereignty and privacy, which can slow down or block AI projects. To help organizations meet these regulations, Google is introducing its Gemini models on NVIDIA Blackwell and Blackwell Ultra GPUs, now available in preview on Google Distributed Cloud.
This setup allows companies to keep their sensitive data and advanced AI models within their own controlled environments. It uses NVIDIA Confidential Computing, a hardware-based security feature that encrypts data during training and fine-tuning. This encryption prevents unauthorized access, even from cloud operators, ensuring proprietary information stays private.
Google is also testing Confidential G4 VMs with NVIDIA RTX PRO 6000 Blackwell GPUs in a public cloud setting. These VMs incorporate cryptographic protections similar to those used in on-premise environments, making high-performance AI hardware accessible without compromising data privacy. This move helps regulated industries adopt powerful AI tools while maintaining compliance and security standards.
Overall, these developments mark a significant step toward making AI inference cheaper, faster, and more secure, paving the way for broader adoption across various sectors and use cases.
What do you think?
It is nice to know your opinion. Leave a comment.