2025 AI Infra Summit - Ian Buck - NVIDIA's Rubin CPX and PD Separation - Advancing Efficient Large-Scale AI Inference and the AI Factory Vision

NVIDIA: Reshaping AI Inference and the "AI Factory" Vision Through Hardware-Software Co-optimization

In the rapidly evolving field of AI, the efficiency of large language model (LLM) inference and their ability to handle long contexts have become critical bottlenecks limiting their widespread adoption. NVIDIA is addressing these challenges by deeply integrating hardware and software advantages through its innovative PD Separation (Prefill-Decoding Separation) strategy and the Rubin CPX GPU, specifically designed for large-scale context processing. The goal is to build an efficient, high-ROI "AI Factory" ecosystem, thereby meeting the growing demand for AI applications and unlocking new commercial potential.

Challenges in AI Inference and the Rise of PD Separation

Building AI infrastructure, especially for LLM inference, is a complex and challenging task. While model training is often considered compute-intensive, inference also faces multiple optimization vectors and trade-offs, including model intelligence, response speed, cost, throughput, and energy efficiency. Ian Buck, Vice President of Hyperscale and HPC Computing at NVIDIA, emphasizes that these trade-offs must be fully considered in infrastructure development and future innovation roadmaps, as the AI field is evolving rapidly, and the complexity of inference necessitates full-stack innovation.

AI Factory Complexity

Source: NVIDIA

Traditional LLM operational models conflate two intrinsically different stages of the inference process, leading to efficiency bottlenecks. PD Separation is a key technology for improving LLM operational efficiency, explicitly "decoupling" two distinct steps in LLM inference:

• Prefill: In this stage, the LLM processes the user's long query and generates the KV cache. This is a compute-intensive task requiring substantial computational resources.
• Decoding: In this stage, the LLM "writes out the answer token by token" based on the KV cache. This is a memory bandwidth-bound task, primarily testing the ability to "retrieve KV cache entries" (memory access) and "generate tokens rapidly" (output).

Before PD Separation, a single GPU simultaneously handled both "understanding the query" and "generating the answer." This led to interference between tasks of different natures, inflexible resource allocation, and ultimately, high Time to First Token (TTFT) and slow Tokens Per Output Token (TPOT) in LLM services, severely impacting user experience and reducing service throughput.

Efficiency Comparison of Combined vs. Separated Processing

Source: NVIDIA

The image above clearly illustrates the advantages of PD Separation: in a scenario where 2 GPUs are allocated for prefill and 1 GPU for decoding (2P1D), separated processing (b) shows significant improvements in both flexibility and efficiency compared to combined processing (a).

How PD Separation Works and Software Implementation

PD Separation significantly boosts the efficiency of the LLM "factory" through specialized division of labor. Its working principle can be understood metaphorically:

1. Establishing a "Thinking Workshop": This refers to the prefill worker nodes, equipped with high-compute GPUs, dedicated to processing users' long queries and generating the key information-containing KV cache.
2. "Expressing" Key Notes: The KV cache is rapidly "shipped" to the decoding workshop via high-speed networking technologies like NVLink or PCIe 5.0. The overhead of this transfer can typically be "hidden."
3. Establishing a "Writing Workshop": This refers to the decoding worker nodes, which receive the KV cache. The GPUs in this workshop may not require extremely high compute power but need fast "KV cache retrieval" and "token generation" speeds to continuously produce answers.

Request Processing with Prefill/Decoding Separation

Source: NVIDIA

The key software enabling this decoupling is NVIDIA Dynamo. This is an open-source project aimed at disaggregating inference workloads, separating the context processing and generation tasks of input queries. This optimization allows for different AI kernels and parallelization techniques to be configured for parallel compute-intensive context prefilling and fast autoregressive generation, thereby significantly increasing overall throughput.

Disaggregated Service

Source: Jax ML Scaling Book

Real-world cases validate the power of Dynamo: for LLAMA models, decoupling can deliver a 2x to 4x speedup using the same number of GPUs. Base 10, an inference aggregation service company, achieved the fastest inference performance across all cloud providers by optimizing the separation of context processing and output generation using NVIDIA Dynamo. Overall decoupling can improve the Time to First Token for models like Quen by approximately 6x and the token output speed for models like DeepSeq by approximately 3x.

Base 10's Performance Improvement with Dynamo

Source: NVIDIA

The significant benefits of PD Separation include: eliminating interference for specialized efficiency; optimizing resource utilization; substantially increasing effective throughput (e.g., UCSD's DistServe system can increase effective throughput by 2x to 4.48x in some tasks); reducing latency (improving both TTFT and TPOT); and achieving higher "profit margins." Higher throughput means serving more users at the same cost, or serving the same number of users at a lower cost, thereby boosting gross margins. For instance, Deepseek achieves high profit margins due to its high throughput, with its V3/R1 models employing PD Separation technology. The industry has widely adopted or explored similar decoupled services, including Deepseek, UCSD Hao AI Laboratory's DistServe system, NVIDIA (Helix Parallelism), and Google (JetStream).

NVIDIA's SMART Platform

Source: NVIDIA

NVIDIA consistently maintains its lead in MLPerf inference benchmarks, attributed to its full-stack optimization, which includes NVFP4, Dynamo software, TensorRT, and the fully distributed inference capabilities of NVLink on GB300 racks.

MLPerf Inference Records

Source: NVIDIA

New Industry Trend: Demand for Large-Scale Long-Context Processing and the Emergence of Rubin CPX

While PD Separation optimizes general inference efficiency, AI applications are evolving towards longer contexts and more complex tasks, driving the need for specialized hardware. Although most models accept up to 256,000 input tokens, some high-value workloads require ultra-long input tokens (1M+ tokens).

High-Value AI Use Cases Drive Long Context Demand

Source: NVIDIA

These high-value use cases include:

• Advanced Coding: Advanced coding chatbots can process 100,000 lines of code or even millions of input tokens of code, transforming AI into a software agent that can increase software developer productivity by 10x.
• Video Processing Generation: Processing 1 hour of high-definition video and generating new content involves millions of tokens. Ian Buck notes that future entertainment interaction will be video-based and interactive. The AI video market is projected to exceed $40 billion by the early next decade.

To meet this emerging demand, NVIDIA has unveiled its new category of GPU products, Rubin CPX. Jensen Huang stated that Rubin CPX is the first CUDA GPU specifically designed for large-scale context AI, where models can process millions of tokens of knowledge simultaneously.

Rubin CPX: Designed for Large-Scale Context AI

Source: NVIDIA

Key features of Rubin CPX include:

• High Compute Power: Delivers up to 30 petaflops of NVFP4 precision computing capability.
• Memory Optimization: Equipped with 128GB of cost-effective GDDR7 memory to accelerate the most demanding context workloads.
• Attention Acceleration Cores: Newly added attention acceleration cores are 3x faster than existing GB300 GPUs, enhancing the ability of AI models to process longer context sequences without sacrificing speed.
• Enhanced Video Capabilities: Includes 4 video encoders and 4 decoders for processing and generating AI video content.

Rubin CPX aims to achieve the highest performance for long-context processing and token revenue, far exceeding existing system design capabilities. For example, the Vera Rubin NVL144 CPX enables companies to monetize at an unprecedented scale, generating $5 billion in token revenue for every $100 million invested. Early industry leaders have shown strong interest in Rubin CPX, including Cursor (an AI-powered software company, whose CEO Michael Truell states that Rubin CPX will transform software creation), Runway (a US generative AI company, whose CEO Cristóbal Valenzuela sees Rubin CPX as a significant leap in performance), Magic (an AI research and product company, whose CEO Eric Steinberger notes that its 100 million token context window model can see entire codebases in context without fine-tuning), as well as leading inference providers like Fireworks and Together AI.

Rubin CPX Unlocks AI Innovator Products

Source: NVIDIA

Hardware-Software Co-optimization: NVIDIA's AI Factory Vision

The introduction of Rubin CPX deepens the PD Separation concept at the hardware level, forming a complete solution for the future AI factory in conjunction with NVIDIA's software ecosystem. Rubin CPX is integrated with NVIDIA Vera CPUs and Rubin GPUs into the new NVIDIA Vera Rubin NVL144 CPX platform.

NVIDIA Vera Rubin NVL144 With Tray

Source: NVIDIA

This integrated NVIDIA MGX system provides within a single rack:

• 8 exaflops of AI computing power, which is 7.5x higher AI performance than the NVIDIA GB300 NVL72 system.
• 100TB of fast memory.
• 1.7 petabytes/second of memory bandwidth.

By inserting 8 Rubin CPX processors into the Vera Rubin rack, long-context processing capabilities can be significantly boosted, achieving a total performance of 8 exaflops, increasing memory to 1.7PB, and fast memory to 100TB.

Vera Rubin NVL144 CPX System Architecture

Source: NVIDIA

Furthermore, customers can attach VRCPX racks as sidecars to Vera Rubin racks, enabling the separation of context processing and generation workloads. This allows for flexible configuration of the ratio between VRCPX and Vera Rubin racks, forming a dual-rack solution.

Vera Rubin CPX Dual-Rack Solution

Source: NVIDIA

NVIDIA Rubin CPX will receive full support from the NVIDIA AI stack, including accelerated infrastructure and enterprise-grade software. This encompasses the NVIDIA Dynamo platform (for efficiently scaling AI inference), the NVIDIA Nemotron model family (providing advanced inference capabilities), NVIDIA AI Enterprise (for production-grade AI, delivering Nemotron models via NVIDIA NIM microservices), and the expanded NVIDIA CUDA-X ecosystem, which boasts a developer community of over 6 million and nearly 6,000 CUDA applications.

NVIDIA's AI infrastructure is not composed of a single chip but rather a family, including GPUs, CPUs, NVLink switch chips, Spectrum switches, and NICs. NVIDIA is committed to making these components work together as a cohesive whole to serve AI models and realize the value and revenue from inference. Its AI roadmap includes the Blackwell series, Rubin series (in addition to Rubin and Rubin Ultra, a newly optimized Rubin CPX processor for context processing has been added), and the future Feynman.

NVIDIA's AI Roadmap

Source: NVIDIA

NVIDIA is collaborating with the entire community to co-develop the AI Factory GigaScale Reference Design, aiming to provide a scalable, future-proof data center architecture that ensures all components work seamlessly together, maintaining uptime and high efficiency.

AI Factory GigaScale Reference Design

Source: NVIDIA

Conclusion and Return on Investment

NVIDIA Rubin CPX is expected to be available in late 2026. NVIDIA emphasizes that its AI factory solutions can deliver significant returns on investment. For example, a $3 million GB200 and NVL72 rack is projected to generate approximately $30 million in token revenue, implying a 10x return on investment for the Blackwell platform. This fully embodies NVIDIA's strategic vision of driving AI infrastructure towards a higher-ROI "AI factory" model through hardware and software innovation.

AI Factory Return on Investment

Source: NVIDIA