As AI models grow larger, support longer contexts and handle more agent-driven tasks, GPU memory is facing increasing demand, tighter supply and higher prices. NVIDIA says its response is a full-stack approach that combines newer memory technologies—including HBM3e and HBM4—with GPU, CPU, networking, storage and software design. The company argues that managing the entire memory hierarchy, rather than simply adding HBM, can increase inference throughput and reduce the cost of useful output.
Agentic inference increases pressure on HBM
Frontier AI models now contain trillions of parameters, the values that encode what a model learned during training. Inference also relies on a KV cache, which stores the context needed to process active requests without repeatedly processing the same information.
Agent workloads can repeatedly invoke models and tools while retaining earlier context. An agent trajectory may process up to 3 million tokens, causing the working memory required for each request to grow over time. Retrieval-augmented generation (RAG) adds further context by incorporating relevant company information into a prompt.
During inference, HBM primarily stores model parameters and KV-cache data. As both demands increase, pressure on the fastest and most expensive memory tier rises.
Why AI factories need a memory hierarchy
Adding HBM capacity to each GPU is an obvious response to rising demand, but capacity alone does not determine performance. The memory must also provide enough bandwidth to move data to and from the compute engines.
HBM can be compared with a warehouse, while memory bandwidth acts like the warehouse’s loading dock. A larger warehouse can hold more inventory, but a restricted dock limits how quickly that inventory can enter or leave. Once the dock is saturated, response times increase. If delays exceed the required service level, additional memory can raise costs and power consumption without delivering a corresponding performance benefit.
Businesses do not keep all inventory in the closest and fastest location. Items moving through a production line remain nearby, while items waiting for use are placed in larger, less expensive storage and brought back when needed. AI factories face a similar trade-off across HBM, CPU memory, local SSDs and shared storage.
More HBM can hold more KV-cache data and support more concurrent users, but all users share the available compute resources and memory bandwidth. If users require rapid responses, bandwidth can eventually become unable to supply data as quickly as the processors consume it. Engineers refer to this limit as the memory wall.
Beyond that point, extra HBM still costs money and consumes power, but does not necessarily improve throughput. Data still needs to be stored, yet not all data requires the fastest and most expensive tier. AI factories therefore distribute data across memory and storage layers according to how quickly it is needed.
Full-stack co-design puts each byte to work
NVIDIA says its full-stack co-design approach makes more effective use of HBM by reducing or relocating model weights, KV-cache data and RAG information. The techniques target different sources of memory inefficiency.
Smaller weights distributed across NVL72
- NVFP4 uses less space for model weights than FP8, which NVIDIA describes as the lowest precision available on its Hopper platform. Recent models using NVFP4 checkpoints include Flux.2 [klein] from Black Forest Labs, heartwarming from Thinking Machines Laboratory, Beam 1.0 from Reflection, MetaMuse Spark from Meta, Step 5 Preview from StepFun and NVIDIA Nemotron 3.5 Super VL. NVIDIA says these examples reflect growing adoption of four-bit precision for faster, more memory-efficient inference.
- Within an NVL72 system, model weights are distributed across up to 72 GPUs. Each GPU therefore stores a smaller portion of the model, leaving more HBM available for KV-cache data and concurrent users.
Smaller KV caches and lower-cost storage
- NVIDIA is applying NVFP4 to the KV cache. Compared with FP8, the smaller footprint allows more context to remain in HBM.
- NVIDIA Vera CPU helps in two ways. Faster tool execution reduces the time the KV cache remains inactive during tool calls, while its large LPDDR memory, connected to NVIDIA Rubin GPUs through the NVIDIA NVLink-C2C interconnect, provides a nearby location for paused contexts before they are needed again.
- When a context is inactive, NVIDIA Dynamo reasoning software can move it to NVIDIA CMX Contextual Memory Storage. CMX is designed to provide more capacity at a lower cost per bit than HBM and return the context before reuse. Unlike local SSDs, CMX is shared across servers. Unlike enterprise storage, it avoids long-term data-protection overhead that is not required for short-lived context data.
NVIDIA CMX Contextual Memory Storage partners include Claudian, DDN, Dell Technologies, Ever Pure (formerly Pure Storage), HPE, IBM, NetApp, VAST and Weka.
- Dynamo’s KV-cache reuse connects the memory layers and manages the cache as a system-wide resource across GPU memory, CPU memory and storage. This allows context to be computed once and reused across the system.
Keeping RAG indexes out of HBM
- RAG converts enterprise documents into embeddings—numeric representations that allow systems to search by meaning rather than by keywords—and organizes them into searchable indexes. NVIDIA says its Nemotron Retriever model produces smaller embeddings with higher retrieval accuracy, while the NVIDIA cuVS library can build and serve the index across CPU memory and storage. The full knowledge base does not occupy HBM; only the passages required by the model are brought into the active context.
NVIDIA says the result is less HBM used for model weights, fewer HBM resources occupied by inactive contexts and more capacity available to serve concurrent users.
More useful output per dollar
Across successive generations of NVIDIA platforms, the company says tokens per dollar per second have grown much faster than HBM capacity.
For SemiAnalysis AgentX DeepSeek V4 Pro workloads, NVIDIA says the GB300 NVL72 delivers up to 10 times higher throughput per dollar than the NVIDIA H200 platform at 100 tokens per second per user, while providing nearly twice the HBM capacity per GPU. The Vera Rubin NVL72 delivers up to 30 times more throughput per dollar than the GB300 NVL72 at 125 tokens per second per user, with approximately 3% more HBM capacity per GPU.
For AI factory operators, an important measure is how many useful tokens a system can deliver within its power budget while meeting response-time requirements. More tokens per watt per second mean more work and potential revenue from the same power supply, while reducing the cost of each token. NVIDIA says its full-stack co-design lowers token costs by increasing the useful work performed by each byte of memory.
For more information, see the video explainer and technical details on how NVIDIA’s full-stack co-design is intended to reduce inference memory pressure.
Source: blogs.nvidia.com