As organizations transition from AI pilots to fully operational AI factories, infrastructure decisions are evolving from peak chip specifications to cost-effective metrics per chip. Token: The quantity of useful tokens delivered per dollar, per watt, and within the specified latency parameters.
Co-designed with NVIDIA’s GPUs, CPUs, networking, and systems, and supported by a robust open-source ecosystem, NVIDIA’s comprehensive inference software continually enhances hardware performance. This is exemplified in NVIDIA Blackwell. The software stack has already cut token costs by up to 5x in just one month with the DeepSeek V4 model.
Leading companies and inference providers are already recognizing the cumulative advantages of NVIDIA’s inference software stack on Blackwell.
- BaseTen leverages NVIDIA’s TensorRT-LLM open-source library to power DeepSeek V4 Pro on Blackwell GPUs for inference tasks, optimizing runtimes to deliver up to 50% more tokens per second.
- Cognition utilizes NVIDIA Dynamo for managing inference GPUs, providing teams with a streamlined scaling path for their reinforcement learning workloads without the need for building infrastructure from scratch.
- Deep Infrastructure applies the NVIDIA inference software stack to deliver cutting-edge open-source models, including DeepSeek V4, achieving performance targets on Blackwell from inception.
- DigitalOcean supported Hippocrates AI in rapidly and efficiently providing healthcare AI through NVIDIA inference software on Blackwell GPUs, boosting inference throughput by 30% while maintaining a response time of under 0.5 seconds for 10 million patient calls.
- AI Together utilized NVIDIA TensorRT-LLM at Blackwell to expedite the transition from model optimization to production endpoints, facilitating real-time coding experiences.
Importance of Software in Inference Economics
Traditional web, search, and software-as-a-service workloads exhibit predictable patterns. A user may browse a page, refresh a feed, or modify a business record; these requests typically follow a consistent software pathway involving reading or writing to a database and expanding by adding similar servers.
Agent AI operates differently.
Agents can reason, plan, invoke tools, initiate specialized subagents, and manage context at scale within multi-turn workflows. They convert a single request into a distributed computing challenge involving hundreds of subagents, thousands of tasks, and various large-scale language models across GPUs, CPUs, DPUs, and storage systems.
The software stack determines whether its complexity leads to underutilized capacity or efficient scalability — ultimately affecting the cost per token.
Cost per token is realized through converting individual optimizations into system-wide performance. NVIDIA’s inference software stack achieves this by linking three essential layers:
- Production Operations: Coordinates distributed services, orchestration, autoscaling, and memory management, ensuring inference runs optimally across compute and storage resources.
- Application Acceleration: Employ runtime optimizations like compute and communication overlap and kernel fusion to maximize model performance while allowing developers more room for customization and tuning.
- Infrastructure Access: Grants access to NVIDIA GPU, networking, memory, and systems capabilities without the need for developers to manage all device instruction sets and data transfer protocols.
When these layers function synergistically, their individual optimizations amplify each other’s effects.
Disaggregated services and massive expert parallel processing through NVIDIA NV Link, NVFP4 accuracy, and multi-token prediction each offer significant benefits individually. Collectively, they can boost throughput by up to 20x.
The graph below illustrates these results. Achieving these advantages in a production environment is intricate and necessitates coordination across the entire inference stack—encompassing production operations, model runtimes, kernels, communication libraries, and hardware access. NVIDIA’s inference software stack is tailored to ensure these layers collaborate effectively, allowing each optimization to build upon the previous ones.
Amplifying Benefits of Full Stack through Open Source
This comprehensive full-stack foundation is bolstered by a thriving open-source ecosystem. Many of today’s widely adopted open-source AI frameworks and inference projects are built natively on NVIDIA CUDA, ensuring new research and software optimizations achieve top performance on NVIDIA GPUs from the outset.
A prime example is PyTorch. Introduced in 2016 with native CUDA support, PyTorch has evolved alongside NVIDIA’s architecture, granting developers direct access to innovations such as Tensor Cores, Transformer Engine, and NVFP4 through familiar frameworks.
When breakthroughs like DFlash speculative decoding—which delivers up to 15x greater throughput on existing hardware—are introduced, they can be immediately implemented on NVIDIA platforms via PyTorch, helping AI factories swiftly translate research advancements into reduced token costs.
Thanks to this same open-source momentum, as new cutting-edge models like DeepSeek V4 emerge, leading inference frameworks such as vLLM and SGLang offer a Day Zero introduction guide for NVIDIA Blackwell architecture—facilitating accessibility across millions of Blackwell GPUs. This is why the performance of DeepSeek V4 on Blackwell has roughly improved by 5x across vLLM within just a month. Additionally, the introduction of SGLang has decreased token costs by approximately 80%.
That’s the open-source flywheel. As more developers enhance the CUDA-native inference pathway, a feedback loop emerges, leading to increased production deployments and continuous software improvements—resulting in higher token output while lowering the cost per token over time.
Discover how software can double your hardware performance in this article. Listen to the NVIDIA AI Podcast on Tokenomics and visit our Inference solutions page.
Source: blogs.nvidia.com