Nvidia ships TensorRT-LLM 3.0 and makes NIM inference microservices generally available across major clouds
Enzo Reno ·
Nvidia released TensorRT-LLM 3.0 and said its NIM inference microservices are now generally available on AWS, Microsoft Azure, and Google Cloud on Tuesday, aiming to cut latency and improve utilization for large language model serving. The company said the software delivers faster decode and better batching on H200 and GB200 based instances, a priority as inference spend grows across enterprises.
Nvidia said internal tests show up to 1.8 times higher tokens per second on Llama 3.1 70B and up to 2.2 times on Mixtral 8x22B compared with TensorRT-LLM 2.5 on H200, measured with FP8 weights, paged attention, and speculative decoding enabled. The NIM release wraps these optimizations in containerized endpoints with standardized APIs, logging, and security policies for production use.
Why this matters now Analysts expect inference to account for the majority of enterprise AI infrastructure bills in 2026, with Omdia estimating inference will exceed 60 percent of total accelerator hours this year. Vendors are under pressure to squeeze more throughput from existing fleets as supply for high end parts remains tight.
Availability and pricing NIM is included in Nvidia AI Enterprise, which is priced at 4,500 dollars per GPU per year for on premises deployments according to Nvidia. Cloud marketplace listings are available with pay as you go terms through AWS Marketplace, Azure Marketplace, and Google Cloud Marketplace. Nvidia said the update adds first class support for H200, L40S, and GB200 based instances as they become available from cloud partners.
The 3.0 release adds continuous batching improvements, FP8 quantization paths with per channel scaling, and a new speculative decoding policy that works with vLLM and Triton Inference Server. Nvidia said schema constrained output and tokenizer level guardrails are now built into NIM endpoints to assist with auditability as the EU AI Act begins phased enforcement through 2026 and 2027.
Nvidia pointed to recent MLPerf Inference results showing its accelerators leading in the data center category for LLM tasks, a position it has held across multiple rounds. MLCommons reported record submissions in 2025 with H200 systems showing higher throughput than prior H100 platforms in server and offline scenarios.
Partners including ServiceNow, SAP, and Adobe have been piloting NIM backed endpoints for internal assistants and content workflows, according to the companies. Systems vendors Dell Technologies, HPE, and Lenovo said they will certify the new stack on current liquid cooled racks aimed at LLM inference clusters.
The move underscores a shift from training centric headlines to production serving economics. If Nvidia can translate software level gains into measurable cost per token reductions, that strengthens its hold on enterprise inference even as competitors push alternatives built on Gaudi, custom ASICs, and managed routing services.