Nvidia makes NIM inference microservices generally available across major clouds and on premises

Nvidia announced general availability of its NIM inference microservices on Tuesday, opening access through AWS, Microsoft Azure, and Google Cloud marketplaces and for on premises deployments via NVIDIA AI Enterprise. The release extends a preview introduced at GTC on March 18, 2024, and is aimed at enterprises that want consistent model runtimes across clusters and regions to rein in inference costs and meet compliance requirements.

The company is bundling NIM under the NVIDIA AI Enterprise subscription, which Nvidia lists at 4,500 dollars per GPU per year for on premises support. Nvidia said NIM containers for popular models, including Meta Llama 3, Mistral 7B and Mixtral, and Stable Diffusion variants, are available through NGC and the cloud marketplaces with enterprise support and SLA options.

Nvidia published internal benchmarks that claim TensorRT LLM based NIM builds deliver between 1.5 times and 2 times higher tokens per second on H100 compared with unoptimized open source stacks such as vLLM and Hugging Face Text Generation Inference, measured on Llama class models at typical batch sizes. MLCommons’ MLPerf Inference results have consistently placed Nvidia accelerators at the top of throughput charts, and analysts expect that advantage to carry to Blackwell generation systems as shipments ramp in 2026.

The general availability spans managed deployment patterns in the clouds and private clusters for data residency. Nvidia said NIM supports request isolation, token level logging, rate limiting, and integration with enterprise identity providers, features designed for regulated sectors that face new obligations under the EU AI Act in 2026 according to the European Commission.

The move places Nvidia’s software directly alongside managed endpoints from AWS Bedrock, Azure AI Foundry, and Google Vertex AI. Gartner research vice president Arun Chandrasekaran has said buyers want vendor supported runtimes that are portable across infrastructure, a requirement that has become more visible as finance and legal teams scrutinize per token costs after initial pilots.

Nvidia controls more than 80 percent of AI accelerator revenue according to Omdia estimates from 2024, and reported 29,600 employees in its fiscal 2024 filing. By broadening NIM’s availability, the company is pushing deeper into the software layer while safeguarding recurring platform revenue tied to its GPUs.

For the broader technology landscape, the release underscores how chip suppliers are packaging software, containers, and support to standardize model operations across clouds and private data centers, a trend that could pressure independent inference servers while forcing cloud providers to differentiate on data, governance, and total cost guarantees.