Microsoft makes AMD MI300X GPU VMs generally available on Azure, targets lower LLM inference costs

Microsoft said Tuesday it has made AMD MI300X GPU virtual machines generally available on Azure in select regions, positioning the hardware for large language model inference and fine tuning at scale. The ND MI300x v5 instances bundle eight MI300X accelerators with a combined 1.5 terabytes of HBM, and are integrated with Azure Machine Learning, Azure Kubernetes Service, and confidential computing options.

The move matters because enterprise buyers have pushed for alternatives to Nvidia capacity, and because memory bound 70 billion parameter models can fit entirely in memory on a single node. Microsoft said the SKUs are now enabled for managed endpoints and autoscaling, which shortens time to production for AI workloads tied to budgeted Q3 and Q4 deployments.

Microsoft said the ND MI300x v5 uses ROCm 6 and supports PyTorch 2.x, vLLM, and Hugging Face Text Generation Inference out of the box. Each VM exposes eight 192 gigabyte MI300X GPUs and high speed interconnect, which the company said enables multi GPU tensor parallelism for contexts above 200,000 tokens without offloading.

In internal tests shared by Microsoft, MI300X nodes sustained higher tokens per second on Llama 3 70B at batch sizes above 16 compared with equivalently provisioned H100 based instances, using vLLM and KV cache optimizations. AMD has published similar results that show MI300X matching or exceeding H100 throughput on Llama class models under ROCm, which aligns with customer pilots reported by cloud providers in 2024.

Azure said customers can orchestrate MI300X fleets through Azure ML Prompt Flow, with token level observability, and can deploy through AKS using GPU partitioning for mixed inference workloads. The company said model serving images include optimized kernels for FlashAttention and grouped query attention on ROCm, and that vector database integrations for pgvector and Redis are validated on these SKUs.

Microsoft is maintaining a multi silicon strategy that includes Nvidia H100 and H200 based NDv5 and NDv6 families, its own Maia accelerators for internal inference, and Google TPU and Oracle collaborations across the ecosystem. Canalys estimated cloud infrastructure services spending hit 356 billion dollars in 2025, with performance per dollar now a primary selection criterion for generative AI projects, which gives AMD based fleets a clearer role in procurement.

For the broader technology stack, reliable MI300X capacity on a major cloud expands the practical supply of high memory GPUs, which can widen model choice and reduce dependency on a single vendor, and it reinforces that enterprise AI planning now hinges on heterogeneous accelerators, software portability, and cost aware throughput engineering.