Azure Blob Storage Integrates with NVIDIA Dynamo to Accelerate AI Inference

Microsoft and NVIDIA have released native integrations between Azure Blob Storage and NVIDIA Dynamo, a suite of tools for running accelerated AI inference on Kubernetes. The integrations address two bottlenecks in the inference lifecycle: loading model weights into GPU memory before serving, and managing the key-value (KV) cache during inference.

Model weight loading is the larger issue. Frontier models often exceed one trillion parameters, with weight files over one terabyte. Weights must be in GPU memory before a model can serve requests, making loading the largest contributor to cold start latency. Faster cold starts enable clusters to scale more quickly under traffic spikes and reduce idle GPU time.

Dynamo ModelExpress coordinates loading, caching, distributing, and reusing weights. Its integration with Run:ai Model Streamer streams weights from Azure Blob Storage into GPU memory using parallel transfer and prefetching, designed to use the full network bandwidth of the Azure VM's NIC. In testing on multiple node types, vLLM with ModelExpress loaded weights up to 7.3x faster and began serving up to 4.6x faster than the default vLLM loader. Gains increase with model size.

The second integration targets the KV cache. Inference generates key-value data that is expensive to compute, so it is cached for reuse. That cache typically consumes scarce GPU memory. NVIDIA's Inference Xfer Library (NIXL) handles high-bandwidth transfers of KV cache between GPU memory and storage tiers, and a new plugin extends that to Azure Blob Storage, allowing tools like LMCache to offload the cache to Blob. On a cache hit, the serving engine reuses cached KV data instead of recomputing it. The teams report up to 2.8x lower time-to-first-token (TTFT) latency compared to recomputing, while freeing GPU memory.

For Azure Kubernetes Service deployments, claimed benefits include central weight storage with Microsoft Entra ID and Azure RBAC access controls, Blob Storage's lifecycle management and durability, and a cache layer scaling with Blob rather than GPU memory. The benchmark figures are the companies' own measurements, not independent results. Deployment recipes for both integrations are available, providing hardware, configuration, and procedure details, and serve as a practical starting point for verifying claims against real workloads.