NVIDIA Snapshot Cuts Kubernetes GPU Startup From Minutes to Seconds

Starting a large AI model can take minutes before it handles a single request. NVIDIA Snapshot takes a different approach: save a fully initialized GPU workload, then restore that state when inference capacity is needed.

Snapshot is an open-source checkpoint and restore system for GPU workloads on Kubernetes. It captures the workload’s CPU and GPU memory as a portable artifact, allowing inference replicas to return online in seconds instead of repeating the full startup process.

How Snapshot skips the slowest startup steps

A normal cold start begins from scratch. The process includes launching the workload, loading the model, running torch.compile, and capturing NVIDIA CUDA graphs before engine initialization returns. Snapshot skips the repeated initialization by restoring a workload from the point where it was checkpointed.

The system provides two operations for GPU Pods: Checkpoint and Restore. To save an already-running Pod, an operator creates a PodSnapshot that names it. To start a Pod from a template and checkpoint it as soon as it becomes ready, the operator creates a SnapshotJob.

Restoring uses a new Pod with the nvidia.com/restore-from annotation, which names the PodSnapshot to use. The restored Pod must run on a node with the same GPU model and driver. The workload needs only a small handshake to pause before checkpointing and resume after restoration.

Infrastructure provisioning and container image loading remain outside the measured restore path. That distinction matters because Snapshot measures the time needed to bring the initialized workload back, not every step involved in preparing a new machine.

Results across GPUs and large models

Tests show inference replicas coming online in 3.6 to 42.7 seconds after restoration. Across every tested configuration, the replica returned in under 45 seconds, producing an 8x to 36x speedup over cold starts that took minutes.

Checkpoint creation takes 15 seconds to 5.5 minutes, depending on model size. Checkpoints can reach 1 TB across eight GPUs, yet the combined checkpoint-and-restore time remained shorter than one cold start in every tested configuration.

Benchmark results using NVIDIA B200 GPUs and VAST NFS storage show where the restore time goes. Restoring GPU state takes 18 to 29 seconds for large restores, while CRIU-based CPU process restoration takes 4.2 to 10.8 seconds.

The benchmark compares two strategies on the same hardware and software stack: a cold start and a Snapshot restore. The cold-start clock runs from process launch until engine initialization returns, before inference. The timing breakdown includes Startup, Model load, and additional phases not fully shown in the provided table.

Designed for multi-GPU inference

Snapshot can checkpoint and restore compatible workloads that span multiple GPUs on one node as a single artifact across ranks. That makes the system useful for large mixture-of-experts models, which may use tensor parallelism, data and expert parallelism, and decode context parallelism.

The repository includes ready-made examples for vLLM, SGLang, and NVIDIA TensorRT LLM. Workloads still need to meet Snapshot’s checkpoint, restore, and runtime compatibility requirements, so the speed gains depend on whether a specific deployment fits those conditions.

Two design partners have supplied concrete results. Crusoe reduced GLM-5.3 Pod readiness on NVIDIA HGX B300 from roughly eight minutes to 31 seconds. Saturn Cloud benchmarked vLLM scale-from-zero inference on the same hardware, cutting startup from two to 8.5 minutes to between six and 34 seconds.

A wider push toward open AI infrastructure

Snapshot arrives alongside other efforts to make AI inference infrastructure more open and easier to operate. ZStack is open-sourcing GPUWorks, a lightweight, enterprise-grade AI inference platform, under the Apache License 2.0.

That focus reflects a larger production challenge: teams must consider the model, deployment approach, and infrastructure together without always taking responsibility for the entire serving stack. Running an open model in production turns model choice, deployment approach, capacity, accelerator, and serving-stack ownership into one engineering problem.

Managed compute in Microsoft Foundry is one way to provide dedicated GPU capacity without requiring customers to build and operate the full serving stack. Customers retain decisions that directly affect the workload, while Microsoft manages much of the serving environment.

Hardware choices remain part of that equation. The AMD Instinct MI300X provides 192 GB of HBM3 memory and up to 5.3 TB/s of memory bandwidth, with 1.5 TB across an eight-GPU platform. On an eight-GPU AMD Instinct MI355X system, ROCm 10 delivered average inference throughput 3.3 times higher than a ROCm 7.0 baseline across GLM-5, Kimi-K2.5, and DeepSeek-R1-0528.

GLM 5.3 Flash is a 320B-parameter model with 18B active per token, available in three configurations optimized for AMD Instinct MI300X and equipped with a 512K context window. In conversational workload tests, an eight-GPU TP4 + DP2 + EP configuration delivered about 1.8 times the output throughput of a four-GPU setup while cutting average time to first token by more than half.

AT&T used managed compute in Foundry across multiple open models and GPU architectures with about 530 GPUs, including around 430 AMD Instinct MI300X GPUs. Phi-4 processed more than 700 billion tokens each month for data preparation and synthetic data generation.

Alphabet’s Google is also introducing two new variations of its tensor processing units as it expands its AI chip lineup. The latest chips address confidential computing and help models access a wider range of information during queries. Alphabet shares gained as much as 4.1% to $296.33 in New York trading on Monday.

The common thread is simple: AI infrastructure has to reduce the time between demand and usable capacity. NVIDIA Snapshot targets that delay at the workload level, turning a fully initialized GPU process into a reusable artifact and bringing replicas back online in seconds.

Stay connected via Google News

Follow us for the latest AI News