Allocate GPU capacity across rentals and inference

RunOS GPU Capacity lets you use a fleet for on-demand GPU VMs, interruptible spot VMs, and model inference. A capacity set groups GPU nodes in one cluster. Ranked pools define which workloads get capacity when they compete for cards.

Choose what each pool can do

Pool	What to use it for
On-demand GPU VMs	Rentals that the capacity arbiter must not interrupt.
Spot GPU VMs	Workloads that can save progress and resume after RunOS reclaims capacity.
Inference	GPU VMs that serve one model behind a router. Set a replica range and use an API key for inference requests.

A capacity set uses one GPU model and one fabric type. PCIe sets allocate single cards. NVSwitch sets can allocate supported blocks of one, two, four, or eight GPUs. A multi-GPU VM in an NVLink partition needs a whole available block on one node.

What triggers a capacity change?

Outside a capacity set, the number of inference replicas follows the request queue within the range you set. Inside a set, the capacity arbiter controls inference replicas. When the arbiter can make a complete plan, it can reclaim capacity from lower-ranked pools. It does not stop on-demand VMs or reclaim capacity below another pool’s floor.

The arbiter starts in dry-run mode. Review its plan before you enable enforcement. This lets you see which workloads would lose capacity before RunOS moves any cards.

For load-based allocation, evaluate how the inference queue and pool priorities affect your fleet. For a time-of-day policy, do not assume a pool rank acts as a calendar schedule. Test any scheduling integration separately.

When does a spot VM get a warning?

RunOS sends an interruption notice before an arbiter stop, a planned capacity-set drain, or a rental reclaim. The default notice is 120 seconds. The guest can read the notice through its instance metadata endpoint.

Outside a capacity set, Kubernetes scheduler preemption sends the guest a stop signal without that advance notice. Design spot workloads to recover from either path. GPU-attached VMs cannot live-migrate.

See the GPU Capacity release notes for more on inference pools. For usage export and tenant allocation, see RunOS for neocloud operators.