SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints
SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints
Amazon SageMaker Inference now supports GPU partitioning with NVIDIA Multi-Instance GPU (MIG), which divides one GPU into hardware-isolated partitions so you can run a different model in each or take a right-sized fraction of a newer accelerator. This post shows how two optional API fields place one model per partition, and what we measured against a whole older GPU.
Selection record
Not admittedSum of both 116 < twice the threshold 120
- Source tier
- Official, first-hand; this tier's threshold is 60
- Pre-filter
- passed：SageMaker推理GPU分区部署AI模型
A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.
Source: AWS Machine Learning Blog · aws.amazon.com