AWS Machine Learning Blog· Siddharth Senger·· 4 d ago
Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod
Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod
AI summary
Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three checks to a PyTorch DDP loop to detect it, localize the faulty node on Amazon SageMaker HyperPod, and hand the node to automatic recovery for replacement.
Selection record
Threshold 60Official, first-handFirst 58Second 34
Not admittedSum of both 92 < twice the threshold 120
- Source tier
- Official, first-hand; this tier's threshold is 60
- Pre-filter
- passed：SageMaker分布式训练静默数据损坏检测教程
A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.
Source: AWS Machine Learning Blog · aws.amazon.com