Skip to content

AWS introduces silent data corruption detection for SageMaker HyperPod

What happened

AI digest

On 12 November 2026, the AWS Machine Learning Blog published a method for detecting and recovering from silent data corruption (SDC) on Amazon SageMaker HyperPod. The method adds three checks to the PyTorch DDP training loop: a non-finite gradient check to catch NaN/Inf; a ring replay audit that compares loss and gradient fingerprints recomputed by another node to spot anomalies; and a weight fingerprint comparison to find corruption in optimiser steps and stored weights. The post does not mention the method's deployment scope, effectiveness data or details of the recovery process.

Written by AI from the reports · updated 3 d ago

Timeline

Follow the reports to see the event from each side.

Nov 12
  1. AWS Machine Learning Blog
    Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

    Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three checks to a PyTorch DDP loop to detect it, localize the faulty node on Amazon SageMaker HyperPod, and hand the node to automatic recovery for replacement.

Heat of this event

Current heat 3·Peak in the comparable range 5(Nov 13 06:00)·Change over 24 hours in the comparable range -51%

0246Nov 1306:00Nov 1314:00Nov 1323:00Nov 1407:00

The trend compares only the same accounts observed without a break, so its range can be narrower than the current heat count. Move the pointer or tap the chart to see each hour; with a keyboard, use the left and right arrow keys.