Silent data corruption is a GPU fault that returns wrong but plausible numbers with no error, quietly poisoning distributed training. This post adds three checks to a PyTorch DDP loop to detect it, localize the faulty node on Amazon SageMaker HyperPod, and hand the node to automatic recovery for replacement.
This article has been indexed from Artificial Intelligence
Read the original article:
Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod