Decoupled DiLoCo: A new frontier for resilient, distributed AI training
Google DeepMind published a paper proposing Decoupled DiLoCo, splitting large-scale training into decoupled compute islands that communicate through asynchronous data streams, isolating local hardware failures.