This is part II of a two-part series, describing our solution for running distributed training on spot instances using TorchElastic and Kubernetes. Part I introduced our overall technology selection, design principles and benchmarks.

Deep Learning development is becoming more and more about minimizing the time from idea to trained model. To shorten this lead time, researchers need access to a training environment that supports running multiple experiments concurrently, each utilizing several GPUs.

Photo by Caleb Jones on Unsplash Almost every leader you talk to will quote building trust as one of the most important things for creating a highly functional team. However, one of the most common mistakes leaders make on a weekly basis, is not providing their team with good “whys”.