Abstract: In deep learning workloads, collective communication across multiple nodes is a critical component in determining overall performance. AllReduce (as well as ReduceScatter and AllGather) is a ...