Title:

A Unifying Framework for Decentralized Machine Learning

Abstract:

Machine Learning (ML) applications have a tremendous impact on our everyday life. However, we currently pay a high price for this (mostly) comfort, as our sensitive data is collected and exploited at scale. The striking imbalance between data ownership and control is partially grounded in the fact that training of ML models currently requires central gathering and aggregation of all data. In contrast, the decentralized training paradigm can offer the users more control over their own data by collaboratively training ML models directly on their edge devices without the need to send their data to a central coordinator first. However, there are many technological challenges to address before such systems are (operational) efficient enough to become reality. In this talk we tackle decentralized training from an optimization perspective. We specifically discuss algorithms for distributed, federated and decentralized topologies and present a unifying framework that allows to derive state-of-the art convergence results for convex and non-convex settings. We discuss two key strategies that allow to reduce communication cost between devices in more detail: firstly, gradient scarification with error feedback, and secondly, local update steps. This will allow us to derive and present the intuition and techniques behind our framework. The end of the talk we will highlight and discuss major challenges that are left on the way to more open and user-friendly machine learning in the future.

Bio:

Sebastian Stich (www.sstich.ch) is a postdoctoral fellow in computer science at EPFL in Lausanne, hosted at the machine learning and optimization lab of Prof. Martin Jaggi. He received his PhD in 2014 from ETH Zurich where he was advised by Prof. Bernd Gaertner and later stayed for a postdoc at UC Louvain, where he was mentored by Prof. Francois Glineur and Prof. Yurii Nesterov. His research focuses on algorithms for distributed and decentralized optimization and machine learning.