The Many Faces of Heterogeneity: Federated, Continual, and Modular Learning

Data heterogeneity is a pervasive challenge in modern machine learning, yet it is typically studied in isolation within separate subfields. For instance, in federated learning, statistical heterogeneity across clients causes local models to drift and overspecialize; in continual learning, temporal heterogeneity in data distributions leads to catastrophic forgetting; in model merging and modular approaches, independently trained components make their composition fragile and unpredictable. In this talk, I argue that these are manifestations of the same fundamental tension: learning from diverse, non-stationary, and decentralized sources, while preserving and composing acquired knowledge. I will present a unified perspective that connects these three settings and show how insights can transfer across them, pointing toward a common research agenda: developing methods that embrace heterogeneity as a design principle rather than treating it as an obstacle to overcome.