Abstract
Data balancing across multiple modalities/sources appears in various forms in several foundation models, such as CLIP and DINO, and achieve near universal representation learning. We show that this iterative algorithm, usually used to avoid representation collapse, enjoys an unsuspected benefit: reducing the variance of estimators that are functionals of the empirical distribution over these sources. We provide non-asymptotic bounds quantifying this variance reduction effect and relate them to the eigendecays of appropriately defined Markov operators. We explain how various forms of data balancing in contrastive multimodal learning and self-supervised clustering can be interpreted as instances of this variance reduction scheme. This is based on joint work with R. Mehta, L. Liu, and S. Pal <https://arxiv.org/abs/2408.15065>.