Abstract
This talk covers two results with a common theme: can simple models provide insight into phenomena observed in the training on (large) neural networks ?
The first part of the talk concerns the popular idea of contrastive representation learning. A key question here is: what makes the simple act of encouraging a point to be close to its augmentation, and far from other points, result in embeddings useful for downstream tasks ? We analyze the InfoNCE loss for linear networks trained on Gaussian Mixtures, and show that this results in meaningfully better dimensionality reduction as compared to spectral methods.
The second part of the talk concerns using the outputs of a model to train itself to better accuracy (with no new data). We study this problem in the context of binary classification with a linear classifier, when the data comes from a linearly separable ground truth corrupted with label noise. We show that if we (a) first learn a classifier, (b) use this to filter out data where the predicted label differs from the given label, and (c) retrain a model on the remaining data, then the new model has higher accuracy than the initial model. This procedure has implications for recovering the accuracy from data that has been artificially corrupted for label privacy.
(Joint work with Parikshit Bansal and Rudrajit Das)