Results 1381 - 1390 of 24786

Classical supervised learning starts from a collection of input-output data pairs corresponding to a well defined task. Predictive accuracy then scales as the number of datapoints, the size of the function class, and the number of optimization steps all increase in appropriate relative proportions. A new learning and prediction paradigm has emerged and gained momentum over the last decade, where one can learn a highly predictive model without task specific data or even a formal definition of the target task. This paradigm, called zero shot prediction, has been successful both for known and unsuspected reasons. Starting from its origins in matching words and pictures, through recent self supervised and contrastive learning from different modalities or sources, I will describe key components driving its empirical performance in AI domains such as computer vision and language modeling.

This is based on joint work with Ronak Mehta, Lang Liu, and Soumik Pal, on data balancing <https://openreview.net/forum?id=8CBPnENQyV> and on zero-shot prediction <https://openreview.net/forum?id=kJQgMGLrow>.

Multitask Learning refers to the problem of aggregating many datasets from separate source distributions to improve performance on a target prediction task. Our aim is to understand (1) sufficient and necessary conditions for speedup in convergence rates over vanilla prediction with just the target data, (2) how such speedup depends on the number of datasets and samples per dataset, and (3) whether such speedup is achievable adaptively, i.e., by procedures with no prior distributional information.

The picture turns out to be mixed as the problem displays sharp gaps between oracle rates and adaptive rates, i.e., there exist situations where no procedure can do better than using the target data alone even though a large subset of datasets are informative about the target task. On the other hand, a bit of information on the relation between sources and target distributions can allow for near optimal adaptive rates. These results lead to many interesting new questions which I'll attempt to properly convey.

The talk is based on various works with collaborators such as S. Hanneke, A. Gretton, M. Z. Li, D. Meunier.

In this talk, we address high-dimensional mean estimation in a collaborative setting where $N$ users contribute data in batches of size $n$. We build on the framework of untrusted batches [Qiao and Valiant, 2018] and formalize a double corruption landscape: an $\epsilon$-fraction of users are entirely adversarial, while the remaining ``good'' users provide data related to a target mean but deviate by a proximity parameter $\alpha$. We address two variants for this deviation: (1) good batches have a mean-shift of $\sqrt{\alpha}$, or (2) an $\alpha$-fraction of samples within each good batch are adversarially corrupted. We provide two Sum-of-Squares (SoS) based algorithms to navigate this tiered corruption. Our algorithms achieve the minimax-optimal error rate: $O(\sqrt{\frac{\epsilon}{n}} + \sqrt{\frac{d}{nN}} + \sqrt{\alpha})$. This rate demonstrates that while heterogeneity $\alpha$ represents an inherent statistical difficulty, the influence of adversarial users is suppressed by a factor of $1/\sqrt{n}$ due to the internal averaging afforded by the batch structure.

Continual learning, the ability of a model to adapt to an ongoing sequence of tasks without forgetting the earlier ones, is a central goal of artificial intelligence. To shed light on its underlying mechanisms, we analyze the limitations of continual learning in a tractable yet representative setting. In particular, we study one-hidden-layer quadratic neural networks trained by gradient descent on an XOR cluster dataset with Gaussian noise, where different tasks correspond to different clusters with orthogonal means. Our results obtain bounds on the rate of forgetting during train and test-time in terms of the number of iterations, the sample size, the number of tasks, and the hidden-layer size. Our results reveal interesting phenomena on the role of different problem parameters in the rate of forgetting. Numerical experiments across diverse setups confirm our results, demonstrating their validity beyond the analyzed settings.