Results 1361 - 1370 of 24762

In this talk, we address high-dimensional mean estimation in a collaborative setting where $N$ users contribute data in batches of size $n$. We build on the framework of untrusted batches [Qiao and Valiant, 2018] and formalize a double corruption landscape: an $\epsilon$-fraction of users are entirely adversarial, while the remaining ``good'' users provide data related to a target mean but deviate by a proximity parameter $\alpha$. We address two variants for this deviation: (1) good batches have a mean-shift of $\sqrt{\alpha}$, or (2) an $\alpha$-fraction of samples within each good batch are adversarially corrupted. We provide two Sum-of-Squares (SoS) based algorithms to navigate this tiered corruption. Our algorithms achieve the minimax-optimal error rate: $O(\sqrt{\frac{\epsilon}{n}} + \sqrt{\frac{d}{nN}} + \sqrt{\alpha})$. This rate demonstrates that while heterogeneity $\alpha$ represents an inherent statistical difficulty, the influence of adversarial users is suppressed by a factor of $1/\sqrt{n}$ due to the internal averaging afforded by the batch structure.

Continual learning, the ability of a model to adapt to an ongoing sequence of tasks without forgetting the earlier ones, is a central goal of artificial intelligence. To shed light on its underlying mechanisms, we analyze the limitations of continual learning in a tractable yet representative setting. In particular, we study one-hidden-layer quadratic neural networks trained by gradient descent on an XOR cluster dataset with Gaussian noise, where different tasks correspond to different clusters with orthogonal means. Our results obtain bounds on the rate of forgetting during train and test-time in terms of the number of iterations, the sample size, the number of tasks, and the hidden-layer size. Our results reveal interesting phenomena on the role of different problem parameters in the rate of forgetting. Numerical experiments across diverse setups confirm our results, demonstrating their validity beyond the analyzed settings.

Large language models are often limited by data, especially when valuable datasets are distributed across institutions or cannot be shared. We introduce FlexOlmo, a new class of Mixture-of-Experts (MoE) models designed for flexible, modular data use. In FlexOlmo, expert modules are trained independently on separate datasets and later merged seamlessly into a single model. This enables distributed training without data sharing, supports the use of closed datasets, and allows data to be opt-in or opt-out at inference time. We scale FlexOlmo to 37B parameters (20B active) and evaluate on 31 diverse downstream tasks. FlexOlmo significantly outperforms models trained on public data only and approaches the performance of an upper-bound model trained on all datasets. By enabling modular integration of closed data while respecting data ownership and control, FlexOlmo offers a practical path toward collaborative, continuous model development.

Large language models (LLMs) have not yet effectively leveraged the vast amounts of data available on edge devices. Federated learning (FL) offers a promising way to collaboratively fine-tune LLMs without transferring private edge data to the cloud. To work within the computation and communication constraints of edge devices, recent research on federated fine-tuning of LLMs uses low-rank adaptation (LoRA) and similar parameter-efficient methods. LoRA-based methods suffer from accuracy loss in FL settings, primarily due to data and computational heterogeneity across clients. In this talk, I will first discuss an adaptive multi-head LoRA method that balances parameter efficiency and model expressivity by reparameterizing weight updates as the sum of multiple LoRA heads. In the second part of my talk, I will discuss other ways to leverage edge data, such as one-shot merging of locally trained models or training query routers personalized to each client's edge data.