Results 731 - 740 of 24393

Nearly all scaling laws research treats English as the evaluation objective. In this work, we redefine scaling laws to support multilingual objectives, and to inform how practitioners should mix hundreds of language sources to optimize for a non-English language target. Our state-of-the-art scaling laws are also used to determine if its better to pretrain from scratch or finetune from a multilingual checkpoint, and how many languages different model sizes can easily support. We release our cross-lingual transfer matrix for practitioner use and downstream analysis.

Mechanisms for continued self-improvement of language models without external supervision remain an open challenge. In this talk, I will present Peer-Predictive Self-Training (PST), a label-free fine-tuning framework in which multiple language models collaboratively improve by using a cross-model-aggregated response as an internal training target. Given a prompt, models generate responses sequentially; a final aggregated answer—often more reliable than any individual response—then serves as the target for learning. To guide updates, PST measures how informative each intermediate response is about the aggregate using pointwise mutual information (PMI) and scales gradient updates accordingly: responses already aligned with the aggregate are updated less, while misaligned or less informative responses are updated more. I will conclude with empirical results on mathematical reasoning tasks and discuss when peer-based self-training helps.

Consider a marketplace of AI tools, each with slightly different strengths and weaknesses. By selecting the right model for the task at hand, a user can do better than simply committing to a single model for everything. Routers operate under a similar principle, where sophisticated model selection can increase overall performance. However, aggregation is often noisy, reflecting in imperfect user choices or routing decisions. This leads to two main questions: first, what does a "healthy marketplace" of models look like for maximizing consumer utility? Secondly, how can we incentivize producers to create such models? Here, we study two types of model changes: market entry (where an entirely new model is created and added to the set of available models), and model replacement (where an existing model has its strengths and weaknesses changed). We show that winrate, a standard benchmark in LLM evaluation, can incentivize model creators to homogenize for both types of model changes, reducing consumer welfare. We propose a new mechanism, weighted winrate, which rewards models for answers that are higher quality, and show that it provably improves incentives for producers to specialize and increases consumer welfare. We conclude by demonstrating that our theoretical results generalize to empirical benchmark datasets and discussing implications for evaluation design. Based on joint work with Manish Raghavan.

Collaborative and federated learning systems are powered by data from many independent participants, yet these participants often have little visibility into how their data shapes the resulting model. In this talk, we will briefly overview a line of work on understanding training data's impact on model behavior. I will then discuss applications of these tools to two common problems arising in collaborative learning settings, namely machine unlearning and data pricing.

This talk studies how response times can be used together with choice data to recover latent preferences. We propose a general methodology for recovering preference parameters from data on choices and response times. Our methods yield estimates with fast convergence rates when specialized to the popular Drift Diffusion Model (DDM), but are broadly applicable to generalizations of the DDM as well as to alternative models of decision making that make use of response time data. An application to intertemporal choice illustrates that response times are not just auxiliary data: they improve prediction and change the substantive estimates we obtain. Based on joint work with Federico Echenique and Michael I. Jordan.

From statistics, scoring rules evaluate probabilistic forecasts of an unknown state against the realized state and are a fundamental building block in the incentivized elicitation of information. Proper scoring rules are ones where the forecaster is incentivized to report their true belief about the unknown state.

The talk will review the classical and recent work on (statistical) scoring rules. This review will include the geometry of proper scoring rules (classical) and the optimization of scoring rules (to induce more effort of the forecaster to learn about the state).

The talk will generalize these statistical scoring rules to give an algorithm for scoring elicited text against ground truth text using domain-knowledge-free queries to a large language model (specifically ChatGPT).

A key application for text scoring is in peer grading where students grade each other's work. In peer grading, a challenge is in incentivizing high quality peer reviews. The textual scoring rule can be used to grade a textual peer review by comparing it to the textual instructor review. This example parallels super alignment, specifically, the more intelligent students are scored using the less intelligent domain-knowledge-free queries that are sent to a large language model.