Peer-Predictive Self-Training for Language Model Reasoning

Mechanisms for continued self-improvement of language models without external supervision remain an open challenge. In this talk, I will present Peer-Predictive Self-Training (PST), a label-free fine-tuning framework in which multiple language models collaboratively improve by using a cross-model-aggregated response as an internal training target. Given a prompt, models generate responses sequentially; a final aggregated answer—often more reliable than any individual response—then serves as the target for learning. To guide updates, PST measures how informative each intermediate response is about the aggregate using pointwise mutual information (PMI) and scales gradient updates accordingly: responses already aligned with the aggregate are updated less, while misaligned or less informative responses are updated more. I will conclude with empirical results on mathematical reasoning tasks and discuss when peer-based self-training helps.