What Really Separates Autoregression and Diffusion? A Synthesis and Path Beyond

Modern generative AI is often viewed through two dominant paradigms: autoregressive models, which underpin much of language modeling, and diffusion models, which have set the standard for high-quality image and video synthesis. These paradigms are usually treated as fundamentally different: autoregressive models are associated with compression and sequential computation, while diffusion models are associated with perceptual sample quality and parallel generation.

In this talk, I will argue that this divide is less fundamental than it may appear. I will use masked diffusion models (MDMs) as a lens: this class of discrete diffusion models is theoretically equivalent to random-order autoregressive models. From this perspective, many empirical differences between autoregression and diffusion, such as parallelism and perceptual quality, arise not from a fundamental distinction between the model classes themselves, but from details of model specification and training techniques. For example, I will present evidence that the loss reweighting technique from diffusion, when applied to any-order MDM/AR models, can yield perceptual sample quality comparable to strong continuous diffusion models.

Next, I will talk about moving beyond both (fixed-order) autoregression and (random-order) diffusion to find new powerful ways of generation. Specifically, I will focus on models that adaptively determine their generation order based on partial context. I will highlight the important role of insertion in flexible generation and planning, and a principled framework we established for scalable maximum likelihood learning of insertion-based generative models.