Abstract
Many problems in large language models -- constrained generation, reasoning and planning, information extraction, alignment with human feedback -- can be understood as Bayesian inference tasks involving intractable posterior sampling under an (autoregressive or diffusion) LLM prior. I survey various such problems and the methods from the probabilistic inference literature that can be used to solve them, including Monte Carlo methods, amortised variational inference using deep RL, and hybrid techniques, and their benefits for learning generalisable yet uncertainty-aware reasoners and planners. Conversely, extracting structured knowledge -- such as relational or causal information -- from pretrained language models presents challenges due to the inherent limitations of prompting methods, which lack guarantees of logical or distributional consistency and calibrated uncertainty measures. It is hypothesised that the same amortised inference techniques can enable faithful extraction of structured knowledge from LLMs, by constructing a symbolic structure that is consistent with a fixed language model's predictions. We conclude by discussing the implications of this direction of research for the development of aligned and probabilistically-guaranteed-safe AI systems.