Evaluating probabilistic decision making by LLMs: a decision-theoretic framework and application to clinical diagnosis

Many recent efforts use language models to assist with high-stakes decisions, for example via triaging medical symptoms, or by helping applicants navigate social services. Such applications require models to engage in decision making under uncertainty, weighing the costs and benefits of different courses of action under uncertainty about the user's true state (e.g., a patient's true diagnosis). How can system-builders evaluate, diagnose, and improve such capabilities? We propose a revealed-preference approach which elicits decisions alongside probabilistic beliefs from a model and infers the utility function which would rationalize those decisions. We then apply this framework to medical diagnosis tasks and trace how failures are jointly attributable to the beliefs and utilities that explain the model's actions.