When Prediction Becomes Policy: What Do Risk Models Actually Learn?

Predictive risk models in child welfare are often defended as forecasting devices. In the field, they behave like policy. They reorganize triage, supervision, service allocation, and documentation, and they thereby reshape the administrative record that later stands in for “ground truth.” This talk starts from that pragmatic observation and asks a simple but consequential question: what do risk models actually learn when “risk” is produced through street level bureaucracy, resource constraints, and discretionary judgment rather than revealed as a stable label?

I synthesize a multi method research program grounded in practice, informed by work with child welfare agencies in Wisconsin and carried forward through collaborations in Ontario. The programme spans a systematic review of deployed child welfare algorithms and their targets, ethnographic study of mandated tools embedded in organisational routines, and computational analyses of case narratives that surface invisible labour, shifting needs, and institutional power over time. Across studies, the same pattern recurs: predictors and proxy outcomes frequently encode agency response and surveillance intensity, and the semantics of “risk” drift across the life of a case in ways that standard prediction setups do not represent.

I then extend this argument beyond predictive risk models to contemporary LLM based workflows. In recent work, we use a local LLM together with practitioner labelling to identify service plan goal relevance in case notes, and we trace thematic trajectories over time. The result is both diagnostic and cautionary: as cases become longer and more complex, LLM judgements become less reliable, and the notes increasingly document emergent concerns that sit outside formal service plans. I close by reframing evaluation as an intervention problem and by offering a practical audit and design orientation for aligning targets, data, and governance triggers with accountable service pathways rather than standalone risk scores.