Original title: Open Problems in AI Risk Modeling: Insights from a Workshop on the Technical Foundations of AI Risk Modeling
Krystal Jackson, Deepika Raman, Jakub Kryś, Sean P. Fillingham +7 · September 2026
Governments want to measure the societal dangers of advanced AI, but experts lack strict math-based tools to do so. Researchers met with 22 experts to figure out how to model these risks. They reviewed five research fields, including cybersecurity and probabilistic risk, and compared two main approaches: estimating risks from specific scenarios and using Bayesian networks to set safety thresholds. They found major hurdles in how we build, test, and govern these models. To make real progress, the authors say we cannot just rely on numbers. We must connect quantitative models with outside evaluations, clear tiered disclosures, and lasting organizations that can regularly update the risk models as systems change.
Imagine you're baking a cake for a competition where the judges' preferences change without notice. Each time you check the recipe, it seems fine, but later you discover that some ingredients behave unpredictably based on the oven's mood, and you can't be sure the cake will turn out well. In this paper, researchers are dealing with the same issue: they need to assess risks from AI systems that behave similarly unpredictably, and they require models that can handle changing conditions and uncertain evidence to ensure safety.
Regulators are increasingly requiring risk assessments for AI systems, but without reliable quantitative methods, they cannot effectively prevent potential catastrophic disasters. This affects society as AI becomes more powerful and integrated into critical areas.
This work does not establish new models or validate existing approaches; it identifies challenges and open problems, leaving the development of rigorous methods for future research.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Open Problems in AI Risk Modeling: Insights from a Workshop on the Technical Foundations of AI Risk Modeling — Krystal Jackson, Deepika Raman, Jakub Kryś et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.