Ai2 introduced BenchMIRT on September 1 to examine what individual questions within LLM benchmarks actually measure. A safety benchmark might reward refusing harmful requests, but some of its questions can also depend on ordinary reading comprehension or reasoning. Averaging all those answers into one score hides the difference.
The researchers analyzed 100 LLMs, all released by March 2025, on 16 benchmarks containing more than 34,000 questions. Their method uses multidimensional item response theory, which estimates how question difficulty and different abilities contribute to a model's answer. Two broad patterns emerged across the selected tests: safety and general reasoning.
Some familiar evaluations were less narrowly focused than their names suggest. The BBQ bias benchmark aligned strongly with general reasoning in the analysis, and some copyright-related HarmBench questions also behaved more like reasoning tests. BenchMIRT additionally identified subsets of questions that preserved much of a benchmark's ability to distinguish models, even when the subset was only about a tenth of the full set.
BenchMIRT did not directly test current 2026 model families. Its discovered dimensions also depend on the selected benchmarks; a different test mix could reveal different patterns.