最新模型
BenchMIRT Uses Multidimensional Item Response Theory to Dissect LLM Evaluations, Revealing That Safety Scores Can Conflate Reasoning Ability
Ai2 has open-sourced BenchMIRT, which separates latent capabilities such as safety and general reasoning based on model response patterns across more than 34,000 questions. Experiments show that some safety benchmarks primarily reflect reasoning performance and that a carefully selected 10% of questions can largely preserve capability rankings, though the data covers only models released by early 2025.