模型評測
TriviaRoomQA Tests Everyday Knowledge Across Six Languages, Revealing Systematic Gaps in 30 Open Models on Pop Culture and News Questions
The new benchmark uses 3,300 parallel questions in six languages and 5,340 French questions to test whether models can access the same everyday and long-tail knowledge across languages. Larger models are more accurate overall, but still fail to eliminate gaps caused by pop culture, recent events, and language switching.
Researchers at INRIA Paris introduced TriviaRoomQA, which uses human-written trivia questions to evaluate open-weight models on everyday, cultural, and long-tail knowledge. The dataset contains 3,300 parallel multiple-choice questions, each available in English, French, Italian, Spanish, German, and Dutch, plus another 5,340 questions in French. The full collection spans 288 topics, 24 broad categories, three difficulty levels, and temporal and geographic labels, making it better suited to identifying knowledge gaps than academic exam benchmarks that report only an aggregate score.
The researchers evaluated 30 European, Asian, and North American models ranging from 7B to 70B parameters using the log-likelihood of answer candidates. The models performed better on history, geography, mathematics, and other encyclopedic topics, but lagged significantly on everyday cultural questions involving celebrities, music, film, and news. Models below 20B achieved accuracy between 0.35 and 0.64, while larger models ranged from 0.41 to 0.72. Although increasing parameter counts improved average performance, it did not eliminate cross-lingual inconsistencies. The study also found that human performance typically declines gradually as questions become more difficult, whereas models may remain near the lower end of the scale from beginner through expert questions in unfamiliar domains. This suggests that the issue is more akin to entire regions of missing knowledge than insufficient reasoning depth.
This is particularly important for multilingual RAG, localized search, and regional customer support: passing an English-language evaluation does not prove that a model can retrieve the same fact when queried in Chinese or another language. In engineering workflows, performance differences across parallel questions can help identify biases introduced by the tokenizer, training data, retrieval indexes, or prompt language. However, the benchmark covers only multiple-choice questions and open-weight models, and its six-language subset is limited to European languages. Its findings therefore cannot be directly generalized to Chinese-language generation, tool-using agents, or closed-source frontier models. Future work should add Chinese and other languages that use non-Latin scripts, while comparing whether parametric memory alone, RAG, or real-time search genuinely narrows the gap.