評測與 RAG
FinMMEval Releases Hidden Multilingual Financial QA Tests: Multiple Choice Near Perfect, Top Short-Answer Score Just 31.18%
FinMMEval at CLEF 2026 has released results for two multilingual financial question-answering tasks, showing that models can achieve 92% to 97.5% on closed-choice questions but falter markedly when generating short answers from cross-lingual evidence. Less than one percentage point separates the top four short-answer systems, also exposing the difficulty of using ROUGE rankings to distinguish factual accuracy.
FinMMEval 2026 released the technical reports and final leaderboards for Task 1 and Task 2 on July 22, providing a sharply contrasting set of stress tests for financial AI. Task 1 consists of financial multiple-choice questions in English, Chinese, Arabic, and Hindi, with 200 hidden test questions per language. The leading accuracy scores ranged from 92% for Hindi to 97.5% for English and Arabic. Participating systems combined techniques such as search or FAISS retrieval, language routing, direct option scoring, self-consistency sampling, confidence checks, and multi-model review.
Task 2 presents questions in English, while the evidence is distributed across financial reports and news in English, Chinese, Japanese, Spanish, and Greek. The 256-question hidden test requires concise answers. The top-ranked system, Calibrated Signals, achieved a macro ROUGE-1 F1 of just 31.18%, with only 0.79 percentage points separating the top four systems. Participating architectures ranged from fixed prompts, BM25, and layout-aware chunking to LangGraph, Weaviate, reranking, Python-based numerical tools, and judging agents, yet no overwhelmingly dominant common approach emerged.
The gap between the two results deserves engineering teams’ attention. Multiple-choice questions constrain the answer space to labels, allowing output validation and direct option scoring to eliminate many formatting errors. Short-answer questions additionally require cross-lingual retrieval, temporal and entity alignment, financial calculations, and answer compression; distortion at any stage can propagate into the final text. This also explains why more complex agentic RAG systems do not necessarily outperform deterministic prompting with a single constrained retry.
However, 31.18% should not be interpreted directly as factual accuracy. ROUGE-1 measures lexical overlap with a single reference answer, potentially penalizing correct synonymous phrasing while rewarding answers that contain key terms but rely on incorrect reasoning. Future evaluations need stronger support for evidence citations, numerical tolerances, entity and date consistency, and human or executable verification. For teams building Chinese-language financial RAG systems, these results are better suited to diagnosing retrieval, calculation, and generation failures than to using a single aggregate score to claim that a system is ready for high-risk decision-making.