RAG 與金融 AI
FinRank Tests Financial RAG With 6,021 Confusing Passages; 7B Embedding Model Reaches Just 44.8% Recall@10
FinRank requires systems to find evidence in SEC filings that matches the correct company, period, and disclosure context, rather than merely generating a numerically correct answer. The dataset and evaluation code are public, but the noncommercial license and annotation limitations will affect how they can be adopted.
FinRank shifts the focus of financial question-answering evaluation from “is the answer correct?” to “does the answer come from the correct disclosure?” The same company may repeat similar figures across different quarters, while different companies may use nearly identical accounting or risk language. Even when a model produces a plausible answer, it may still cite the wrong year, form, or legal entity. The new benchmark therefore separates retrieval, reranking, and hard-negative identification into distinct tasks, preventing misplaced evidence sources from being obscured by the final answer.
The public dataset contains 1,185 manually written question-answer pairs drawn from 10-K and 10-Q filings submitted by 22 US companies between 2024 and 2025, spanning the pharmaceutical, oil and gas, and automotive industries. Each example includes a reference answer, an average of 1.96 supporting passages, and negative examples selected from the same filing, other reporting periods for the same company, or peer-company filings, for a total of 6,021 hard negatives. The global retrieval corpus consists of 5,230 deduplicated supporting or distracting passages rather than a complete repository of SEC filings. The results therefore measure controlled passage ranking, not end-to-end document retrieval capability.
Baseline results show that scaling up an embedding model does not automatically resolve source confusion. In the experiments, the best 7B instruction-tuned embedding model achieved only 44.8% Recall@10 on the combined corpus. Encoders with fewer than one billion parameters outperformed BM25 by no more than 3.5 points, while a finance-domain-adapted model underperformed BM25 by 9.7 points. When randomly selected negatives were replaced with manually chosen confusing negatives, pairwise classification accuracy fell by 13.0 to 20.5 percentage points across methods, indicating that many existing RAG scores primarily benefit from distractors that are easy to eliminate.
The repository provides JSONL files, fixed data splits, validators, baseline code, and a traceable patch history, making it suitable for testing hybrid retrieval, cross-encoder reranking, and query planning with metadata constraints. However, 442 negative passages have text identical to supporting passages for other questions, so users must decide whether to filter them based on the provided labels. Each example also has only one primary annotator, and no formal inter-annotator agreement analysis was conducted. The dataset is licensed under CC BY-NC 4.0, so commercial financial systems cannot treat it as an unrestricted training set.