AI 評測／資料科學代理
DeepChart Audits the AI Charting Data Pipeline Stage by Stage, Showing That Executable Charts Do Not Guarantee Correct Data
DeepChart evaluates data extraction, numerical derivation, and chart rendering separately across 1,482 instances, exposing data hallucinations concealed by appearance-based scores. With ultra-long 10-K inputs, the average source-data F1 score falls from 0.385 to 0.167, showing that larger context windows cannot replace evidence retrieval and localization.
Researchers from the University of Science and Technology of China and Huawei have released DeepChart, which divides data-science chart generation into three stages: Extract, Reason, and Visualize. A model first identifies source values in research papers, 10-K filings, or industry reports, then calculates the derived data required by the chart, and finally outputs executable Python or HTML code. Rather than evaluating only the finished chart, the benchmark uses source-data F1, derived-data F1, Visual Accuracy Score (VAS), and Execution Rate (ER) to pinpoint errors.
The dataset contains 724 base questions and 1,482 context variants spanning 30 chart types. Text inputs average 220,600 tokens, while report-based multimodal inputs average 218 pages. Zero-shot experiments show that industry-report tasks achieve an average ER of 0.782 and a VAS of 0.447, but source-data and derived-data F1 scores of only 0.149 and 0.276, respectively. In other words, the code may run and the chart may look plausible even when the underlying numbers come from incorrect evidence or calculations.
The full 10-K stress test makes the problem even clearer. When the input changes from table-centered Normal context to Ultra-Long context, average VAS drops from 0.421 to 0.269, source-data F1 from 0.385 to 0.167, and derived-data F1 from 0.249 to 0.139. For engineering teams building analytics agents, this means retaining auditable intermediate data, validating calculations, and performing retrieval or evidence reduction before generation, rather than checking only whether the chart renders or the code executes successfully. One limitation is that the ultra-long-context test contains only a fixed set of 50 instances, while numerical F1 is merely an approximation of faithfulness. In addition, the original PDFs for some commercial reports are not distributed with the GitHub repository and must be obtained separately from Zenodo.