ERUnderstand Turns ER Diagrams into Structured Tests, Exposing Clear VLM Failures on N-ary Relationships and Weak Entities

ERUnderstand formally benchmarks a task common in database engineering but insufficiently covered by mainstream multimodal evaluations: reconstructing machine-readable data models from ER/EER diagram images. The dataset contains 2,960 diagrams, 17,415 entities, 15,305 relationships, and 66,856 attributes. Its sources include educational materials; real-world schemas from Spider, BIRD, and Northwind; and controlled samples generated through an LLM-assisted Graphviz pipeline. Each diagram is paired with a standardized JSON representation recording elements such as primary keys, cardinalities, weak entities, inheritance, and multivalued and composite attributes. This makes it possible to evaluate whether the structure was actually reconstructed, rather than merely whether a model can answer questions about an image.

The ten vision-language models tested already show solid recognition of basic elements, with F1 scores above 0.74 for common entities, relationships, or attributes. Performance declines rapidly, however, on fine-grained notation. The paper’s abstract reports minimum F1 scores of 0.28 for weak entities, 0.14 for multivalued attributes, and 0.07 for relationships involving three or more entities. More detailed analysis shows that non-reasoning models score below 0.01 F1 on N-ary relationships. Reasoning models improve this to about 0.31, but that remains insufficient for directly producing trustworthy schemas. Reasoning boosts overall performance by roughly 15% to 25%, but does not eliminate topology errors.

The error patterns are particularly important for engineering applications. Models tend to hallucinate relationships between nearby entities that appear semantically plausible, while missing nodes that are farther apart but genuinely connected by lines. When researchers replaced labels with random strings or shuffled semantic roles, performance declined across models. This indicates that the models partly rely on linguistic priors such as “ID is usually a primary key,” rather than consistently reading underlines, double borders, dashed lines, and connection topology. Interactive self-checking can reduce omissions, but is less effective at correcting underlying visual-structure reasoning errors.

The dataset and evaluation code are publicly available on GitHub and can be used to test database copilots, legacy-system migration tools, or document-parsing pipelines. However, the independent dual-annotation experiment covered only 30 diagrams and achieved an average agreement rate of 89.72%. The evaluation also used a single prompt with zero temperature and did not compare the models against conventional pipelines combining OCR, shape detection, and graph tracing. Engineering teams should therefore continue treating VLM output as a candidate schema and supplement it with syntax validation, connection tracing, and human review.