AI 代理與資料工程
QueryProof Uses Rule-Based Gates to Block Incorrect SQL, Raising a 7B Agent’s Business Truth Rate to 56.2%
QueryProof delegates ambiguity detection, SQL allowlisting, and post-execution validation to deterministic rules, leaving only metric resolution and SQL drafting to the model. On an 80-question frozen test set, it outperformed a 32B baseline without the same rule-based wrapper, although the advantage’s confidence interval still crossed zero after resampling by task family.
The newly released WarehouseReliabilityBench attempts to address a blind spot in Text-to-SQL evaluation: even SQL that executes successfully—or closely resembles a reference query—may use the wrong revenue definition, ignore the data cutoff date, or read deprecated columns after a schema update. The dataset contains two synthetic warehouses and 400 frozen tasks covering standard queries, ambiguous business definitions, insufficient data, schema drift, and prompt injection. For roughly half of the tasks, the correct behavior is not to return a number, but to request clarification, abstain, or refuse.
QueryProof therefore avoids an open-ended agent loop in favor of a finite-state workflow comprising `PARSE`, semantic-layer lookup, ambiguity checks, SQL drafting, AST validation, execution, and result verification. The 7B model is responsible only for selecting candidate metrics and writing SQL. Safety decisions, date parsing, column allowlists, and whether an answer should be returned are controlled by rules that read a YAML semantic layer and the physical catalog. The execution environment also uses read-only DuckDB, a `sqlglot` AST allowlist, timeouts, and result-size limits to block DDL, DML, and external-data functions.
On the single-run, 80-task test set, the pure 7B QueryProof system achieved a Business Truth Rate of 56.2%. Adding an escalation route to a 32B model reduced the score to 53.7%, while the directly prompted 32B baseline scored 30.0%. The routed version’s estimated cost per correct answer was 71% lower, and the rate at which incorrect outputs were labeled as successful answers fell from 75.4% to 35.1%. This was not a fair model-size comparison, however: the 32B baseline did not receive the same rule-based wrapper, and the ten task families were correlated. After resampling by family, the confidence intervals for both between-system differences included zero. The project has released its raw outputs, preregistered protocol, and one-command reproduction workflow, but it remains a research prototype evaluated on synthetic data. The next important step is to measure rule-maintenance costs against real semantic layers, permission models, and continuously changing warehouses—and to determine how many answers that are executable but wrong in business terms still escape deterministic checks.