代理評測
AeroCopilotBench Turns Flight Procedures Into Executable State Machines, Yet the Best Agent’s Safe Pass Rate Is Still Only 72.6%
The new benchmark uses 73 abnormal cockpit tasks to test whether agents can observe state, operate systems, and comply with hard safety constraints throughout—not merely answer aviation knowledge questions. GPT-5.6 Sol achieved the highest success rate, but similar knowledge scores did not correspond to similar task success rates, showing that static question answering cannot replace interactive safety testing.
A team at Beihang University introduced [AeroCopilotBench](https://arxiv.org/abs/2608.16349), advancing aviation-agent evaluation from multiple-choice questions to a stateful virtual cockpit. Tier-1 contains 1,200 aviation knowledge questions compiled from FAA materials, the CFR, and flight manuals. Tier-2 encodes emergency and abnormal procedures for the Cessna 172S and Piper PA-44-180 into 12 scenario templates, 73 tasks, and 259 task-specific hard safety constraints.
Its AeroCopilot Operational Environment (ACOE) is not a continuous flight simulator, but a partially observable, deterministic discrete-state machine. Agents can use only 12 fixed tools to read instruments one at a time, modify writable controls, or query frozen weather, performance, and regulatory data. Hidden state, final objectives, and scoring rules cannot be read directly. Valid but dangerous actions are actually executed and permanently recorded in the trajectory, so an agent cannot erase a safety violation by restoring the correct configuration later. Each task allows up to 48 decision turns, and success requires both completing every objective and recording zero violations across the entire trajectory.
The environment is also packaged as an MCP server. Using reference action sequences for all 73 tasks, the researchers verified that the native and MCP interfaces produce identical responses and scores. This takes advantage of [MCP’s standardized interface for model tools](https://modelcontextprotocol.io/docs/getting-started/intro), although the formal model experiments still used the native function-calling path.
Twelve models ran each Tier-2 task three times. GPT-5.6 Sol ranked first with a 72.6% success rate and a 97.3% safety compliance rate; the best open-weight model was GLM-5.1 at 53.0%. More notably, Qwen3.5-397B-A17B and DeepSeek V4 Pro differed by only 0.41 percentage points in knowledge-question accuracy, yet their interactive success rates differed by 27.4 percentage points. Across 451 failures from three representative models, the main problems were omitted critical procedures, incorrect semantic priors, failure to constrain actions using the latest state, and drift during the later stages of long workflows.
The limitations are equally clear: ACOE does not simulate aerodynamics, sensor noise, fault progression, or real-time pressure, and it uses a closed-book setting without electronic checklists. The paper’s page also does not link to a public code repository. For now, it is better suited as a methodology for agent orchestration and regression testing than as evidence that models can safely enter real-world cockpits.