多模態代理與評測
RuleMaze Breaks Visual Planning Into Tool Calls, Raising a 3B Model’s Success Rate on Unseen Rules From 56.3% to 88.0%
RuleMaze uses executable verifiers to test whether multimodal models can understand mazes, follow natural-language rules, and complete path planning. By separating perception, movement, and rule verification, DMP substantially outperforms standard fine-tuning, though testing remains limited to synthetic grids and the verifiers underwent manual correction.
A team including researchers from Peking University has released RuleMaze, an effort to move multimodal models’ ability to “follow rules” from textual responses to executable spatial actions. Each task provides a maze image, a goal, and natural-language constraints—for example, after entering an orange area, the next move must be upward. The model’s sequence of up, down, left, and right actions must both reach the destination and avoid violating any rules along the way. The dataset is divided into RegularMaze and QuestMaze, the latter featuring keys, health points, and dynamic objectives. Each environment contains 3,600 training samples and 400 test samples, including 300 tasks that use rules not seen during training.
The data is generated through a Language–Logic–Function pipeline: an LLM first devises a rule, translates it into a logical expression, and then produces a Python verification function that accepts a complete trajectory. The system enumerates maze paths and retains only configurations with at least one rule-compliant path as well as rule-violating distractor paths. The automatically generated verifiers were 93.3% accurate before manual inspection. The authors subsequently reviewed them one by one, meaning the final dataset was not created through a fully automated process.
The team also trained a Disentangled Multimodal Planning (DMP) controller based on Qwen2.5-VL-3B. Rather than completing all reasoning in a single text output, the model alternates between calls to LocateStart, InspectGrid, ExecuteMove, VerifyRule, and VerifyEndpoint. Tools handle perception and state transitions, while the controller primarily learns when to observe, move, and verify. On RegularMaze tasks with unseen rules, standard supervised fine-tuning (SFT) achieved a 70.3% exact-match rate for complete paths, compared with 90.0% for DMP. On QuestMaze, the rate rose from 56.3% to 88.0%. For difficult QuestMaze rules, the gap widened further, from 40.0% to 84.0%. The public project includes data generation, LoRA training, and per-checkpoint evaluation workflows. The reference configuration requires approximately four RTX 4090 GPUs and 12 hours of training.
These results provide stronger evidence that an external state machine paired with executable checks outperforms packing every constraint into the model than that a 3B model has learned to generalize real-world rules. Unseen rules still come with dedicated verifiers at test time. Moreover, the mazes are discrete, fully observable, and enumerable environments, far removed from the sensor noise encountered in road or robotics scenarios. The next step is to determine whether verifiers that have not been manually corrected can remain reliable, and whether the method retains its advantage under continuous control, conflicting rules, and partial observability.