FTF-bench Shows Multimodal Agents Sacrifice Necessary Conditions; FTF-RL Introduces Hierarchical Reward Training

The “First Things First” study targets a failure mode often obscured by general instruction-following scores: an agent may select a seemingly better product or route while violating the user’s non-negotiable conditions. The team created FTF-bench, which contains 3,649 interface screenshots paired with natural-language requests across e-commerce, hotel booking, maps, and ride-hailing. The dataset includes 2,098 e-commerce tasks, 1,093 hotel-booking tasks, and 458 maps and ride-hailing tasks, with conditions and answers verified by human annotators.

The tasks are divided into three decision structures. In the first, only one candidate satisfies all necessary conditions. In the second, multiple candidates qualify, allowing the agent to rank them according to “nice-to-have” preferences. In the third, no candidate satisfies the necessary conditions, so the correct behavior is to decline to provide an answer. This design separates condition parsing, feasibility assessment, and preference optimization, making it possible to determine whether a model misread the interface, confused condition priorities, or selected an option despite the absence of a valid solution.

The authors also propose FTF-RL, trained on Qwen2.5-VL models of different sizes. Its reward function simultaneously checks the output format, final answer, and whether each requirement is correctly classified as necessary or preferential, while encouraging the model to produce intermediate reasoning. The paper reports improved task success rates across all three scenario types, as well as transfer gains on external benchmarks including LogicVista, MathVision, and InfoQA. Providing the base model with correct condition labels also substantially improves performance, supporting the conclusion that part of the bottleneck lies in requirement parsing rather than visual recognition alone.

For teams building agents for procurement, travel, or interface operation, these findings suggest that evaluation cannot rely solely on final preference scores. Test sets should explicitly include cases with no valid solution. At runtime, systems can first separate hard constraints from soft preferences using structured fields, then prevent the planner from using the latter to compensate for violations of the former. However, the GitHub repository currently contains only documentation and explicitly states that the code, documentation, and pretrained models are still being prepared. External researchers therefore cannot yet fully reproduce the training process or assess biases in data generation. For now, FTF-RL should be regarded as a research result rather than a delivered training tool.