代理評測
TREK Validates Travel Agents With Executable Rules; the Best Model Achieves Just a 46.2% Task-Perfect Rate
Instead of using an LLM to judge travel plans, TREK verifies entities, budgets, opening hours, transportation, and implicit requirements individually. Even the best of 15 agents fully passed only 46.2% of solvable tasks, showing that locally correct answers remain difficult to assemble into executable itineraries.
Released on July 29, TREK frames travel planning as a reproducible tool-agent benchmark rather than asking another language model to subjectively judge whether an itinerary is reasonable. The benchmark contains 800 multi-constraint tasks: 533 are solvable, while 267 are constructed to be provably unsatisfiable. The latter also include typed reasons—such as routing, entity, or budget constraints—requiring agents to correctly refuse when a task cannot be completed instead of forcing an answer.
The tasks are built on a fully synthetic, internally consistent knowledge base covering 375 cities, 13 traveler personas, and 212,530 records for flights, hotels, attractions, and car rentals. Agents cannot query the tables directly and must instead search and submit plans through a REST API sandbox with structured errors. The evaluator checks nine dimensions individually, including specified entities, cities and budgets, unstated amenity requirements, whether an item actually exists in the database, opening hours, day-by-day transportation feasibility, and refusal reasons for unsatisfiable tasks. A result counts as task-perfect only when every condition is satisfied simultaneously.
After testing 15 LLM agents, the team found that the strongest model, GPT-5.6, achieved a 46.2% task-perfect rate on solvable tasks. The median was just 6.6%, and the lowest score was zero. Implicit persona requirements were a shared bottleneck across all models. This result is closer to real-world deployment than a single “information correctness” score: even if 90% of an itinerary is reasonable, it still fails the user if a flight does not exist, a transfer is impossible, or the total cost exceeds the budget.
The repository includes the tasks, knowledge base, gold answers, agent runner, and deterministic evaluator. All 800 gold answers can be rescored offline and receive 1.0, demonstrating that the scoring ceiling is attainable. However, synthetic prices and schedules eliminate the data drift, anti-scraping restrictions, and payment states found on real websites, while the official agent runner is primarily designed for Amazon Bedrock. The current results were also generated by the authors themselves, and the GitHub repository showed no signs of external adoption at launch. The next questions are whether third parties can reproduce the leaderboard and whether the model rankings will hold after introducing real-time inventory, tool failures, and cross-language requirements.