PDDLCoder Lets Agents Iteratively Fix PDDL, Achieving an 89.6% Executable Success Rate on 106 Unseen Planning Problems

When large language models perform long-horizon planning, they often generate action sequences that violate preconditions, omit necessary steps, or cannot reach the goal. PDDLCoder instead positions the model as a “formalization agent”: it creates PDDL domain and problem files from natural-language descriptions, then hands them to Fast Downward to find a plan. The agent can run for up to 50 iterations and independently choose among eight tools, including file editing, VAL syntax checking, planner translation, plan generation, and semantic feedback. Unlike a fixed repair pipeline, it separately handles syntax errors, unsolvable models, and plans that are semantically non-executable.

The researchers also released NL-pddlgym, which contains 23 domains and 711 problems. The test set fully holds out four domains—Elevator, Hanoi, Ring and Peg, and Satellite—with 106 problems in total; the longest plan contains 557 steps. Evaluation does not compare generated text or PDDL against reference answers for similarity. Instead, each plan is executed step by step in the pddlgym environment, and it counts as successful only if every action is executable and the final state satisfies the goal. With DeepSeek V4 Flash, PDDLCoder solved 95 problems, or 89.6%. Direct chain-of-thought planning achieved 74.5% on the same test, while adapted versions of existing formalization methods reached at most 45.3%.

The improvement does not come for free: successful configurations used an average of about 173,000 input tokens, mainly because the complete PDDL, plans, and tool outputs were repeatedly fed back to the model. Performance also varied dramatically across models. Gemma 4 31B achieved a 61.3% success rate, gpt-oss-120B reached 52.8%, and Qwen3.6-35B-A3B managed only 10.4%, while two other evaluated models scored zero. This shows that a symbolic planner can guarantee only that a plan is valid within “the world written by the model,” not that the world faithfully represents the natural-language description.

From an engineering perspective, a key question is whether this kind of executable intermediate representation can be extended to probabilistic, partially observable, or continuous-state environments. In the current results, each model, problem, and configuration was run only once, so sampling variance was not estimated; the test also includes only four held-out domains. The good news is that the authors have released the code, Docker configuration, prompts, per-problem results, and an MIT license, allowing external teams to rerun the experiments directly and determine whether the high token cost produces consistent gains.