AgenticGenPlan Generates Robot Planning Programs Without LLM Calls During Evaluation

The AgenticGenPlan study, published on September 24, has a coding agent explore a simulator and then generate robot control programs that can be reused across problem instances. The authors tested Claude Code with Opus 5, and Codex with GPT-5.6 Sol and GPT-6 Astra, across 28 environment types, 980 programs, and 98,000 evaluations. Once generated, each program was frozen; no large language model was called during evaluation. [Paper](https://arxiv.org/html/2609.30233v1)

The method addresses task and motion planning (TAMP): for example, deciding which obstacle to move first requires considering both whether the robot arm can reach it and whether it will collide with anything. The main experiment provided only task descriptions, the simulator interface, and Python, NumPy, and SciPy. Agents designed their own tests and revised their strategies inside isolated containers, without access to the environment's source code. Each synthesis run had a $20 model-usage budget. Each method and environment was tested in five independent runs; each resulting program was then evaluated on 100 held-out instances, with a 60-second limit per instance. [Experimental setup](https://agenticgentamp.github.io/)

Instances within the same environment can vary in object count, arrangement, and geometry, but share the same state, action, and reward definitions. Generalization here therefore mainly means adapting to new configurations of the same type of problem; it should not be read as evidence that a system can operate an unfamiliar robot. [Task definition](https://agenticgentamp.github.io/)

Across the 16 environment types with classical planner baselines, GPT-6 Astra, Opus 5, and GPT-5.6 Sol achieved average success rates of 95%, 82%, and 56%, respectively, while the planners reached 47%. Astra's average across all 28 environment types was 86%, however, so 95% is not the score for the full evaluation suite. From an engineering perspective, the method concentrates exploration and reasoning costs in strategy generation, then reuses the program on new configurations. The paper's execution-time comparison does not include the upfront synthesis cost. [Results and timing definitions](https://arxiv.org/html/2609.30233v1)

The limitations are clear: the study uses fully observable object states and does not yet test performance under perception errors. Some dynamic 3D tasks remain difficult, and the opacity of model training data means benchmark contamination cannot be fully ruled out. This preprint provides evidence from simulated environments; generalization to real robots remains to be tested. [Study limitations](https://arxiv.org/html/2609.30233v1)

The public code gives future evaluations a starting point, but using the default settings may not reproduce the main experiment. The repository notes that setting only `approach=agentic` still allows access to source code; a strict black-box option must also be enabled. The original evaluation seeds must be requested from the authors. Engineering teams should fix tool versions, access permissions, and evaluation seeds, and account separately for strategy-generation fees and execution costs to determine whether the gains apply to their own workloads. [Reproduction guide](https://github.com/tomsilver/robocode)