3D 生成與具身 AI
iARCS Uses LLMs to Generate Reward Programs for 3D Scenes, Reducing the Object Collision Rate from 52.67% to 40.45%
iARCS first corrects collisions and walkability issues in an indoor scene generator, then compiles natural-language requirements into executable reward functions for a second stage of reinforcement learning. In 3D-FRONT experiments, it outperforms MiDiffusion on geometric metrics, but each new requirement requires a separately trained LoRA, and the authors have not yet released an implementation.
3D scene generation is often evaluated using distributional metrics such as FID to measure visual similarity. For robot training or spatial design, however, furniture collisions, out-of-bounds objects, and inaccessible pathways can render seemingly plausible data unusable. [iARCS](https://arxiv.org/abs/2608.06161) reformulates these non-differentiable constraints as diffusion reinforcement learning: in the first stage, it uses general geometric rewards to correct the base MiDiffusion model; in the second, Gemini converts user descriptions into measurable conditions and Python reward programs.
During training, the system treats diffusion denoising steps as an MDP and updates the model using DDPO. Every ten epochs, it reads the reward statistics and asks the LLM to revise the functions or their weights. This is not a single model that directly accepts arbitrary text and generates every type of scene in a zero-shot manner: the paper trains a separate LoRA policy for each task condition. The two-stage design is intended to first establish a shared baseline for collisions, boundary compliance, and walkability, preventing task-specific rewards from sacrificing basic physical quality.
On the 3D-FRONT indoor dataset, iARCS achieves an object collision rate of 40.45%, compared with MiDiffusion’s 52.67%. The proportion of scenes containing collisions falls from 81.67% to 64.63%, the out-of-bounds rate drops from 5.89% to 3.04%, and the reachability rate rises from 85.7% to 87.82%. It also generates 4,000 scenes to fine-tune MiDiffusion again, showing that synthetic data can improve the base generator. [MIT-SPARK’s 3D-FRONT tools](https://github.com/MIT-SPARK/ThreedFront) provide the same data format and evaluation scripts, offering a starting point for external reproduction.
However, FID worsens from MiDiffusion’s 1.34 to 1.60, indicating that geometric usability and similarity to the data distribution still involve a trade-off. More importantly, LLM-generated programs may fail to resolve ambiguous requirements, while repeated RL increases the training cost for each condition. No official code, weights, or cross-dataset validation are currently available. Future work should examine whether the model can exploit loopholes in generated rewards and whether a single adapter can generalize to unseen combinations of constraints.