DRACO Assigns Reinforcement Learning Credit to Each Agent Step Using Dynamic Rubrics—Without Accessing Task Answers During Training

Reinforcement learning for long-horizon agents typically requires programmable verifiers, such as unit tests or final environment states. For customer service, research, and cross-tool tasks, however, such answers are difficult to obtain. IBM Research and Carnegie Mellon University have introduced DRACO, which makes training “outcome-blind”: during training, it does not access task-success signals or reference answers. Instead, a frozen judge model dynamically generates evaluation rubrics from the task instructions and each rollout.

The rubrics for each set of trajectories are first merged and deduplicated, after which any criteria passed by every trajectory—and therefore no longer discriminative—are removed. For each criterion, the judge outputs pass, fail, or not applicable and identifies the responsible agent steps. DRACO first calculates trajectory rewards from the proportions of passed and failed criteria. After GRPO normalization, it redistributes the advantage according to step quality: better steps receive positive updates from successful trajectories, while worse steps receive negative updates from failed trajectories. The formulation preserves the total update magnitude and sign for the entire trajectory and normalizes by the number of tokens in each step, preventing verbose outputs from inherently receiving larger gradients. It requires neither a separately trained credit-assignment model nor a separate judge call at every position.

The researchers used LoRA with Qwen3.6-27B and Qwen2.5-32B-Instruct for GRPO training. The main experiments used only 90 AppWorld tasks, with each run conducted on eight H100 GPUs; rubric generation and scoring were handled primarily by GPT-5.4. On AppWorld test-normal, DRACO achieved a Task Goal Completion score of 85.3—15.9 points above the untrained model and 5.3 points above GRPO using sparse ground-truth rewards. In zero-shot transfer to τ-bench Banking, it also outperformed the base model by 5.3 points. The code, prompts, hyperparameters, and evaluation pipeline have been released under the Apache-2.0 license.

The central risk is being “consistently wrong”: the dynamic rubrics have no independent answers against which they can be verified, and the paper does not report calibration agreement between the judge and human annotators. Improvements in final task performance also do not prove that credit was assigned to the correct steps. Future work should focus on robustness across judge models and human evaluations, variance across repeated training runs, and whether the cost of judge calls remains sustainable on larger task sets.