機器人與模型評測
Physical Robot Arm Tests Show GPT-6 Astra Scoring 19/20 on Pick-and-Place, but Just 2/20 on Precision Insertion
RoboCurve had GPT-6 Astra and Claude Fable 5.1 control the same dual-arm robot using an identical agent policy, revealing a sizable gap on a simple pick-and-place task. When asked to precisely insert a puzzle piece into a recess, however, both models succeeded only twice, showing that vision-language agents remain constrained by contact control and error correction.
Independent evaluation team RoboCurve connected GPT-6 Astra to two I2RT YAM six-degree-of-freedom robotic arms and used three camera views and the open-source Inspect Robots agent framework to perform two physical manipulation tasks. The first required picking up a red block and placing it in a bowl. Astra succeeded in 19 of 20 trials, compared with eight for Claude Fable 5.1 and one for Fable 5. Astra took an average of 2.5 minutes per trial, generated about 2,100 tokens, and incurred an estimated model cost of $0.94; Fable 5.1 averaged 6.8 minutes, 12,900 tokens, and $2.12.
The second task, involving precision insertion, changed the picture. The models had to grasp the center knob of a circular puzzle piece, align it with a recess in a board, and insert it. Astra and Fable 5.1 each succeeded in only two of 20 trials. Astra often managed to move the object above the recess but stalled during the final pose adjustment, contact, and insertion stages. This suggests that better scene understanding, faster planning, or greater token efficiency does not automatically translate into more reliable closed-loop precision control. For embodied agents, end-effector error, depth perception, control frequency, and local recovery after failure are equally critical.
This evaluation is more auditable than a single demonstration video: RoboCurve published per-trial scores, transcripts, videos, and rerun data for all 120 trials. Inspect Robots stores parsed configurations, Git commits, and package versions, and validates the observation space, action space, and control rate before execution. However, each model was tested only 20 times per task, scores were assigned manually according to the highest completion stage reached, and costs were estimated using API prices at the time. The results are sufficient to demonstrate the difference between 19 and eight successes, but not to establish that the two models scoring 2/20 are genuinely equivalent. The next priority should be testing with more objects, occlusions, and perturbations, along with automated force-sensing or geometry-based scoring, to avoid extrapolating a high success rate on one simple task into claims of general-purpose robotic capability.