機器人與具身 AI
Isaac 0.5 Unifies Video Understanding and Robot Control, but Its Key Future-Perception Training Objective Remains Closed
Perceptron AI has released the weights for the 36-billion-parameter Isaac 0.5 model, along with a LeRobot integration, enabling a single sparse backbone to output language, spatial coordinates, and continuous or discrete actions. The company claims that general video can dramatically reduce the need for teleoperation data, but the figure is extrapolated from offline action loss, and the core self-supervised objective remains proprietary.
Perceptron AI has released Isaac 0.5, a 36-billion-parameter model that combines video understanding, embodied reasoning, spatial localization, task-progress estimation, and robot control within a single backbone. The team has also provided model weights, training and inference code, a LeRobot integration, a reference policy server, and deployment materials covering more than 35 robot configurations.
Isaac is based on the Qwen family of vision-language backbones. Each MoE layer contains 256 routed experts and a null route; different text, image, state, and action tokens can select between zero and eight experts, in addition to shared experts and a residual path. According to the technical report, this ultra-sparse configuration activates an average of only about 2.5 billion parameters per token. Discrete control uses a separate vocabulary of 2,048 FAST action tokens. For continuous control, the backbone state conditions a Flow expert, which generates action chunks through a 36-layer diffusion transformer. The model can predict the next action chunk while the current one is being executed, reducing latency in closed-loop control.
The training data includes 1 million hours of general video, 375,000 hours of first-person video, 375,000 hours of UMI handheld-gripper video, and 100,000 hours of robot, game, and simulation experience. At a fixed offline action-loss threshold of 2.50, the authors estimate that increasing general-video data from 1,000 hours to 1 million hours reduces the required teleoperation data from 5,884 hours to 28 hours.
This 210-fold figure is not a physical-robot success rate, nor does it demonstrate that video can replace demonstration data for specific hardware at the same ratio. More importantly, the method used to construct future frames as “percept” targets—and the corresponding loss implementation—has not been released. A clean checkout is also currently insufficient to run the complete training or inference pipeline, which requires additional configuration of components such as mHarmony and TensorStream. Future scrutiny should focus on cross-robot reproducibility, real-world closed-loop success rates, and whether the proprietary objective prevents full reproduction of the training process.