NVIDIA Cosmos 3 Edge Connects Visual Understanding and Robot Actions on Jetson with Dual Transformer Towers

NVIDIA has released Cosmos 3 Edge on Hugging Face, compressing the Cosmos world model to 4 billion parameters with the goal of running physical AI across hardware tiers including Jetson, GeForce, RTX PRO, and DGX. The model contains two Transformers: an autoregressive tower that processes visual and text tokens for understanding and reasoning, and a diffusion tower that processes visual, audio, and action tokens for prediction, generation, and neural simulation. Each retains its own normalization layers and MLPs, while sharing multimodal attention to map “what is happening now, what may happen next, and what action should be taken” into a shared representation.

Action data can flow in both directions: the model can derive actions and their visual consequences from the current state, or infer possible actions from observed changes. NVIDIA has also released a manipulation policy post-trained on the DROID dataset, along with checkpoints and training scripts. At a robot observation resolution of 640×360, Cosmos 3 Edge is claimed to output 32 actions per inference on Jetson Thor at 15 Hz, making it suitable for keeping decision-making close to the sensors and reducing cloud round-trip latency.

This is still not a plug-and-play general-purpose robot controller. The 15 Hz figure and VANTAGE-Bench ranking are vendor-reported, and real-world performance will be affected by cameras, control cycles, action representations, quantization methods, and safety constraints. Developers should prioritize testing end-to-end sensor-to-actuator latency, long-horizon closed-loop error, cross-robot transfer, and whether a conventional controller can take over when failures occur. The announced vLLM optimization also has yet to be released.