多模態模型
Cosmos 3 Edge Shrinks the World Model to 4B, Handling Vision, Text, and Robot Actions on a Single GPU
NVIDIA has released the Cosmos 3 Edge weights and inference pipeline, combining autoregressive reasoning and diffusion-based multimodal generation in a 4B Mixture-of-Transformers model. The model accepts text, images, and action trajectories, and a robot policy variant fine-tuned on DROID data is also available, though physical consistency and edge-device latency still require independent verification.
NVIDIA has released Cosmos 3 Edge, scaling down the Cosmos 3 world model—originally designed for data centers—to 4B parameters and positioning it as a starting point for physical AI on a single GPU and at the edge. The model retains the Mixture-of-Transformers architecture: an autoregressive Transformer handles discrete text and reasoning outputs, while a diffusion Transformer generates continuous signals such as images, video, audio, and actions through iterative denoising. By placing both generation paths within a shared framework, the goal extends beyond visual question answering: the system can infer future states from observations and then generate simulated content or control sequences.
The released package includes the general-purpose Cosmos3-Edge and Cosmos3-Edge-Policy-DROID, which is fine-tuned on DROID robot manipulation data. The input interface supports text, RGB images, and two-dimensional action trajectories; the latter can represent frame-by-frame control values such as joint positions, gripper states, or camera poses. The accompanying Cosmos Framework provides inference, training, and evaluation pipelines, along with training recipes for DROID, LIBERO, and physical plausibility scoring. This allows researchers to compare visual reasoning, world generation, and policy learning within the same architecture without having to connect separate language, video, and control models.
“Edge” still does not mean the model can operate in real time on every embedded platform. Actual VRAM requirements, quantization methods, image resolution, video length, and the number of diffusion steps can all significantly affect latency. The model card also warns that the model may misinterpret spatial geometry, causal relationships, temporal order, and object states. Engineering teams should next focus on end-to-end benchmarks on Jetson and desktop GPUs, closed-loop manipulation success rates, and whether controllers can use safety constraints and real-time sensing to prevent hallucinated future predictions from becoming physical actions.