Code World Model Stores World State as Executable Programs, Then Hands It Off to a Video Model for Rendering

Researchers at Westlake University and Nanyang Technological University have introduced [Code World Model](https://arxiv.org/abs/2608.25927), which separates “how the world evolves” from “what the world looks like” into two computational paths. A coding agent interprets events and writes or modifies programs, while executable code updates positions, relationships, rules, and event history at every tick. Compared with predicting the next frame from short video clips alone, this state can preserve characters that have left the frame, as well as consequences that do not occur until much later.

Rather than feeding code directly into the video generator, the system deterministically compiles the updated state into a frame-by-frame proxy video. The prototype represents scene layout, object positions, occlusion, motion paths, and camera trajectories using fixed log-depth maps and semantic ID maps, while text prompts supply identity, appearance, and action semantics. The video model can therefore focus on textures and fine-grained motion instead of having to infer hidden rules at the same time.

The researchers use MiniMax-H3 Ref2VA as the visual backbone and build 9,420 paired five-second samples from 157 gameplay recordings totaling roughly 5.6 hours. Training adds rank-128 LoRA adapters to all 50 Transformer blocks, yielding approximately 596M trainable parameters, and runs for three epochs on eight H800 GPUs. Target videos contain 124 frames at 1344×768 resolution and 24 FPS, while input proxies are downsampled to 336×192, indicating that the intermediate representation is deliberately designed to retain only coarse spatiotemporal constraints.

The [project page](https://buaacyw.github.io/cwm/) shows examples of maintaining a consistent world state over long periods, switching visual styles every 60 seconds, and generating different characters and scenes. The paper claims that the model can follow object-position, camera, and motion constraints, but the evidence consists of only five sets of qualitative examples, with no measurements of control accuracy, video quality, latency, or comparative ablations. The real-world video pipeline also demonstrates only how to construct proxies from KITTI-360 geometry; actual fine-tuning still uses gameplay data. The work is therefore better viewed as a prototype interface between inspectable state and generative rendering than as a real-time open-world engine. The key next steps are to release the complete codebase, add long-horizon state stress tests, and quantify how errors introduced by agent-driven code modifications propagate into the visual output.