影片生成與模型架構
V-RAE Uses Frozen Visual Representations as Video Latents, Converging Up to 6× Faster in K600 Generation Training
Instead of training a video VAE solely for pixel reconstruction, V-RAE adds temporal compression and a decoder on top of semantic features from models such as DINOv3, SigLIP2, and V-JEPA 2.1. The authors found that the new latents improve both generation and future-frame prediction, though they have so far demonstrated only low-resolution, short-clip, class-conditional generation.
Current video generation models typically use a VAE to compress pixels into latents, then train a diffusion transformer to learn their distribution. The problem is that VAEs mainly optimize for texture reconstruction: a high reconstruction score does not necessarily mean the latents are easy to generate. V-RAE instead uses a frozen visual foundation model as its encoder, evaluating representations from DINOv3, SigLIP2, EUPE, and V-JEPA 2.1. A learnable attention-pooling module then merges features from adjacent time steps, preserving semantics while shortening the sequence. A spatiotemporal Transformer decoder with 3D RoPE reconstructs multiple frames from each latent step. During training, only the pooling module and decoder are updated. At the generation stage, V-RAE is frozen while a rectified-flow DiT is trained on its latents.
On Kinetics-600, the authors reported a reconstruction rFVD of 2.13 for the V-JEPA 2.1 variant. Using the same DiT and training configuration, the best variant achieved generation gFVD scores of 117.86 on UCF101 and 19.16 on K600, while converging up to six times faster. More notably, conventional rFVD showed Pearson correlations of only 0.200 and 0.473 with downstream generation quality. The team therefore proposed tFVD, which measures coherence after interpolating along the latent timeline and decoding the result. Its correlation with generation quality rose to 0.621 and 0.919.
This suggests that video tokenizer selection should not be based solely on pixel reconstruction: semantic structure and the temporal geometry of the latent space can also directly affect DiT sample efficiency. However, the experiments remain focused on 256×256 resolution, 17-frame evaluations, and class-conditional generation, while using 100-step Euler sampling. High-dimensional representation latents may also increase memory usage and attention costs. As of verification, the [project page](https://v-rae.github.io/) provides result demonstrations, but the paper entry does not list any publicly available code or weights. The next questions are whether the same advantages will hold for text-conditioned long-form video, lower sampling-step counts, and large-scale open-source reproductions.