視覺模型／自監督學習
LeVJEPA Removes the Target Encoder and Predictor, Cutting Video Pretraining Compute to as Little as One-Twentieth of V-JEPA 2
LeVJEPA uses a single encoder, a cross-view invariance loss, and SIGReg to prevent representation collapse, eliminating the need for an EMA target network, stop-gradient, or a pixel decoder. Under the same data and training schedule, the authors report a 5.6× to 20.8× compute difference, though the results remain limited to frozen-encoder benchmarks based on models retrained by the team.
Video self-supervised learning typically prevents representation collapse in one of two ways: V-JEPA-style methods maintain an exponential moving average target encoder, apply stop-gradient, and use a capacity-constrained predictor, while VideoMAE-style methods reconstruct masked pixels. LeVJEPA instead uses the same ViT encoder to process global and local views of a 16-frame video clip, minimizes the mean squared error between the `[cls]` embeddings of those views, and then applies SIGReg to constrain the embedding distribution toward an isotropic Gaussian, ruling out the trivial solution in which every sample maps to the same vector. The resulting training graph contains only the encoder and projector, with gradients flowing through both sides of the comparison.
Additional savings come from randomly dropping input tokens. The authors trained ViT-S, ViT-B, and ViT-L for 240 epochs on the same 20% subset of Kinetics-710. Compared with their own retrained V-JEPA 2 models, LeVJEPA used 5.6× to 20.8× fewer total FLOPs while achieving similar or higher accuracy. In an equal-compute comparison using ViT-B, LeVJEPA achieved 61.0% top-1 accuracy in frozen probing on ImageNet-1K, versus 51.6% for V-JEPA 2; on Kinetics-400, the scores were 44.6% versus 40.7%. On Something-Something-v2, however, LeVJEPA trailed with 40.4% versus 42.5%. This shows that the claimed “20×” figure is not a fixed speedup available across all models and datasets, nor does it imply a universal advantage in end-to-end video understanding.
LeVJEPA can also use block-causal attention: bidirectional attention within each frame, while attention across frames is restricted to the current and past frames. Testing showed no decline in frozen-probe accuracy, and newly added video frames do not require historical frames to be re-encoded, making the approach particularly useful for streaming perception and autoregressive world models. The project has released its code and checkpoints. The next things to watch are independent reproductions, scaling results on larger and more diverse video datasets, and whether low-cost pretraining translates into practical gains for detection, tracking, planning, and robotic control.