最新模型
CMuon Splits Fused DiT Weights Before Orthogonalization, Training a 675M Model Twice as Fast as AdamW
CMuon finds that applying Muon orthogonalization to fused QKV, AdaLN, and FFN weights as a whole causes interference between functionally distinct subspaces. The study improves late-stage convergence through blockwise momentum orthogonalization, although the claimed 2× speedup has so far been validated only on a single 675M DiT configuration.