Rollplex Fits VLM Reinforcement Learning Prefix Computation into Rollout Idle Time, Accelerating Training by up to 1.30× on 32 H800 GPUs

Vision-language model reinforcement learning workloads differ from text-only workloads: across the four video tasks evaluated in the paper, prompts accounted for a median 79%–98% of each sample’s tokens. Visual encoding and prefill are therefore no longer minor upfront costs. Conventional synchronous pipelines still execute rollout, reference-model scoring, actor forward and backward passes, and weight updates sequentially, preventing underutilized compute units during decoding from taking on subsequent work.

The [Rollplex paper](https://arxiv.org/abs/2608.14498) divides each model stage into a prefix that does not depend on generated results and a suffix that must wait for response tokens. Prefix computation for the reference model and training actor can run concurrently with rollout decoding. After generation finishes, suffix computation resumes from the retained boundary KV state. All computation still uses the same actor snapshot; Rollplex employs neither stale policies nor speculative responses, so it does not relax the ordering of synchronous on-policy updates.

Straightforward colocation is infeasible. Qwen2.5-VL-32B’s weights, gradients, Adam states, training activations, three sets of KV caches, and workspaces require approximately 165 GiB per GPU in total, exceeding the H800’s 80 GB capacity. Rollplex uses CUDA Virtual Memory Management to reserve virtual addresses, mapping physical HBM pages only for the period between a tensor’s production and its final consumption. Activations can be offloaded or recomputed, while FP32 optimizer states are processed through chunked streaming. Another mechanism classifies weights by tensor layout: identically sharded weights are directly aliased, layouts related by transposition share storage through logical views, and only incompatible arrangements such as fused QKV require copied and reordered data. This allows training to use TP=8 and rollout to use TP=4 without retaining a complete second copy of the actor.

On 32 H800 GPUs with Qwen2.5-VL-32B and four video reasoning tasks, the authors report speedups of 1.23–1.30× over sequential colocation and 1.57–2.24× over disaggregated deployment. [ROLL](https://github.com/alibaba/ROLL) is already open source and integrates Megatron-Core, vLLM, and Ray, but its current public main branch does not yet clearly identify a complete implementation that can directly reproduce Rollplex. Engineering teams should watch for the code release, evaluate the sensitivity of host offload to PCIe and NUMA topology, and determine whether the gains persist as model size, video length, or faster decoding kernels alter the available overlap window.