GitHub Repo
Transformers Community Reports Eight-GPU B200 Storage Test, Revealing Bottleneck in Centralized Model-Weight Writes
A Transformers community test published October 30 found that distributed checkpoints can reduce storage time for large model weights. The results also highlight host-memory and format-merging costs, but do not yet cover multiple nodes or full training state.
On October 30, the Transformers community submitted an eight-GPU NVIDIA B200 benchmark comparing the current model-weight loading, centralized storage, and distributed checkpoint (DCP) paths. The new contribution is public measurements and reproduction scripts; the test used a specified main commit and did not modify the library implementation. Test report
In an eight-way fully sharded data parallel (FSDP) configuration, centralized storage took 158 seconds for Qwen2.5-72B, compared with 52 seconds for DCP. For Qwen3-235B-A22B, the respective times were 772 seconds and 219 seconds. The authors note that the centralized path gathers the full weights in the main process, converts the format, and writes the file while other processes wait. In the 235B case, peak memory use in the main process reached 775 GB. Measurements and bottleneck analysis
DCP is designed so that each compute process saves its own shard in parallel, and shards can be repartitioned at load time to match a different cluster topology. PyTorch documentation explains that these checkpoints typically consist of multiple files and are loaded into storage preallocated by the model. This helps explain why DCP suits distributed recovery workflows, and why engineers should account for file layout and loading methods in deployment design. PyTorch DCP documentation
Faster shard writes do not mean the model delivery workflow is complete. The report also found that consolidating the 72B DCP weights into a single safetensors file still took 202 seconds. For teams that need to export models, this means measuring the costs of periodic saves, failure recovery, and final export separately; the time for just one stage cannot estimate the overall benefit. Consolidation test
These figures are still from a single-machine community test. Most cases were run only once, and the benchmark covers weights only, not optimizer state or multi-node communication. Independent reproduction, tests with full training checkpoints, and performance across different storage devices are useful next steps. Recovery across versions also needs validation, since PyTorch explicitly states that DCP state_dicts are not guaranteed to be backward compatible across versions. Test scope, compatibility note