GitHub Repo
Transformers Proposes Integrating HSDP and Expert Parallelism, Adjusting MoE Gradient Synchronization
A development draft appeared in Transformers on November 3, enabling hybrid sharded data parallelism and expert parallelism to work together while changing cross-replica synchronization and scaling for expert gradients. The proposal still depends on another development branch and has not been merged or reviewed.
Hugging Face Transformers contributors opened PR #50293 on November 3, proposing support for hybrid sharded data parallelism (HSDP) alongside expert parallelism. The draft removes the check that rejects configurations where dp_replicate_size > 1 and ep_size > 1, and adjusts sharding and gradient handling for expert parameters. As of the time of review, the proposal remained a Draft, targeted the hsdp development branch, and depended on another proposal; it should not be treated as support in an official release. Source: development proposal
The work addresses how multiple parallelism strategies share device groups. The official PyTorch tutorial describes HSDP as a two-dimensional configuration: FSDP sharding within a host and data-parallel replication across hosts. DeviceMesh manages communication groups for each dimension and can also create submeshes for different parallelism strategies. Training code that schedules both ordinary model layers and expert layers needs its device layout to match the scope of gradient synchronization. Source: PyTorch tutorial
According to the commit description, when the number of data-parallel replicas is greater than one, the experts responsible for token dispatch use a two-dimensional (dp_replicate, efsdp) mesh for sharding; ordinary dense modules use (dp_replicate, fsdp). Expert gradients also undergo an all-reduce across replicas and are divided by dp_size. The author says that adding replicas also increases the number of batches participating in computation, so corresponding scaling is needed, and describes this as consistent with torchtitan's approach. Source: commit description
By inference from the design, this could let MoE training combine expert partitioning with data-replica scaling; actual throughput, memory requirements, and numerical consistency still need to be measured. The public proposal does not include performance benchmarks, and its page shows no reviews. Engineering teams should track whether the base branch is merged and validate results across different replica counts, expert-parallelism levels, and gradient-accumulation settings, especially whether the scaling preserves the intended effective-batch semantics. Source: proposal status