模型訓練與微調
ShadowPEFT Lands in Hugging Face PEFT Main, Replacing Distributed LoRA Weight Deltas with Cross-Layer State
ShadowPEFT models task adaptation as a small trainable model running alongside a frozen backbone and is now integrated with PEFT’s standard configuration, saving, and loading interfaces. Early experiments outperform LoRA and DoRA at comparable parameter counts, but additional memory usage, dual KV caches, and the lack of a stable release remain deployment trade-offs.
Hugging Face has added ShadowPEFT to the PEFT main branch, allowing developers to use a parameter-efficient fine-tuning method distinct from LoRA through the existing `get_peft_model`, `save_pretrained`, and `from_pretrained` interfaces. LoRA adds independent low-rank weight deltas alongside selected linear layers; ShadowPEFT instead creates a small, pretrainable shadow network that maintains task state propagated across the depth of the Transformer.
At each decoder layer, a correction is first injected into the frozen backbone through a low-rank bottleneck based on the difference between the backbone hidden state and the shadow state. After the backbone completes the layer’s computation, the shadow state is updated using a candidate value and a gate. This bidirectional information flow allows later layers to use task signals accumulated earlier rather than relying solely on local weight deltas. Because the updates form an input-dependent computational trajectory, ShadowPEFT cannot be merged back into the original weights like LoRA; autoregressive decoding must also maintain KV caches for both the backbone and the shadow network.
The sidecar has its own prediction head and, after training, can be detached with `unload_shadow()` to run independently. It can also be initialized from a smaller existing model. This enables a design in which simple requests are handled on-device first, while complex requests are processed together with the larger backbone, but it does not mean the same small model can be reused unconditionally across different backbones.
The authors trained Llama 3.2 3B on MetaMathQA and evaluated it on GSM8K, reporting that ShadowPEFT achieved 48.1% exact match with 8.66 million trainable parameters, compared with 46.9% for LoRA and 46.2% for DoRA. The trade-off was peak memory usage of 28.2GB, versus 22.3GB for LoRA; the results also cover only a limited set of models and datasets in a single-A100 environment. Engineering teams should also note that the feature is currently available only on PEFT `main`, must be installed from source, and has not yet entered a stable release. Key areas to watch next include long-context decoding costs, compatibility across different backbones, and whether the detached shadow can maintain quality in real-world edge-cloud workloads.