模型訓練／LLMOps
Together Fine-Tuning Service Lets LoRA Train MoE Experts Directly and Exposes the Tokens Actually Sent to the Trainer
Together AI has added Expert LoRA, per-example weighting, pre-training data validation, and real-time metrics, allowing developers to tune MoE expert layers instead of only attention modules. In the company’s experiments, knowledge recall rose from 15% to as high as 89%, but the comparison was limited in scope, and expert and attention adapters currently cannot be used together.
Together AI expanded its managed fine-tuning service on September 11, with Expert LoRA standing out as the most technically significant addition. Conventional LoRA typically attaches low-rank adapters to attention modules such as `q_proj` and `k_proj`; in MoE models, however, a large share of the parameters and domain knowledge resides in the expert feed-forward layers. The new interface lets users specify `w_up`, `w_gate`, and `w_down` through `lora_trainable_modules`, while controlling adapter size with a configuration that shares low-rank factors across experts.
Together injected 200 fictional facts into Gemma 4 26B‑A4B and Qwen3.6 35B‑A3B. It reported recall of up to 89% after training the expert layers, compared with a maximum of 15% for the attention-only version; on MMLU‑Pro, the scores were 75.3% and 71.5%, respectively. The team argues that tuning only attention can cause some routed experts to gradually stop participating. However, this was a narrow knowledge-injection experiment designed by the vendor, so it is not sufficient to conclude that fine-tuning in every domain will deliver gains of the same magnitude.
The update also addresses common but easily overlooked data issues in fine-tuning pipelines. Before starting a run, users can inspect the tokens, loss masks, and truncation status produced after applying the actual tokenizer and chat template. Afterward, they can download a subset of the data exactly as received by the trainer after tokenization, packing, and collation. The server validates message roles, tool-call pairings, reasoning fields, preference data, and image decoding in advance. It also supports per-example loss weighting, a packing toggle, gradient accumulation, early stopping, and step-level loss, gradient norm, and learning-rate metrics through the API and CLI.
The deployment constraints still require careful review: Expert LoRA is available only for the listed Mixtral, DeepSeek, Kimi, and Nemotron models, and a single job cannot target both expert and attention modules. Expert adapters are also not merged into full model weights and must be served through dedicated endpoints. The article notes that deploying intermediate adapters directly during training remains a preview of a future capability and cannot yet be incorporated into existing production workflows.