開放模型／代理基礎設施
Solar Open 2 Extends a 250B MoE’s Context to 1 Million Tokens With Hybrid Linear Attention
Upstage has released the weights for Solar Open 2, which uses 250B total parameters and activates 15B parameters per token for long-horizon agentic workflows. The model retains softmax attention and a KV cache in only one-quarter of its layers, but its million-token capability and Korean office-agent results are still based largely on the team’s own evaluations.
Upstage has released Solar Open 2, an open-weight model designed for tool use, coding, and document-centric office workflows. It uses a sparse mixture-of-experts (MoE) architecture with approximately 250B total parameters, routing each token to about 15B parameters. In addition to 320 routed experts, it includes one shared expert. The weights support English, Korean, and Japanese, while the model card lists a 48-layer architecture, a hidden size of 4,096, and a 196,608-token byte-level BPE vocabulary.
The long-context design is the real highlight. The model alternates one softmax-attention layer with three linear-attention layers, so only 12 layers need to retain a conventional KV cache. The linear layers use a gated delta rule extended to negative feature values and remove positional encodings such as RoPE entirely. Upstage estimates that this reduces the memory and compute required for inference at 1 million tokens to roughly one-quarter of that required by an all-softmax model of the same shape. For agents that accumulate large volumes of tool outputs, code, and intermediate artifacts, this has greater deployment significance than simply increasing the advertised context window.
For training, the team selectively transferred 5.69B still-compatible parameters from Solar Open 1, representing only about 2.3% of the new model, and pretrained the remainder from scratch. It then curated a 10T-token data mixture from a 20T-token corpus pool. Post-training first produced 12 domain experts, which were then merged into a single checkpoint through multi-teacher on-policy distillation.
The official report claims that Solar Open 2 leads similarly sized open models on MMLU-Pro, LiveCodeBench, and APEX-Agents. Its score on the Korean Ko-GDPval benchmark reportedly rose from 3.4 for the previous generation to 86.8. However, Ko-GDPval is an internal office-agent benchmark, and the compared models may not use the same agent framework. Engineering teams should next examine native inference-framework support for hybrid attention and NoPE, real-world memory scaling at 1 million tokens, and whether third parties can reproduce its success rates on long-horizon workflows.