開放模型與推論
Hy4 Preview Advances Million-Token Open Models With a 770B MoE, Sparse Attention, and Native MTP
Tencent has released Hy4 Preview and FP8 weights. The model activates 49B of its 770B parameters per token and supports a native context window of 1 million tokens. It uses sparse attention with cross-layer index reuse and provides vLLM and SGLang deployment images, though its enormous weights and vendor-reported evaluations still make real-world adoption difficult to assess.
On August 28, Tencent unveiled [Hy4 Preview](https://www.tencent.com/tencent-releases-and-open-sources-tencent-hy4-preview/), releasing BF16 and FP8 weights, a fine-tuning workflow, and an Apache 2.0 license. The model backbone has 770B parameters and activates 49B per token. Of its 78 layers, only the first uses a dense FFN; each of the remaining 77 layers contains 256 routed experts and one shared expert, with eight routed experts selected at a time. This keeps compute requirements far below those of a dense model of comparable size, but the complete BF16 files still total about 1.56 TB. The “49B active parameters” figure should therefore not be interpreted as meaning the model can run on hardware typically used for a conventional 49B model.
Another architectural highlight is [Gated DeepSeek Sparse Attention](https://github.com/Tencent-Hunyuan/Hy4-preview): its indexer selects no more than 2,048 positions, while IndexCache reuses sparse indices across layers to control attention costs over a 1-million-token context window. Four-way identity Hyper-Connections expand the flow of residual information. The model also includes one native Multi-Token Prediction layer with 10B total parameters, of which 0.7B are active. Tencent’s official vLLM example can speculate three tokens at a time and specifies the FLASHMLA_SPARSE backend. SGLang also provides prebuilt images for x86 and Arm, with examples for both frameworks launching the FP8 model using eight-way tensor parallelism.
Tencent reports that Hy4 scores 65.7% on SWE-bench Pro public and 85.4% on Terminal-Bench 2.1. In an internal blind evaluation, 163 experts rated it slightly ahead of GLM-5.3 and Kimi K3 across 203 engineering tasks. These figures, however, were produced primarily by the developer, and the task harnesses, inference budgets, and hardware costs were not fully aligned. The model card also acknowledges that Hy4 can overthink and repeatedly verify its work. Engineering teams should next watch for independent long-context stress tests, real-world throughput of the sparse kernels across different GPUs, and whether FP8 quantization preserves agentic and tool-calling capabilities.