最新模型
Hy4 Preview Opens 770B MoE Weights, Targeting Long-Context Agents with Sparse Attention and Built-In MTP
Tencent has released Hy4 Preview’s Apache 2.0-licensed weights, an FP8 version, and deployment recipes for vLLM and SGLang. The model activates roughly 49B of its 770B parameters per token. It supports a 1-million-token context window, though evidence for its quality and self-optimization capabilities still comes primarily from Tencent’s internal evaluations.
Tencent released Hy4 Preview on August 28. It is a 78-layer mixture-of-experts model with 770B total parameters. The first layer uses a dense FFN, while each of the remaining 77 layers contains 256 routed experts and one shared expert. For each token, the model selects eight routed experts and activates the shared expert, resulting in roughly 49B active parameters in the backbone. The model also includes one native MTP layer with approximately 10B total parameters and 0.7B active parameters, enabling speculative decoding by predicting subsequent tokens in a single pass.
The attention module uses Gated DeepSeek Sparse Attention, with IndexCache reusing sparse indices across layers. The model card specifies an index top-k of 2,048 and a native context length of 1 million tokens. Its residual path uses four identity Hyper-Connections. Tencent has released BF16 and FP8 weights, a fine-tuning workflow, and prebuilt images for vLLM and SGLang. The recommended FP8 serving configuration uses eight-way tensor parallelism, a sparse-attention backend, and three to four speculative tokens, while exposing an OpenAI-compatible API. This allows research teams to inspect the complete architecture, but deploying 770B weights still requires data-center-class infrastructure. Activating only 49B parameters per token does not make its memory requirements equivalent to those of a conventional 49B model.
Tencent says Hy4 helped explore training methods, data strategies, evaluation techniques, and low-level kernels, while operator fusion and communication optimizations increased inference throughput by 31.8% over an internal baseline. This is better understood as an R&D feedback loop in which the model assists with experiments, rather than evidence that it can modify its own weights without supervision. In another blind evaluation covering 203 engineering tasks and conducted by 163 internal experts, Hy4 received an average score of 2.99 out of 4, only slightly above GLM-5.3 at 2.92 and Kimi K3 at 2.94. The tasks, scoring data, and reproducible harness have not yet been released. Engineering teams should now watch for third-party long-context and agent evaluations, FP8 quality degradation, real-world KV cache costs, and the excessive reasoning and repeated verification issues already acknowledged by Tencent.