推論系統
Chord Rewrites the INT4 MoE Kernel for Kimi K2.x, Selecting GPU Scheduling by Inference Phase
Novita AI and vLLM have open-sourced Chord, a W4A16 CUDA operator tuned for Kimi K2.x serving workloads on H200, B200, and B300 GPUs. Public benchmarks show performance of up to 2.15× that of Humming, but the figures cover only the MoE layer kernel and cannot be directly translated into end-to-end throughput.
Novita AI and the vLLM team have open-sourced Chord, a W4A16 MoE CUDA operator for the Kimi K2.x family. Inputs remain in BF16, while weights are quantized to INT4 and dequantized using a BF16 scale for each group of 32 weights. Rather than serving as a general-purpose GEMM replacement, it is designed around the actual routing shapes of gate, up, and down projections in sparse models, with separate scheduling optimizations for prefill, decode, tensor parallelism, and expert parallelism.
Chord provides two kernel families. The `indexed` path retains Humming’s routing interface and supports EP8 prefill, decode, and mixed TP8 workloads on H200, as well as EP8 decode on B200 and B300. The `grouped_contiguous` and `grouped_masked` paths are derived from DeepGEMM’s data layout and handle prefill and decode, respectively. Based on the actual number of tokens received by each expert—not merely the total number of tokens in the batch—the scheduler selects the block size, stream-K configuration, CTA residency, and pipeline depth. The kernels also use WGMMA, TMA, warp specialization, and cached cubin descriptors to avoid repeating configuration searches for each small decode workload.
The authors compared Chord with the public version of Humming using the same GPUs, shapes, and routing distributions. On H200, EP8 prefill was approximately 1.11× to 1.20× faster, while decode was approximately 1.16× to 1.24× faster. On B300, reported decode performance was 1.81× to 2.15× faster. Engineering teams should note that the B300 baseline used Humming’s default strategy, which has not yet been tuned for the GPU, and that the measurements excluded routing, communication, and activation operations. They therefore do not indicate an equivalent speedup for the complete vLLM service. Although the package can be connected to some vLLM versions through a Humming-compatible entry point, formal integration of the grouped paths is still in progress. Before deployment, teams must also verify WNA16 group-scale support, the quantization format, and their own token-per-expert distribution.