MoAI Showcases GLM-5.1 on 32 MI300X GPUs, Integrating Heterogeneous GPU Routing and Disaggregated Inference into a Single Service

Korean AI infrastructure company Moreh demonstrated the MoAI Inference Framework at AMD Advancing AI 2026, serving GLM-5.1 chat inference across four nodes equipped with a total of 32 AMD Instinct MI300X GPUs. In addition to generating responses, the demonstration interface displayed GPU utilization, tokens per second, time to first token (TTFT), and time per output token (TPOT) in real time. This allowed observers to distinguish the performance of prompt processing from token-by-token generation instead of relying on a single average speed metric.

MoAI is positioned as a data-center-grade inference control layer. It can assign compute-intensive, highly parallel prefill workloads and memory-bandwidth-bound decode workloads to separate GPU pools, then route requests based on prefix-cache hits, request length, device performance, and service-level objectives. The underlying engines listed in the official documentation include Moreh’s modified version of vLLM, upstream vLLM, and SGLang, while the supported hardware spans NVIDIA, AMD, and Tenstorrent accelerators. The accompanying Performance Gateway handles request distribution, MoAI Fabric manages cross-device KV cache transfers, and the planned Autopilot is intended to automatically select parallelization strategies and resource allocations based on traffic and latency constraints.

This design is notable because enterprise data centers often retain multiple generations of accelerators, sometimes from several vendors. If existing GPUs can be used for prefill or workloads of particular lengths, organizations may be able to reduce the cost of replacing all hardware for a single model. Moreh’s website claims that cross-vendor prefill-decode disaggregation can deliver 1.7× higher throughput and that prefix-cache-aware routing can reduce TTFT by as much as 20×. However, these figures come from different scenarios and should not be treated as results from this GLM-5.1 demonstration. The press release did not disclose model precision, quantization format, batch distribution, network topology, or measured tokens per second (TPS). For now, the demonstration shows that the system is operational, not that it has established a performance lead under fair, comparable conditions. Future areas to watch include publication of deployment configurations, failover behavior, and whether moving KV caches between heterogeneous nodes becomes a new bottleneck under long-context traffic.