推論系統
Colibrì Tiers Trillion-Scale MoE Weights Across VRAM, RAM, and NVMe, but Low Memory Usage Does Not Mean Fast Inference
Colibrì, which has recently gained traction in the community, uses a pure C engine to load sparse experts on demand, enabling models with 744B to 2.8T parameters to run on non-data-center hardware. What it lowers is the resident-memory requirement; actual decoding speed remains heavily constrained by SSD bandwidth, expert hit rates, and quantization quality.
Colibrì has recently regained attention in the technical community as it added support for more frontier MoE models. It currently lists nine model families, including GLM‑5.2/5.3, Inkling, Kimi K3, DeepSeek V4, and Qwen3.8 Flash Next. Its core approach is not to squeeze all weights into memory, but to treat VRAM, RAM, and NVMe as a unified tiered storage system. For the 744B-parameter GLM‑5.2, for example, the roughly 17B-parameter dense portion remains resident in about 9.9 GB of RAM using int4, while nearly 20,000 routed experts stay in a roughly 370 GB disk container and are read only when selected by the gate.
To reduce the time each token spends waiting on disk, the runtime tracks expert hotness, maintains per-layer LRU caches and a pinned hot set, and lets the router prefetch one layer ahead. Experts selected repeatedly within the same batch are read only once. Dual-SSD mode distributes read-only weights using deterministic hashing and falls back to the primary drive if a read fails. The project also implements CUDA, HIP, Metal, Vulkan, NUMA, and CPU kernels, while integrating MTP speculative decoding, a compressed MLA KV cache, and persistent conversation state into the same engine.
The engineering value of this design lies in separating “can it be loaded?” from “is it fast enough?” Official records show that cold-start decoding on a 25 GB development machine reaches only about 0.05 to 0.1 token/s; a 128 GB CPU desktop reaches around 1.8 token/s after warm-up; and only a fully resident configuration using six RTX 5090 GPUs reaches roughly 5.8 to 6.8 token/s. These figures come from different machines and workloads and should not be treated as a side-by-side leaderboard.
Deployers should also note that the recommended GLM‑5.2 container is itself about 372 GB. In project testing, the older row-wise int4 version lost roughly nine percentage points in quality; if the MTP head is quantized to int4, its acceptance rate may also fall close to zero. The next things to watch are whether the community can reproduce performance using a fixed model, prompt, cache state, and hardware configuration, and whether hotness learning overfits to a single workload.