推論系統
FreeToken Dynamically Splits Expert Computation Across CPU and GPU, Enabling 284B MoE Inference on a Single RTX 5090
Instead of offloading MoE experts according to fixed rules, FreeToken dynamically divides work between CPU execution and GPU caching based on measured memory and PCIe bandwidth. The authors report that an RTX 5090 can run a 284B model at 22–25 tokens per second, though testing was conducted by the development team and the full weights still require substantial system memory.
FreeToken is an Apache 2.0-licensed inference engine released by researchers from UC Berkeley, MIT, and other institutions. Rather than treating a personal computer as a scaled-down data center, it treats the GPU, CPU, system memory, and PCIe as a unified heterogeneous compute pool. A sparse Mixture-of-Experts (MoE) model activates only a small number of experts for each token but must still retain the complete expert weights. With conventional static offloading, a GPU cache miss triggers either a synchronous weight transfer or sends the entire workload to a CPU constrained by DRAM bandwidth.
FreeToken’s `q*` strategy first measures actual system-memory and PCIe bandwidth, then determines layer by layer how many cache-missed experts should be transferred to the GPU and how many should be computed directly on the CPU. The two paths execute in parallel, and their partial outputs are merged afterward, without trading accuracy for speed through approximate computation. The engine also uses cross-layer double buffering to hide transfers of the next layer’s weights behind computation for the current layer, while a global LRU cache tracks experts whose popularity changes continuously during generation.
For coding agents, the system saves the KV cache or recurrent state at semantic boundaries such as reasoning segments, tool calls, and conversation turns. When an agent deletes old tool output or edits content in the middle of the context, it can recompute the suffix from a still-valid anchor instead of prefilling the entire context again. VRAM can also be reallocated between the expert cache and the growing KV cache without reloading the model.
The paper reports 77–83 tokens per second for Qwen3.6-35B-A3B and 22–25 tokens per second for the 284B DeepSeek-V4-Flash on an RTX 5090. It also reports that an 8GB RTX 4060 laptop can run the 35B model. This does not mean that large models require only enough capacity to fit in GPU memory: the complete expert pool remains in system memory, and a 284B FP4 deployment, for example, requires storing roughly 140GB of weights. Engineering teams should wait for independent reproductions and evaluate RAM capacity, NUMA topology, PCIe generation, quantization quality, time to first token, and multi-request concurrency. The current results cannot be directly generalized to every model or typical consumer hardware.