推論系統
FreeToken Dynamically Allocates CPU, GPU, and PCIe Resources to Run a 35B MoE Model with 8GB of VRAM
Instead of using fixed offloading, FreeToken uses measured bandwidth to decide whether to transfer cache-missed experts to the GPU or execute them directly on the CPU. The paper also demonstrates a 753B model served by a single workstation GPU, but the full weights still require system memory; this should not be interpreted as “fitting all 753B parameters on one GPU.”
FreeToken is an MoE inference engine redesigned for personal computers. Conventional hybrid CPU–GPU solutions determine at load time which experts remain on the GPU. FreeToken instead treats the GPU, CPU, system memory, and PCIe as a single variable resource pool. It first benchmarks the hardware with `ft bench bw`, then decides at each decoding step whether an expert missed by the cache should be transferred to the GPU over PCIe or executed directly on the CPU, overlapping the two paths.
The design handles prefill and decode separately. Prefill for long prompts touches nearly every expert in each layer, so the system uses layer-by-layer, double-buffered streaming to hide transfers of the next weight batch behind the current computation. Decode has a sparser working set and therefore uses a global LRU expert cache and bandwidth-adaptive scheduling. For coding agents that repeatedly append tool results, remove reasoning blocks, or compress context, FreeToken also preserves semantic anchors, recurrent state, and the KV cache to avoid recomputing everything from scratch on each turn. Remaining VRAM can be dynamically reallocated between expert slots and KV pages without restarting or reloading the weights held in system memory.
The paper demonstrates a 35B MoE model on a laptop GPU with 8GB of VRAM, DeepSeek-V4-Flash 284B on a gaming desktop, and GLM-5.2 753B on a single 96GB RTX PRO 6000 Blackwell. These figures refer to total model parameter counts and do not mean that all weights reside on the GPU. Performance depends heavily on system memory capacity, CPU memory bandwidth, PCIe generation, quantization format, and the expert-routing distribution. The project is open source under the Apache 2.0 license, provides OpenAI- and Anthropic-compatible endpoints, and supports more than 20 models. For now, however, it primarily targets NVIDIA RTX 30, 40, and 50 Series GPUs as well as Blackwell; support across AMD hardware, Apple Silicon, and configurations with slower memory remains to be validated.