FreeToken Dynamically Splits MoE Cache Misses, Runs a 35B Model at 39.3 token/s on an 8GB RTX 4060

FreeToken reframes local MoE inference as a joint scheduling problem spanning the CPU, system memory, PCIe, and GPU, rather than treating a personal computer as a scaled-down server. The complete expert weights remain resident in system memory, while non-expert weights stay on the GPU. The remaining VRAM becomes a cross-layer LRU expert cache whose capacity can be adjusted at runtime. The correctness of this configuration does not depend on the cache hit rate: the host always retains the authoritative copy of the weights, and GPU capacity affects only performance.

When decoding encounters `m` experts missing from the cache, the system does not always choose either data transfer or CPU offload. Instead, using measured host-memory bandwidth `B_H` and PCIe bandwidth `B_P`, it determines the number of experts to transfer to the GPU as `q≈m·B_P/B_H`; the rest are computed in place on the CPU. The two branches run concurrently and their partial outputs are then combined, without dropping experts or approximating routing. The paper also notes that short-term routing exhibits locality. With the same cache capacity on an RTX 5090, FreeToken records expert-read miss rates of 16% for Qwen3.6 and 39% for DeepSeek-V4-Flash, lower than KTransformers’ 41% and 59%, respectively.

Another source of overhead in agentic workloads is repeated prefill after tool calls. FreeToken saves checkpoints of the KV cache and loop state at special-token boundaries. When a conversation is truncated or rewritten, it recomputes only the suffix following the most recent valid anchor. For model-layer loading, double buffering hides the next layer’s PCIe transfer behind computation for the current layer. The [paper](https://arxiv.org/abs/2608.16157) reports 77–83 token/s for Qwen3.6-35B-A3B on an RTX 5090 and 39.3 token/s on a laptop with an 8GB RTX 4060 using an [NVFP4 checkpoint](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4). The model has 35B total parameters and activates approximately 3B per token.

These results are still based on author-controlled, single-request tests. Workloads, CPU thread counts, and quantization formats were not fully identical across all engines, and the evaluation does not cover multi-tenant throughput or power consumption. More importantly, although the paper links to a GitHub repository, it still returned a 404 when checked, preventing independent reproduction of the kernels, state cache, and dynamic VRAM reconstruction. Engineering teams should assess memory requirements and the model support matrix once the code becomes public, as well as whether long-running agent workflows can sustain the same tail latency.