推論系統
FreeToken Dynamically Allocates CPU, GPU, and PCIe Resources to Run a 35B MoE Model with 8GB of VRAM
Instead of using fixed offloading, FreeToken uses measured bandwidth to decide whether to transfer cache-missed experts to the GPU or execute them directly on the CPU. The paper also demonstrates a 753B model served by a single workstation GPU, but the full weights still require system memory; this should not be interpreted as “fitting all 753B parameters on one GPU.”