端側推論與開源執行時
Edge0 Streams MoE Experts from SSD, Requiring Just 2.9 GiB of Active Memory for a 35B Model at Short Context Lengths
The new open-source framework uses predictive routing to preload experts for the next step and Recover-LoRA to compensate for int4 quantization loss, so the entire 23GB checkpoint does not need to remain resident in memory. Official M4 Pro tests measured 14.9–17.7 tokens per second, but the framework currently supports only Apple Silicon, and its memory and quality results have yet to be independently reproduced.
Edge0 reframes the deployment of large sparse MoE models as a storage-tier management problem: all expert weights remain on SSD in int4 format, while memory mapping loads only the experts required by the current token at runtime. Active memory therefore depends on the working set rather than the model’s total parameter count. The first Apache 2.0 preview release includes a Qwen3.5-MoE-based 35B-A3B model and a Ling 3.0-based 8B-A1B model. The models, LoRA adapters, and routing predictors are distributed in the same directory and can be served through the OpenAI-compatible `/v1/chat/completions` endpoint.
Loading weights from SSD only after routing decisions are made would typically leave each layer waiting on I/O. Edge0 therefore trains an additional prerouter head that predicts the experts likely to be selected one step ahead, overlapping reads with the current forward pass. The project claims that this design can increase decoding throughput by up to 59%, although the benefit varies with SSD latency, model size, routing width, and prediction hit rate; incorrect prefetches can also waste bandwidth. The backend interface has been separated from the MLX implementation, but CUDA support remains only a placeholder.
Another component, Recover-LoRA, distills from an FP16 teacher into a frozen int4 model in an attempt to recover quantization losses without merging the adapter into the base weights. In the project’s comparison across five benchmarks, the 35B version averaged 3.9 points below its FP16 base model, while the 8B version was 2.8 points lower. These are still OpenCompass results produced by the team itself, and the two models use different base models.
In tests on a 24GB Mac mini with an M4 Pro, using a prompt of approximately 3,300 tokens and 200 timed decoding tokens, the 35B version achieved 14.9–17.7 tokens per second and reached a peak active-memory footprint of 2.9 GiB at short context lengths. The complete quantized file is still about 23GB and remains stored on disk. The KV cache grows separately at longer context lengths, so “running a 35B model in 3GB” should not be interpreted as the total storage or system-memory requirement. Key engineering questions now include real-world performance on phones, cold-start behavior and latency variance under concurrent requests, SSD write endurance, and whether a CUDA backend can preserve the same working-set and throughput advantages.