模型與本機推論
POCKET Compresses a 35B Sparse MoE to 8.2 GB, Runs on Phones and CPU-Only Systems With Standard llama.cpp
VIDRAFT has released the weights for its POCKET series, quantizing a 35B model down to as little as 8.2 GB while retaining roughly 3B active parameters per token. The models run directly with upstream llama.cpp, although the compression method remains undisclosed and current performance and quality evidence comes primarily from the development team.
Korean team VIDRAFT has released POCKET, a family of on-device models based on Darwin-36B-Opus. Rather than having a phone scan all 35B weights for every token, POCKET uses a sparse Mixture-of-Experts (MoE) architecture with 256 experts, activating only eight at a time to keep the active parameter count at roughly 3B. The team also provides GGUF quantizations ranging from IQ1_M to Q4_K_M, with file sizes from 8.2 GB to 21 GB. Separate versions are available for Korean, English, and Apple MLX.
The most notable technical aspect is compatibility: POCKET reportedly requires no runtime modifications and can run directly with standard llama.cpp. Existing LM Studio, Ollama, and custom CPU inference workflows therefore do not need to maintain a dedicated fork. On the same 8-vCPU Hugging Face Space, the team measured about 20.5 tokens/s, compared with roughly 6.1 tokens/s for the 27B dense Bonsai model. It also reported higher decoding throughput on a 16-thread Xeon system, an M3 Pro, and an H100. This is consistent with the memory-bandwidth-bound nature of decoding: a sparse model moves fewer active weights at each step, so it can outperform a smaller dense model despite having more total parameters.
However, “running a 35B model on an iPhone” does not necessarily translate into a practical mobile assistant. The team has yet to publish its own iPhone measurements for tokens per second, time to first token, peak memory usage, sustained-load temperature, or power consumption. Its public quality comparison is also limited to 400 HellaSwag questions, where a score of 61% versus Bonsai’s 60% is insufficient to establish reasoning, Chinese-language, or coding capabilities. The weights are licensed under Apache-2.0, but the compression recipe used to produce the extremely low-bit versions remains proprietary. Engineering teams should next retest prefill performance, long-context memory usage, and thermal throttling on real devices, while checking whether the lowest-bit variants exhibit domain-specific degradation.