本機推論／系統
JustFit Compresses and Dynamically Schedules KV State, Enabling a 24 GiB MacBook to Complete a 210K-Token Workload
By combining 4-bit KV, component residency management, and cross-request state transitions in MLX, JustFit enables a 27B model to perform long-context inference within 24 GiB of unified memory. The reported 6.93× capacity increase comes from a single-machine, model-specific test, and some historical environment details and checkpoint identities cannot yet be fully reproduced.
JustFit expands the bottleneck for running large language models locally from whether the weights fit in memory to the entire runtime live set: in addition to quantized weights, the KV cache, reconstruction workspace, speculative decoder, output head, and concurrent request states all compete for Apple Silicon’s unified memory. Building on MLX and Qwen3.8-27B MXFP4, the authors propose three components that work together.
KVExec stores KV using a TurboQuant-derived 4-bit representation with 256-token pages and reads the compressed state directly during decoding. During prefill, it unpacks each layer in place, applies the inverse Hadamard rotation, and immediately passes the result to standard SDPA, avoiding the need to retain floating-point KV for all layers simultaneously. The paper estimates that the model’s KV payload per position falls from 65,536 bytes with FP16 to 16,640 bytes, but stresses that compression alone is insufficient because reconstructing a single layer at a 192K context length can still require about 768 MiB.
PhaseSwap unloads or restores components according to the execution phase and an “owner lease.” For example, it temporarily releases the roughly 644 MiB output head during the middle of prefill. StateTrans preserves the target model cache, reclaims page references, and coordinates component lifecycles when requests join or finish, or when the system transitions from single-request speculative decoding to continuous batching. This allows state transitions without rebuilding the entire context.
On an M4 Pro MacBook with a 21,000 MiB process limit, all three trials completed a 196,608-token input plus a 16,384-token output, totaling 212,992 positions. The mlx-vlm baseline used in the paper could complete only 30,720 positions. A separate test with a 32K-token input reached 19.11 tokens/s, while a single-seed AIME 2026 evaluation answered 29 of 30 questions correctly.
These figures should not be conflated into a single performance conclusion: capacity, throughput, and mathematical evaluation were measured using different workloads, and the data span several code revisions. The authors have released an mlx-vlm branch and some pinned snapshots, but acknowledge that the historical GPU core count, macOS, Python, and MLX versions, as well as the exact bit-level identity of the original checkpoint, are not fully documented. The next priorities are independent reproduction using fixed images, validation across other models and hardware, and determining whether long-context quality remains stable under KV compression.