邊緣 AI／推論系統
mzCache Restores Mobile LLM Memory in Parallel Across CPU and GPU, Cutting Time to First Token by 2.1× to 5.5×
The MobiCom 2026 study divides model weights and the KV cache into shared buffers, allowing the GPU to begin prefill while the CPU restores evicted data. The work targets recovery latency after multitasking on smartphones, rather than improving the model’s generation speed itself.
Researchers at Seoul National University and the University of California, Berkeley have proposed mzCache to address a problem affecting on-device LLMs: after a user switches to a camera, game, or video app, Android’s memory-reclamation mechanism may evict the model’s weights and KV cache. According to the paper, models with fewer than 3B parameters can still occupy 5–7 GB when paired with long contexts. For Qwen3-0.6B, for example, the KV cache for a 32K context is approximately 3.5 GB. Conventional zRAM uses LZ4 or Zstd, but reduced the size of the tested KV caches by only about 0.2%–9.3%. If memory pressure persists, the Low-Memory Killer may terminate the model process outright. The [mzCache paper](https://arxiv.org/abs/2609.01338) divides weights by layer and the KV cache into fixed-size token blocks, storing them in OpenCL Shared Virtual Memory buffers. This avoids the all-or-nothing retention or release required by traditional large contiguous memory regions.
The system’s core mechanism is not simply swapping; it determines the eviction order based on how to achieve the fastest recovery. mzCache prioritizes evicting later Transformer layers while retaining the earlier layers used first during inference. When a new request arrives, the GPU immediately begins prefill using the layers still in memory, while the CPU restores weights from flash storage and decompresses KV blocks from the compressed region in forward-pass order. A custom OpenCL attention kernel can directly stream noncontiguous KV pointers without first merging them into a complete buffer. The implementation consists of roughly 6,000 lines of C/C++ built on llama.cpp and uses ARM NEON for KV compression.
The researchers evaluated storage-backed partial offloading on commercial smartphones with Qwen3-0.6B and EXAONE-4.0-1.2B, reporting a 2.1×–5.5× reduction in TTFT. In multitasking tests on a Galaxy S25+, the model was terminated during every test cycle under the standard OS path, whereas mzCache preserved the session state. The [research team’s publications page](https://sites.google.com/view/kyunghanlee/publications) lists the paper as accepted by MobiCom 2026. However, no public source code has been found so far, and the evaluation is limited to Qualcomm Adreno, two small models, and a small number of smartphones. Portability to NPUs, Metal, or larger models—as well as additional peak power consumption and flash-write costs—will still need to be validated before deployment.