模型壓縮與 CPU 推論
CompactifAI Compresses Llama 3.3 70B to About 65 GiB, Runs It on a Single Xeon 6 with vLLM CPU
Multiverse Computing has unveiled a compressed and healing-trained version of Llama 3.3 70B that can be served on Xeon 6 using vLLM CPU and Intel AMX. Official tests show nearly double the throughput, but output remains below 4 tokens per second, and neither the checkpoint nor the raw results have been released.
Multiverse Computing connected its CompactifAI-compressed version of Llama 3.3 70B to vLLM’s CPU backend, aiming to deploy a large model on an Intel Xeon 6 server without accelerators. The company says the model files were reduced from about 130 GiB to 65 GiB. Its method goes beyond quantization, using tensor networks to reduce the model representation before performing “healing” training. The serving layer uses vLLM CPU, while matrix operations leverage Xeon’s Advanced Matrix Extensions (AMX). vLLM’s official documentation also confirms Intel Xeon as one of its supported CPU paths, although the CompactifAI checkpoint is not provided by the upstream vLLM project.
The test system used only one 32-core, 64-thread Xeon 6737P and about 1 TB of memory, with tensor parallelism set to 1. With fixed sequences of 1,024 random input tokens and 1,024 random output tokens, output throughput for a single concurrent request increased from 2.00 to 3.86 tokens/s, while total token throughput rose from 4.02 to 7.81 tokens/s. Mean time to first token (TTFT) fell from 6.94 seconds to 3.71 seconds, and time per output token (TPOT) dropped from 494.71 milliseconds to 255.76 milliseconds. Testing extended to 256 concurrent users, with the company claiming a maximum throughput improvement of 107%.
On quality, BoolQ, GSM8K, HellaSwag, and MMLU scores declined by about 0.95% to 2.48%, while WinoGrande improved by 6.86%. The latter result may stem from the healing training, but it could also reflect evaluation variance or template sensitivity. This release demonstrates that a 70B-class model can fit on a single-socket CPU server, but it does not mean the interactive experience is approaching GPU performance: at under 4 tokens/s with a TTFT of several seconds, it is better suited to low-speed batch processing, data-sovereignty requirements, or failover scenarios. Although the company disclosed its method and workload configuration, it has not released the compressed weights, command-line arguments, container versions, or raw JSON results. Engineering teams should next request downloadable artifacts and rerun the tests using realistic prompt lengths while measuring KV cache pressure, tail latency, energy consumption, and cost per request.