推論系統
Laguna S 2.1 INT4 With DFlash: Four RTX 3090s Tested at 200K Context and a Peak 282.5 tokens/s
A publicly available, reproducible community vLLM recipe deploys Laguna S 2.1 INT4 on four 24GB RTX 3090 GPUs and handles a real 190K-token request using a precision-matched DFlash draft model. The reported 282.5 tokens/s is a peak under a specific configuration, not sustained throughput, while the quality and compatibility of low-bit quantization remain highly dependent on the checkpoint and inference engine.
A newly published community deployment recipe combines Poolside’s Laguna S 2.1 INT4 with DFlash speculative decoding in vLLM 0.25.1, delivering a 200K context window on four RTX 3090 GPUs with 24GB of VRAM each. The author did more than verify that the model could start: the system continued generating after receiving a 190,002-token prompt. Peak decoding speed from the same endpoint reached 282.5 tokens/s, while the model and draft checkpoints require about 85GB of storage in total.
Laguna S 2.1 is a Mixture-of-Experts (MoE) coding model with roughly 118B total parameters and about 8B active parameters per token. It has 256 routed experts, top-10 expert selection, and one shared expert. Of its 48 attention layers, 36 use a 512-token sliding window and 12 use global attention. Its BF16 weights occupy approximately 235GB, making INT4 crucial for running it on previous-generation consumer GPUs. The official vLLM recipe also requires the quantized main model to be paired with a DFlash draft model at the same precision, preventing mismatched numerical formats from degrading the acceptance rate or output quality.
DFlash is not simply a smaller model. It proposes multiple candidate tokens in parallel before each verification step, allowing the main model to validate them in a single pass. The vLLM documentation reports an average accepted length of about 3.1 tokens for natural-language text, with code typically achieving higher values. However, the Laguna configuration must use Triton MoE because the default DeepGEMM backend is incompatible with the draft-model path. In addition, once speculative decoding is enabled, parameters such as `min_p` or `logit_bias` cannot be added arbitrarily, or vLLM will reject the request.
The result shows that large sparse coding models with long context windows can run on secondhand workstation-class hardware, but it cannot be translated directly into multi-user serving capacity. The peak figure was reported without full details on concurrency, power consumption, or the distribution of sustained output speeds. Community quantization tests on Apple Silicon also show that some extremely low-bit variants may fit in memory yet be almost incapable of completing quality evaluations. The next step should be to compare against a same-system baseline without DFlash, measure acceptance rates across different prompt lengths, and evaluate concurrency degradation with a 200K-token KV cache.