最新模型
NVIDIA Releases Nemotron 3.5 Bolt, Compressing a 74B Model with Mixed Precision for Local Agents
On October 15, NVIDIA announced Nemotron 3.5 Bolt 74B-A9B 24GB, combining structural compression, low-bit quantization, and speculative decoding. 24GB is the design target; the official full deployment examples still list 28GB and 32GB configurations, so engineers should recalculate memory needs based on context length and the draft model.
NVIDIA released Nemotron 3.5 Bolt 74B-A9B 24GB on October 15, compressing a large vision-language model into the deployment range for local agents. It also made quantized weights, a BF16 checkpoint, and a speculative decoding draft model available. The official announcement lists supported paths including llama.cpp, Ollama, LM Studio, and vLLM, enabling developers to test local tool-calling and document-understanding workflows. Official announcement
Bolt is compressed from Super VL, a 120B model that activates 12B parameters per token. It retains 74B total parameters and activates about 9B per token. Star Elastic searches for the number of experts and intermediate dimensions in each layer that fit a memory budget, then uses post-training recovery to restore capabilities. Quantization uses mixed precision: the main expert weights use IQ2_XXS at about 2.0625 bits per element, Mamba linear layers use NVFP4, and some other weights remain at Q8_0. So a “2-bit model” does not mean every tensor uses the same precision. Technical details
Inference uses the LiLiCorr draft model. On an RTX Spark, NVIDIA measured about 84 tokens per second on SPEED-Bench-Coding with llama.cpp, but the Qwen3.6 and Gemma 4 configurations in the same table decoded faster. Bolt is positioned as a trade-off among capacity, quality, and speed. These are still vendor measurements and should not be treated as the completion speed of full agent tasks. Benchmark methodology and results
When deploying the model, it is especially important to unpack what “24GB” means. The model card specifies a maximum context length of 256K, but the current llama.cpp guide provides two configurations: 28GB at 65,536 tokens and 32GB at 262,144 tokens. Both use only one parallel sequence. This shows that the capacity target in the model’s name is not a guarantee of memory requirements for a complete service; visual inputs, the cache, and the draft model must also be included in the budget. Model card and deployment guide
For Chinese teams, the next step is to validate Chinese-language understanding, parameter-generation accuracy, and long-task latency using real documents and tool workflows. The primary languages listed by NVIDIA are English and programming languages; Chinese is not included in the supported-language list. The public results available so far are insufficient to establish the quality of agents working in Traditional Chinese. Language and use-case details