邊緣 AI 與工具呼叫
Needle 2 Compresses a 45M-Parameter Tool Model into 14MB and Constrains On-Device Calls with Schema Grammars
Cactus Compute recently improved Needle 2’s Python API, cross-platform execution, and Apache 2.0 licensing, while continuing to update its model card. It performs tool selection and structured extraction with 28MB of peak memory, although its speed and accuracy claims still come primarily from official tests.
Needle 2 shrinks a tool-calling model to a size that can be embedded in phones, browsers, Raspberry Pi devices, and even some microcontrollers. The model has just 45 million parameters and is compressed with Cactus Quants into an approximately 14MB CQ2 two-bit binary. According to the company, a complete session uses around 28MB of peak memory. Around August 20, the project released a series of Python package updates that added type hints to the public API, Python 3.9 compatibility, a fix for CLI crashes, and platform-specific download modes. The model card was updated again on August 22, and the license for both the code and weights has changed from MIT to Apache 2.0.
The architecture is not simply a smaller Transformer. Needle 2’s Simple Attention Network replaces conventional FFNs with fixed Walsh–Hadamard transforms paired with learnable gates. It also incorporates GQA, an engram key-value memory based on hashed n-grams, and multi-lane hyper-connections with four residual streams. Inference uses a 256-token sliding window while keeping tool descriptions fixed as a KV sink, allowing memory usage to remain bounded as conversations grow. When more than five tools are available, a built-in retrieval head first creates vectors for their schemas and passes only the five highest-scoring tools to the generator. Tools that are not selected are removed from the grammar for that call instead of merely having their sampling probabilities reduced.
On the output side, JSON Schema is compiled into a byte-level grammar, allowing field types, enum values, ranges, and string formats to be enforced during decoding. Each call also includes a confidence score calculated as the lower of the calibrated head’s score and the call-token probability, enabling applications to use a threshold to decide whether to execute the call directly or hand it off to a cloud model. However, the grammar can guarantee only syntactic validity; it cannot ensure that the correct tool was selected or that the parameters are semantically correct. The 256-token window is also unsuitable for long-form reasoning. The company claims that a Raspberry Pi 5 can decode 500 tokens per second, but standardized hardware, power-consumption, and third-party accuracy benchmarks are still lacking. Permission checks and confirmation steps therefore remain necessary before deploying the model to control physical systems.