硬體
esp32-ai Fits a 28.9M-Parameter Model on the ESP32-S3, Reaching 9.88 Tokens/s Offline
The open-source esp32-ai project uses 4-bit quantization and tiered memory allocation to keep roughly 25 million embedding parameters in flash, reading only the rows needed for each token. The model can generate text offline on a microcontroller costing under $10, though its capabilities remain limited to short stories and single-domain question answering.
Developer slvDev has released esp32-ai, demonstrating a 28.9-million-parameter language model generating text directly on an ESP32-S3 microcontroller without a network connection or server. The project reports an end-to-end throughput of 9.88 tokens per second. The target hardware has just 512KB of SRAM, 8MB of PSRAM, and 16MB of flash, while the 4-bit model file is 14.9MB. The code, firmware, training pipeline, and quantization workflow are available on GitHub under the MIT License. The repository has recently accumulated about 3,800 stars, providing a clear signal of community adoption.
The key is not repeatedly moving the entire model into RAM, but distributing parameters across memory tiers according to access frequency. Activations and normalization weights, which are used multiple times for every token, remain in the fastest SRAM. The dense core and output head, each scanned once per position, reside in PSRAM. Lookup tables containing roughly 25 million parameters remain in slower flash. During inference, each token requires reading about six rows, totaling roughly 450 bytes, so most weights never need to be loaded into working memory. This approach borrows Gemma 3n’s Per-Layer Embeddings (PLE), but scales the mobile-class memory strategy down to a microcontroller.
The project provides both a TinyStories story-generation model and a Barista coffee question-answering model. Its download script verifies model assets against fixed SHA-256 hashes and file sizes, while the deployment script generates headers, compiles the firmware, and flashes the device. The research directory also includes ablation and quantization code, allowing hardware developers to inspect the published figures rather than relying solely on a demonstration video.
The limitations are equally clear. The dense core responsible for sequence modeling contains only about 4 million parameters; most of the remaining parameters belong to lookup tables. The TinyStories version cannot reliably follow instructions, answer factual questions, or write code. The reported 9.88 tokens/s was also measured by the project author using a specific board and model, with no cross-chip comparison or data on power consumption and long-term thermal stability yet available. What the project demonstrates is that memory allocation can increase the size of models that an MCU can accommodate—not that microcontrollers can already replace general-purpose LLMs. The next questions are whether PLE can support genuinely embedded workloads such as sensor classification and voice commands, and how random flash reads affect power consumption and device longevity.