ESP32-S3 Runs a 28.9-Million-Parameter Model from 16MB Flash Using Hierarchical Embeddings, Generating About 9.5 Tokens per Second

The open-source esp32-ai project has recently attracted attention in the technical community: its developers deployed a 28.9-million-parameter language model, approximately 14.9MB after 4-bit quantization, on an ESP32-S3. The test hardware has only a dual-core 240MHz LX7 CPU, 512KB of SRAM, 8MB of PSRAM, and 16MB of Flash, yet the model can generate short stories at roughly 9 to 9.5 tokens per second. Inference requires no network connection, smartphone, or external server. The project also provides workflows for training, export, quantization, host-side numerical validation, and flashing, rather than merely presenting a prerecorded demo.

The key is not a faster matrix-compute core, but a reorganization of where parameters reside in the memory hierarchy. Roughly 25 million parameters belong to hierarchical embedding tables and are stored in memory-mapped Flash; for each generated token, only a small number of relevant rows are read. The dense core responsible for the actual Transformer computation contains about 560,000 parameters and remains in faster SRAM or PSRAM along with the working buffers. This is similar to the per-layer embeddings approach Google uses in Gemma 3n: large lookup data remains in inexpensive, slower memory, avoiding the need to move the entire model at every step. For memory-bandwidth-constrained MCUs, this matters more than simply reducing weight size.

The engineering significance lies in demonstrating that generative models can become part of firmware, making them suitable for research into offline text interfaces, narrow-domain classification, sensor-data summarization, and fallback processing during network outages. Data does not need to leave the device, and both cloud latency and recurring costs can be reduced. However, the current model primarily uses TinyStories as its data source, so the demonstration reflects simple narrative generation and pattern learning—not reliable question answering, reasoning, or controller functionality. Its parameter count is also inflated by the large embedding tables and therefore cannot be compared directly with a dense model of similar size. The next areas to examine are accuracy on real-world instruction data, peak memory usage and power measurements, and whether erroneous outputs can be safely constrained by deterministic firmware rules.