邊緣 AI／推論系統
28.9M-Parameter Language Model Runs on ESP32-S3, Using Per-Layer Embeddings to Bypass the 512KB SRAM Limit
A developer deployed a 4-bit, 28.9M-parameter language model on an approximately $8 ESP32-S3, achieving a measured generation speed of about 9.5 tokens/s without any network connection. The key is not loading the entire model into memory, but keeping the Per-Layer Embeddings—which account for most of the parameters—in flash and reading only the rows required for each token.
A recent experiment that has attracted attention in the developer community deployed a 28.9M-parameter language model on an ESP32-S3 microcontroller. The test hardware has only 512KB of SRAM, 8MB of PSRAM, and 16MB of flash. After quantization, the model is about 14.9MB, with a measured end-to-end generation speed of 9.5 tokens per second. Inference, sampling, and text output are all performed on-device.
The design uses Per-Layer Embeddings derived from the Gemma family. Around 25 million of the model’s parameters reside in embedding lookup tables and do not need to participate in matrix operations for every token, so they remain in the slower but higher-capacity flash memory. Each inference step retrieves only about six rows, totaling approximately 450 bytes. Frequently reused compute kernels are stored in SRAM, while the output head and workspace use PSRAM. This memory hierarchy avoids moving the complete weights into high-speed memory and prevents the capacity of large lookup tables from translating directly into an equivalent computational cost.
The model was trained on the synthetic TinyStories dataset and is primarily capable of generating short, simple English stories. It does not provide reliable question answering, instruction following, code generation, or factual knowledge. Consequently, its 28.9M parameters cannot be compared directly with the effective capacity of general-purpose LLMs: most are sparsely accessed embedding data, while the core that operates on every token remains very small.
The project has released its C firmware along with training, ablation, and quantization code, providing an inspectable path to reproduction. From an engineering perspective, the more important issues to examine are flash random-read latency, power consumption, and endurance, as well as whether the same approach can be extended to sensor classification, voice commands, or compact tool models. The current single-board results are still insufficient to demonstrate suitability for long-running, real-time, or battery-powered products.