模型與開源工具
Gemma 4 12B Directly Processes Images and Audio with a Single Transformer, Reducing Multimodal Deployment Complexity
Google has released the open-weight Gemma 4 12B model, replacing separate vision and audio encoders with linear projections. The new architecture simplifies fine-tuning and local inference, but the official performance figures still require independent verification.
Google’s Gemma 4 12B uses a dense, decoder-only Transformer. Its biggest change is not its parameter count, but the direct integration of multimodal inputs into the language model’s weight path. Traditional vision-language models typically use separate image or audio encoders to extract features before passing them to an LLM. Gemma 4 12B instead projects 48×48-pixel patches into the hidden space with a single matrix multiplication, while 16 kHz audio is divided into 40-millisecond frames and linearly projected. Image positional information is added using factorized X- and Y-coordinate lookup tables.
This “encoder-free” design eliminates multi-stage execution and fragmented memory allocation. Because text, images, and audio ultimately share the same Transformer weights, LoRA or full fine-tuning can modify cross-modal behavior within a single training loop, without separately coordinating frozen vision and audio towers. For teams building document recognition, voice assistant, or video analysis systems, model packaging, memory planning, and inference scheduling could all become simpler.
The 12B version accepts text, images, and audio. Google says it can run on devices equipped with 16 GB of VRAM or unified memory. LiteRT-LM also provides an OpenAI-compatible local API server and prefix caching. The technical report additionally describes a thinking mode, a quantization-aware training variant, and a multi-token prediction draft head for speculative decoding. For long-context processing, the model combines sliding-window and global attention; Google claims this can reduce global KV cache usage by up to 37.5%.
Engineering teams should still independently measure time to first token, VRAM usage curves as audio duration and image count increase, and cross-modal quality after quantization. Although directly projecting raw signals reduces preprocessing, it also couples perception and reasoning more tightly within a single model. Whether this approach can consistently outperform dedicated encoders on specialized imagery, accents, noise, and long videos remains to be established through independent evaluation.