多模態檢索
NeoMME Encodes Text and Images with a Single Bidirectional Transformer, Compressing Visual Document Indexes to 6 kB per Page
Hcompany has open-sourced the 260M- and 800M-parameter NeoMME models, which process text tokens and raw image patches with the same encoder instead of attaching a vision tower or causal decoder. Its retrieval variants produce both dense and late-interaction representations, although the current efficiency and accuracy figures come primarily from evaluations conducted by the authors.
Hcompany has released NeoMME-260M and NeoMME-800M, along with Retriever variants fine-tuned for visual document search. Rather than connecting a SigLIP-style vision tower to a generative language model, NeoMME divides raw images into 32×32 patches and, after applying separate projections, feeds them and text tokens into the same bidirectional Transformer. Both model sizes support 16,384 tokens. Most layers use symmetric sliding-window attention, with global attention applied every six layers and in the final layer to control the computational cost of high-resolution pages.
Its pretraining also differs from that of conventional autoregressive VLMs. The model is trained from scratch using a masked discrete-diffusion text objective: patches from document images remain visible while the model reconstructs masked text conditioned on the images, with no pixel-reconstruction loss. It therefore outputs contextualized representations rather than directly generating answers; the base weights still need task-specific heads for retrieval, classification, or information extraction.
In a single forward pass, NeoMME-Retriever produces both a single-vector dense embedding and a late-interaction embedding that preserves token- and patch-level granularity. The latter can generally retain layout details but results in very large indexes. By combining hierarchical token pooling with asymmetric quantization, the team reduced NeoMME-260M’s per-page representation on ViDoRe v3 from about 1.5 MB to 6 kB—a 255-fold reduction—while retaining more than 95% of the original nDCG@10.
The authors report nDCG@10 scores of 0.523 and 0.556 on ViDoRe v3 for the 260M and 800M models, respectively. With 2048×2048 inputs on an NVIDIA L40S, the 260M model encodes 51.3 pages per second, 1.97 times the throughput of ColModernVBERT. For Chinese-language PDF RAG, the significance goes beyond the smaller model size: indexing page images directly allows tables, columns, and scanned layouts to contribute to retrieval before OCR, while representations of roughly 6 kB make late-interaction indexes for large document collections more practical to deploy.
All checkpoints are licensed under Apache 2.0 and have been integrated with Transformers and Sentence Transformers. However, some comparison-model scores come from MTEB, whereas the NeoMME evaluations were run by the authors themselves; the speed results also cover only one GPU and one resolution. The model card explicitly states that comprehensive safety, bias, and privacy evaluations have not yet been completed, and that the base model cannot be used directly as a generative assistant. The next areas to watch are independent reproductions, Traditional Chinese scanned documents, different input resolutions, and the recall of compressed indexes on real-world multilingual data.