NeoMME puts text and images inside one encoder. The system introduces multilingual multimodal models in 260M and 800M sizes, without relying on a separate pretrained vision tower, pretrained text encoder or decoder, or causal language model. A single bidirectional Transformer processes text tokens and raw image patches together.
That design makes NeoMME a foundation encoder for generating vector representations from text, images, or both. The model family was trained from scratch with a masked discrete-diffusion objective, treating multimodal encoding as one shared problem instead of stitching together separate towers and hoping the pieces cooperate.
One architecture for text and image inputs
NeoMME divides each image into non-overlapping 32×32 patches, then projects those patches with a small MLP. Images keep their aspect ratio and size, allowing the encoder to process visual inputs without reducing them to a fixed shape before they enter the Transformer.
The model supports a context length of 16,384 tokens. Most layers use symmetric sliding-window attention, while every sixth layer and the final layer use global attention, giving the encoder local processing with scheduled access to the full sequence.
NeoMME also uses grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs. Its BPE tokenizer has a 131k-token vocabulary trained from scratch on multilingual text, code, mathematics, and machine-produced image transcripts — a broad menu for a model that is expected to handle more than ordinary prose.
Pretraining follows a masked discrete-diffusion text-denoising process. For each text-only example, the system samples a corruption rate uniformly between 0 and 1, then masks eligible text tokens at that rate. Multimodal examples use corruption rates between 0.3 and 1; image patches remain visible while NeoMME reconstructs the masked text.
Retrieval performance without the storage bloat
NeoMME was fine-tuned for visual document retrieval using ColPali’s page-image approach. The resulting NeoMME-Retriever returns dense and late-interaction embeddings in one forward pass, combining two retrieval representations without requiring separate encoding runs.
Both model sizes sit on the ViDoRe v3 Pareto frontier for nDCG@10 and model size. That places the 260M and 800M encoders in the useful space where retrieval quality and model scale are measured together, rather than treating a larger checkpoint as an automatic victory.
The 260M model also targets practical document-processing workloads. At a matched 2048×2048 image input size on an NVIDIA L40S GPU, it encodes about 51 pages per second — about twice the throughput of ColModernVBERT.
Storage is another target. Hierarchical token pooling and asymmetric quantization reduce late-interaction index storage from roughly 1.5 MB to 6 kB per page. That is a dramatic reduction in the size of page-level indexes, where every stored embedding can multiply across large document collections.
The trade-off is architectural discipline: NeoMME keeps image patches visible during multimodal denoising, uses a shared bidirectional Transformer, and produces retrieval embeddings in one pass. The approach does not remove the engineering demands of visual document search, but it compresses several moving parts into one encoder — a refreshing change from the usual tower-and-adapter furniture.
NeoMME is available in Hugging Face Transformers, and all model checkpoints are released under the Apache 2.0 license. The combination of 260M and 800M options, multilingual input, multimodal processing, retrieval support, and reduced index storage gives the project a clear focus: efficient encoding rather than another text-only model wearing a vision accessory.
