NeoMME starts from zero. H Company researchers released the open-source family on September 3, 2026, with 260M- and 800M-parameter multimodal and multilingual encoders. The models are trained from scratch and published under the Apache 2.0 license.
That approach sets NeoMME apart from many recent visual document retrievers, which are adapted from pretrained generative vision-language models. NeoMME is not based on an existing pretrained vision tower, text encoder, or text decoder — a refreshing refusal to inherit someone else’s foundation and call it a new building.
H Company positions the design against earlier encoder efforts including ModernBERT and ModernVBERT. Both NeoMME sizes share the same architecture, giving the family one structural design across its 260M- and 800M-parameter models.
One Transformer for Text and Images
NeoMME sends text tokens and raw image patches through one shared bidirectional Transformer. Text inputs use factorized token embeddings, while images are divided into non-overlapping 32×32 patches and projected with a small multilayer perceptron.
Both input types enter the same Transformer encoder, rather than passing through separate pretrained towers. Images keep their aspect ratio and size, and the model supports a context length of 16,384 tokens.
Most layers use symmetric sliding-window attention to handle the long context. Every sixth layer and the final layer use global attention, creating points where information can cross the full sequence instead of remaining inside local windows.
The architecture also includes grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs. These components sit inside the shared encoder rather than being split between dedicated text and image systems.
Pretraining Across Text, Code, and Documents
H Company trained a BPE tokenizer from scratch with a 131,072-token vocabulary. Its training material includes multilingual text, code, mathematics, and machine-produced image transcripts.
NeoMME is pretrained from scratch as a discrete masked-diffusion text denoiser. For text-only examples, the corruption rate is sampled uniformly between 0 and 1; multimodal examples use corruption rates between 0.3 and 1.
The pretraining mix covers multilingual text, code, mathematics, natural images, and document images. Each model processes about 524 billion packed input tokens, including 290 billion tokens from text-only examples.
That training recipe gives the encoders a broad input mix without relying on an existing pretrained vision tower, text encoder, or text decoder. The result is still an encoder family, not a generative vision-language model wearing an encoder badge.
Fine-tuned retrieval variants of NeoMME sit on the model-size Pareto frontier of the ViDoRe v3 visual document retrieval benchmark. The result links the from-scratch architecture to a concrete retrieval position across its available model sizes.
NeoMME therefore combines a shared bidirectional Transformer, long-context processing, multilingual tokenization, and image patches inside one open-source family. Its defining choice is straightforward: build the multimodal encoder itself, train it across text and images, and release it under Apache 2.0.
