ByteDance just dropped Lance—a single AI model that handles image and video understanding, generation, and editing all at once. That’s a rare feat. Usually, these tasks demand separate systems because they pull in opposite directions.

Understanding images and videos relies on semantic, language-aligned features. Generating them needs detailed, continuous data that preserves texture, geometry, and motion. Most companies build separate models and try to patch them together after training. ByteDance skipped the patchwork.

Lance processes text, images, and videos as one shared sequence of tokens. It uses a clever mix of semantic visual tokens for understanding and continuous latent tokens for generation. Both live in the same context and run through a novel 3D causal attention mechanism. Text tokens see the past only; visual tokens get full bidirectional attention.

The model splits into two “experts.” One focuses on understanding and reasoning from text and semantic visuals. The other handles generation and editing in the continuous latent space. They share context but don’t fight over parameters. Training balances next-token prediction for understanding with flow matching for generation.

A tricky detail: visual tokens of different types get jumbled in the same sequence. ByteDance fixes this with Modality-Aware Rotary Positional Encoding (MaPE). MaPE shifts token positions by modality, keeping spatial order intact but separating groups in time. Without it, performance drops noticeably across generation, editing, and understanding tasks.

Lance’s training pipeline spans four stages. It starts with pre-training on a massive mix of image-text and video-text pairs. Then continual training blends multi-task data—editing, generation, and understanding samples—gradually increasing editing difficulty. Next, supervised fine-tuning sharpens instruction following and identity preservation. The final stage adds reinforcement learning to polish performance.

The result is a rare unified AI that natively bridges image and video tasks from captioning and visual question answering to subject-driven generation and multi-turn editing. It outpaces typical systems that split understanding and generation or limit themselves to images only.

ByteDance’s Lance isn’t just academic. Its architecture suggests a future where AI models handle complex, multimodal workflows without stitching together separate components. That could simplify pipelines, reduce latency, and improve consistency—especially in video editing where temporal coherence is critical.

In a field crowded with specialized tools, Lance stakes out a bold claim: one model to rule all visual modalities and tasks. If it scales well beyond lab benchmarks, it might rewrite how we build creative AI systems for both content creators and enterprise use.

For now, Lance stands as a technical milestone—proof that integration beats separation when done right. ByteDance’s approach to combining discrete token types and dual experts inside a single transformer architecture could inspire similar innovations across AI research.