Awesome Multimodal Modeling

A Comprehensive Survey & Curated List of Multimodal Modeling
_{From Traditional Fusion to Native & Unified Architectures}

_{Overview · Traditional · MLLMs · UMMs · NMMs ·Closed Source Models · Resources}

📢 News

[2026-10-19] Updated the curated model and design-analysis coverage; added a dedicated Benchmarks & Evaluation section and matching website collection, with explicit date and release-status qualifications.
[2026-06-06] 🚀 We released two new open-source model collections:
- Unified Multimodal Model Zoo (UMM Zoo) – a curated collection of unified multimodal models for understanding, generation, reasoning, and editing.
- Native Multimodal Model Zoo (NMM Zoo) – a curated collection of native multimodal models that natively integrate perception, reasoning, and generation.
[2026-04-13] ⭐ The repository has already gained over 100 stars in just one day! Thank you all for the incredible support. We will keep updating this list with more cutting-edge models and resources. Your continued stars and PRs are warmly welcomed!
[2026-04-12] 🎉 We are excited to launch Awesome Multimodal Modeling — a curated reading list organized by architectural paradigms. A comprehensive survey paper is coming soon! Stay tuned.

About This List

In this section: At a Glance · Curation Principles

This repository provides a structured, community-maintained survey of multimodal models, covering the full evolutionary arc from early fusion methods to today's natively-trained omni-models. We emphasize precise architectural definitions and classification, especially for the often-conflated categories of Unified Multimodal Models (UMMs) and Native Multimodal Models (NMMs).

Scope: Primary focus on image + text modalities; audio/video/3D are annotated where present. Omni/any-to-any models are marked with Omni.

At a Glance

Dimension	Coverage
Primary scope	Image + text multimodal models, with explicit annotations for video, audio, and omni extensions
Core taxonomy	Traditional multimodal models, MLLMs, UMMs, and NMMs with explicit architecture/training qualifications
Key distinction	`U+G unification` for UMMs; architectural and optimization coupling for NMMs, with overlapping membership allowed
What makes this repo different	Architecture-first categorization, fusion-aware definitions, and curated links to adjacent awesome lists
Intended audience	Researchers, students, and engineers building or surveying multimodal systems

Curation Principles

Coverage audit: the additions and release-status updates in this revision were checked against primary sources through 2026-10-19. This is a curated selection, not an exhaustive index of every paper published by that date.

Principle	Rule
Source quality	Prefer official conference proceedings, OpenReview, ACL Anthology, CVF Open Access, arXiv, and official project pages
Classification policy	Category assignment is based on this repository's architecture-first definitions, which may differ from authors' own branding
Venue policy	Prefer the published venue; label author-reported acceptances where proceedings have not been checked. Keep first-paper dates separate from conference years
Date policy	`First public: YYYY-MM-DD` records the earliest verified exact-date public announcement; retain month precision where only a month is known. Distinguish paper submission, API availability, weights release and revision dates. Website year filters use the venue/release year; chronological sorting uses the explicit first-public date where recorded, otherwise the dated venue/release
Availability policy	Distinguish code, downloadable weights, API previews and future promises; record model-weight licenses separately from code licenses
Scope discipline	Models, benchmarks, datasets, and analysis papers are tracked separately to avoid mixing artifacts
Inclusion bar	We prioritize landmark papers, broadly adopted benchmarks, open implementations, or papers that clarify important taxonomy boundaries

Classification note: for ambiguous models sitting between MLLM, UMM, and NMM, this list records the category that best matches the training recipe and architectural coupling, not just the paper title. Architecture-native and training-native qualifications distinguish fusion structure from training history.

Back to Top

1. Introduction & Definitions

In this section: 1.1 Multimodal Model Evolution Stages · 1.2 Scope & Taxonomy · 1.3 Architecture Diagrams

1.1 Multimodal Model Evolution Stages

Subtopics: Traditional Multimodal Models · Multimodal Large Language Models (MLLMs) · Unified Multimodal Models (UMMs) · Native Multimodal Models (NMMs)

We use the following precise, architecture-first definitions throughout this list. Understanding these distinctions is critical for correctly classifying modern models.

Traditional Multimodal Models

Pre-2023 mainstream era

Representation alignment or fusion of modality-specific processing streams, without a large language model backbone. Covers cross-modal retrieval, classification and captioning; alignment-only dual encoders and models with joint fusion layers are distinguished by their architectures. Examples: CLIP, ALIGN, ViLBERT, BLIP.

Multimodal Large Language Models (MLLMs)

Pretrained-backbone multimodal language models

Combine a pretrained visual backbone or visual abstractor (e.g., ViT/CLIP/SigLIP, Q-Former, cross-attention adapter) with a pretrained LLM through a connector. The defining property is inheritance from strong pretrained unimodal backbones rather than joint multimodal pretraining from scratch. These models are primarily text-output understanding/reasoning systems, even when auxiliary generators are attached externally.

Key characteristics:

✅ Pretrained visual encoder / abstractor
✅ Pretrained LLM backbone
✅ Connector layer or cross-attention bridge
❌ No end-to-end multimodal pretraining from scratch
❌ No native image generation inside the same backbone

Examples: LLaVA, Qwen-VL, InternVL, MiniCPM-V, CogVLM

Unified Multimodal Models (UMMs)

Single framework for Understanding + Generation (U+G)

A single framework that handles both multimodal understanding and visual generation. UMMs may reuse pretrained components or modular tokenizers; the defining feature is U+G unification, not whether the model is trained from scratch.

Key characteristics:

✅ Unified understanding + generation
✅ Shared model interface or shared backbone for U+G
⚠️ May use pretrained components
⚠️ May use decoupled encoders / modular tokenizers
⚠️ If a model is also natively trained from scratch, its architectural details belong primarily in NMMs (§5)

Examples: Show-o, Janus, OpenUni, BAGEL, BLIP3-o

Native Multimodal Models (NMMs)

Architectural integration and multimodal optimization

NMMs are described along two dimensions: architectural nativity captures how early and persistently modalities share core computation; optimization nativity captures when multimodal objectives enter training and which core parameters they optimize. Entries that establish only one dimension are qualified as architecture-native or training-native. Training from scratch provides strong optimization evidence, but is not assumed for every entry. UMM and NMM membership can overlap.

Key characteristics:

✅ Fusion structure and training history are recorded separately
✅ Pretrained components and initialization are annotated per model
✅ Architecture-native entries may inherit pretrained backbones
✅ Input: text tokens + image patches/tokens
✅ Output: text (understanding focus; generation optional)

NMMs are described by fusion topology and input architecture, with initialization recorded separately. §5.2 records supported early-fusion architectures; §5.3 groups encoder-based systems without inferring fusion depth from encoder presence. Training-native entries without a confirmed architectural grouping are listed separately in §5.5:

NMM — Early Fusion

Modality tokens or embeddings enter shared core computation before the first shared backbone block. Separate tokenizers, input encoders, modality-specific experts, or output decoders can remain; their presence does not by itself imply late fusion.

Shared backbone computation over multimodal states
Discrete tokens, continuous latents, or pixel patches
Modality interaction from the first shared block
Initialization and trainable components annotated separately
Examples: Chameleon, Emu3, Transfusion; Show-o, Show-o2, OneCat, and Llama 4 carry architecture-native qualifications

NMM — Late Fusion

Cross-modal interaction begins only after modality-specific backbone layers, rather than at the first shared block. This is a claim about where modalities interact, independent of whether the components were pretrained or initialized from scratch.

Identify the modality-specific stages and the first interaction layer
Record initialization and multimodal optimization separately
A dedicated vision encoder alone does not establish this topology; encoder-based systems with unverified fusion depth are grouped in §5.3

1.2 Scope & Taxonomy

Multimodal Models
├── 2. Traditional Multimodal Models
│   ├── 2.1 Multimodal Representations & Alignment
│   │   ├── Multimodal Representations
│   │   ├── Multimodal Fusion
│   │   └── Multimodal Alignment
│   └── 2.2 Multimodal Pretraining
├── 3. Multimodal Large Language Models (MLLMs)
│   ├── 3.1 Taxonomy Based on Vision Adapter
│   └── 3.2 Omni MLLMs
├── 4. Unified Multimodal Models (UMMs)
│   ├── 4.1 Taxonomy by Generation Paradigm
│   │   ├── Diffusion-Based UMMs
│   │   ├── Autoregressive (AR) UMMs
│   │   │   ├── Pixel Encoding
│   │   │   ├── Semantic Encoding
│   │   │   ├── Learnable Query Encoding
│   │   │   ├── Hybrid Encoding (Pseudo)
│   │   │   └── Hybrid Encoding (Joint)
│   │   └── Hybrid (AR + Diffusion) UMMs
│   │       ├── Pixel Encoding
│   │       └── Hybrid Encoding
│   ├── 4.2 Any-to-Any / Omni UMMs
│   └── 4.3 Related Generators (U+G Unverified)
└── 5. Native Multimodal Models (NMMs)
    ├── 5.1 Design Analyses & Scaling Laws
    ├── 5.2 Early Fusion NMMs
    ├── 5.3 Encoder-Based NMMs
    ├── 5.4 Any-to-Any / Omni NMMs
    └── 5.5 Training-Native Models

1.3 Architecture Diagrams

The NMM diagrams below distinguish fusion topology from input architecture. The early-fusion diagram illustrates a from-scratch variant; initialization and input interfaces are specified per model in §5. Encoder-based entries do not inherit a strict late-fusion label from these schematic examples.

┌─────────────────────────────────────────────────────────────────┐
│           TRADITIONAL MULTIMODAL MODEL                          │
│                                                                 │
│  [Image] ──► [CNN/ViT Encoder] ──┐                              │
│                                  ├──► [Fusion] ──► [Output]     │
│  [Text]  ──► [LSTM/BERT]       ──┘                              │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│           MLLM — PRETRAINED BACKBONES + VISUAL INTERFACE         │
│                                                                 │
│  [Image] ──► [Pretrained ViT/CLIP] ──► [Projector/Q-Former]     │
│                                                │                │
│                                                ▼                │
│  [Text]  ──────────────────────────► [Pretrained LLM] ──► [Text]│
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│           UMM — UNIFIED UNDERSTANDING + GENERATION              │
│                                                                 │
│  [Image/Text Input] ──► [Shared/Modular Tokenizer]              │
│                                │                                │
│                                ▼                                │
│                   [Unified Transformer]                         │
│                       │           │                             │
│                       ▼           ▼                             │
│             [Text Output]      [Image Output]                   │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│           NMM — EARLY FUSION (Trained from Scratch)             │
│                                                                 │
│  [Text tokens] ──┐                                              │
│                  └──► [Single Decoder Transformer] ──► [Text]   │
│  [Image patches ──► Linear Patchify] ──┘                        │
│   (raw pixels, minimal preprocessing)                           │
│   Multimodal interaction from Layer 1                           │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│           NMM — ENCODER-BASED INPUT ARCHITECTURE               │
│                                                                 │
│  [Image] ──► [Vision Encoder] ──► [Visual Interface]           │
│                                         │                       │
│                                         ▼                       │
│  [Text]  ──────────────────────► [Backbone] ──► [Text]         │
│   Initialization and first interaction layer recorded per model │
└─────────────────────────────────────────────────────────────────┘

Back to Top

2. Traditional Multimodal Models

In this section: 2.1 Multimodal Representations & Alignment · 2.2 Multimodal Pretraining

Pre-chat-MLLM and non-native multimodal systems that established the basic vocabulary of alignment, fusion, retrieval, captioning, and multimodal pretraining.

2.1 Multimodal Representations & Alignment

Subtopics: Multimodal Representations · Multimodal Fusion · Multimodal Alignment

Multimodal Representations

Paper	Venue	Links	Notes	Task
Identifiability Results for Multimodal Contrastive Learning	ICLR 2023	Paper	Theoretical identifiability analysis of contrastive multimodal learning	representation learning
Unpaired Vision-Language Pre-training via Cross-Modal CutMix	ICML 2022	Paper	Introduces CutMix-style augmentation for unpaired VLP	vision-language pretraining
Balanced Multimodal Learning via On-the-fly Gradient Modulation	CVPR 2022	Paper	Balances modality learning via dynamic gradient reweighting	multimodal optimization
FLAVA: A Foundational Language And Vision Alignment Model	arXiv 2021	Paper	Unified architecture for vision-language understanding and generation	foundation multimodal model
UniT: Multimodal Multitask Learning with a Unified Transformer	arXiv 2021	Paper	Single transformer for multiple multimodal tasks	multimodal multitask learning
Perceiver: General Perception with Iterative Attention	ICML 2021	Paper	General-purpose architecture for high-dimensional multimodal inputs	general multimodal architecture
Learning Transferable Visual Models From Natural Language Supervision	ICML 2021	Paper	Contrastive vision-language pretraining at scale	vision-language contrastive learning
VinVL: Revisiting Visual Representations in Vision-Language Models	arXiv 2021	Paper	Improved visual features for VL tasks	vision-language representation improvement
12-in-1: Multi-Task Vision and Language Representation Learning	CVPR 2020	Paper	Unified multi-task learning across 12 VL tasks	multi-task learning
Learning Video Representations using Contrastive Bidirectional Transformer	arXiv 2019	Paper	Contrastive transformer for video representation learning	video contrastive learning
OmniNet: A Unified Architecture for Multi-modal Multi-task Learning	arXiv 2019	Paper	Unified encoder-decoder for multimodal tasks	unified multimodal architecture
Learning Representations by Maximizing Mutual Information Across Views	arXiv 2019	Paper	InfoMax principle for cross-view representation learning	self-supervised learning
ViCo: Word Embeddings from Visual Co-occurrences	ICCV 2019	Paper	Learning word embeddings from visual context	vision-language embeddings
Learning Factorized Multimodal Representations	ICLR 2019	Paper	Factorized latent space for multimodal data	representation disentanglement
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping	NeurIPS 2014	Paper	Fragment-level image-sentence alignment	vision-language alignment
DeViSE: A Deep Visual-Semantic Embedding Model	NeurIPS 2013	Paper	Early deep vision-to-language embedding model	vision-language embedding
Multimodal Deep Learning	ICML 2011	Paper	Foundational multimodal deep learning framework	multimodal deep learning

Multimodal Fusion

Paper	Venue	Links	Notes	Task
Robust Contrastive Learning against Noisy Views	arXiv 2022	Paper	Robust contrastive learning under noisy multi-view inputs	contrastive learning
Attention Bottlenecks for Multimodal Fusion	NeurIPS 2021	Paper	Introduces bottleneck attention mechanism for efficient multimodal fusion	multimodal fusion
VMLoc: Variational Fusion For Learning-Based Multimodal Camera Localization	AAAI 2021	Paper	Variational multimodal fusion for camera localization tasks	multimodal localization
Trusted Multi-View Classification	ICLR 2021	Paper	Confidence-aware weighting for multi-view classification	multi-view classification
Deep-HOSeq: Deep Higher-Order Sequence Fusion for Multimodal Sentiment Analysis	ICDM 2020	Paper	Higher-order sequence fusion for multimodal sentiment analysis	multimodal sentiment analysis
What Makes Training Multi-Modal Classification Networks Hard?	CVPR 2020	Paper	Analyzes optimization challenges in multimodal classification	theoretical/empirical analysis
DeepCU: Integrating Both Common and Unique Latent Information for Multimodal Sentiment Analysis	IJCAI 2019	Paper	Separates shared and private latent representations for fusion	multimodal sentiment analysis
XFlow: Cross-modal Deep Neural Networks for Audiovisual Classification	IEEE TNNLS 2019	Paper	Cross-modal feature exchange network for audio-visual tasks	audio-visual classification
MFAS: Multimodal Fusion Architecture Search	CVPR 2019	Paper	Neural architecture search for optimal multimodal fusion design	architecture search
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision	ICLR 2019	Paper	Neuro-symbolic model combining perception and reasoning	neuro-symbolic learning
Efficient Low-rank Multimodal Fusion with Modality-Specific Factors	ACL 2018	Paper	Low-rank factorization for efficient multimodal fusion	efficient fusion
Memory Fusion Network for Multi-view Sequential Learning	AAAI 2018	Paper	Memory-based fusion across temporal multimodal sequences	sequential multimodal learning
Tensor Fusion Network for Multimodal Sentiment Analysis	EMNLP 2017	Paper	Tensor-based full interaction modeling across modalities	multimodal sentiment analysis

Multimodal Alignment

Paper	Venue	Links	Notes	Task
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment	arXiv 2023	Paper Code	Language-centered alignment across image, video, audio, depth, thermal, and IMU modalities	multimodal alignment
CLIP	ICML 2021	Paper	400M+ image-text pairs; dual-encoder (Vision Transformer + Text Transformer); contrastive alignment of separate image/text embeddings; no joint image-text backbone	zero-shot image classification, retrieval
CoMIR: Contrastive Multimodal Image Representation for Registration	NeurIPS 2020	Paper	Contrastive learning for multimodal image registration alignment	multimodal alignment
Multimodal Transformer for Unaligned Multimodal Language Sequences	ACL 2019	Paper	Transformer-based alignment for unaligned multimodal sequences	sequence alignment
Temporal Cycle-Consistency Learning	CVPR 2019	Paper	Uses cycle-consistency for temporal cross-modal alignment	temporal alignment
Deep Canonical Correlation Analysis	ICML 2013	Paper	Deep learning extension of CCA for cross-view representation alignment	representation alignment

2.2 Multimodal Pretraining

Paper	Venue	Links	Notes	Task
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation	NeurIPS 2021 Spotlight	Paper	Momentum distillation for aligning vision-language representations before fusion	vision-language pretraining
Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling	CVPR 2021	Paper	Sparse frame sampling for efficient video-language pretraining	video-language pretraining
UniT: Multimodal Multitask Learning with a Unified Transformer	arXiv 2021	Paper	Unified transformer for multitask multimodal learning	unified multimodal pretraining
Large-Scale Adversarial Training for Vision-and-Language Representation Learning	NeurIPS 2020	Paper	Adversarial training improves robustness of vision-language representations	robust multimodal pretraining
Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision	EMNLP 2020	Paper	Grounds language tokens in visual context via voken supervision	vision-grounded language modeling
Integrating Multimodal Information in Large Pretrained Transformers	ACL 2020	Paper	Injects multimodal signals into large pretrained transformer architectures	multimodal transformer pretraining
VL-BERT: Pre-training of Generic Visual-Linguistic Representations	arXiv 2019	Paper	Joint vision-language BERT-style pretraining	vision-language pretraining
VisualBERT: A Simple and Performant Baseline for Vision and Language	arXiv 2019	Paper	Early unified transformer for vision-language understanding	vision-language pretraining
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks	NeurIPS 2019	Paper	Two-stream transformer for cross-modal vision-language learning	vision-language pretraining
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training	arXiv 2019	Paper	Cross-modal encoder for universal vision-language representations	vision-language pretraining
LXMERT: Learning Cross-Modality Encoder Representations from Transformers	EMNLP 2019	Paper	Cross-modality transformer encoder for vision-language reasoning	vision-language pretraining
VideoBERT: A Joint Model for Video and Language Representation Learning	ICCV 2019	Paper	Joint discrete token modeling for video and language	video-language pretraining

Back to Top

3. Multimodal Large Language Models (MLLMs)

In this section: 3.1 Taxonomy Based on Vision Adapter · 3.2 Omni MLLMs

Models that connect a pretrained visual encoder / abstractor to a pretrained LLM. Primarily text-output understanding and reasoning systems, defined by inherited pretrained unimodal backbones rather than multimodal pretraining from scratch.

3.1 Taxonomy Based on Vision Adapter

These groups describe the vision-to-language interface, rather than attention variants inside the language backbone. MLP/Others includes linear projections, spatial merging and pixel shuffle; Q-Former covers query-transformer connectors; Cross-Attention refers to bridges attending to separate visual features. Hybrid Adaptor covers combinations such as learned pooling plus projection, or projection plus visual experts inside the language backbone.

MLP/Others Projector

Paper	Venue	Links	Notes	Task
WeVisDoc: From Coverage to Capability for Robust End-to-End Document Parsing	arXiv 2026-09-17	Paper Code HF	Paper first submitted: 2026-09-17. Fine-tunes Qwen3-VL-2B/4B-Instruct, retaining their visual encoder and merger/projector; page images and instructions produce structured Markdown, LaTeX formulas and HTML tables. Coverage-driven and residual-error-driven training target degraded documents. Code and 2B/4B checkpoints available as of 2026-10-19.	document parsing, OCR, table/formula recognition, robustness
VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understanding	arXiv 2026-09-14	Paper	Paper first submitted: 2026-09-14. Qwen2.5-VL-7B initialization retains its vision encoder and projector; SFT and trajectory-level DAPO learn adaptive viewing speed, revisiting and note taking for long-video answers. Text/video input produces text and viewing actions; code and checkpoint release not established in the checked primary source as of 2026-10-19.	long-video understanding, active exploration, temporal reasoning
LFM2.5-VL-3B	Model release 2026-08-12	Announcement HF Implementation	First public: 2026-08-12. Pretrained LFM2.5-2.6B with SigLIP2 NaFlex 400M, pixel-unshuffle and a two-layer MLP projector; text/image/multi-image inputs produce text, grounding coordinates and tool calls. Weights and inference implementations available under the Liquid AI model license; aimed at low-latency edge use.	image understanding, multi-image reasoning, OCR, grounding, tool use, edge deployment
PaddleOCR-VL-1.6: Expanding the Frontier of Document Parsing with Under-Optimized Region Refinement and Progressive Post-Training	arXiv 2026-06-02	Paper Code HF	First public: 2026-05-28 (weights); paper first submitted 2026-06-02. 0.9B document VLM retaining the 1.5 native-resolution encoder, adaptive MLP connector and ERNIE language backbone; region refinement and progressive post-training improve recognition. Document-image/text inputs produce structured text, tables, formulas and spotting coordinates. Code and weights available.	document parsing, multilingual OCR, tables, formulas, text spotting
PaddleOCR-VL-1.5: Towards a Multi-Task 0.9B VLM for Robust In-the-Wild Document Parsing	arXiv 2026-01-29	Paper Code HF	First public: 2026-01-29 (paper and weights). 0.9B VLM combines a native-resolution visual encoder, adaptive MLP connector and ERNIE-4.5-0.3B; document-image/text inputs produce structured text and localized recognition. Adds seal recognition, text spotting and robustness training; full page parsing uses the accompanying layout pipeline. Code and Apache-2.0 checkpoints available.	document parsing, multilingual OCR, seal recognition, text spotting
GLM-4.6V / GLM-4.6V-Flash	Model release 2025-12-08	Announcement Code HF Flash	First public: 2025-12-08. 106B and 9B vision-language variants with 128K trained context; visual patch merging and gated MLP projection connect the vision path to the language backbone. Text/image/video inputs produce text and tool calls, with visual observations passed directly to tools. Code and MIT-licensed weights available; native tool use does not establish from-scratch multimodal training.	image/video understanding, document understanding, visual tool use
SmolVLM2	Model release 2025-02-20	Announcement HF Paper	First public: 2025-02-20. Video-trained 256M, 500M and 2.2B family extending the SigLIP/SmolLM projector design; text/image/multi-image/video inputs produce text. Uses video/image training mixtures and efficient visual compression for device deployment. Checkpoints and inference/fine-tuning support available under Apache 2.0; the shared family report was submitted 2025-04-07.	image/video understanding, multi-image reasoning, efficiency, on-device deployment
Idefics3	arXiv 2024	Paper HF	SigLIP vision encoder with pixel-shuffle spatial compression and projection to a Llama 3 backbone; replaces Idefics2’s Perceiver resampler.	visual understanding, document understanding
MiniMax-01: Scaling Foundation Models with Lightning Attention	arXiv 2025	Paper	MiniMax-VL-01 uses a 303M ViT, a two-layer MLP visual projector and MiniMax-Text-01; Lightning/Softmax hybrid attention describes the language backbone.	visual understanding
VideoJev: System One Model for Video Understanding and Interaction	arXiv 2026	Paper Code	4B / 35B-A3B; initialized from Qwen3.5-4B / Qwen3.6-35B-A3B, retaining their pretrained ViT, visual projection and language backbone; candidate-scoring heads unify temporal grounding and multiple-choice QA.	video QA, temporal grounding
Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning	ICML 2026	Paper Code	Qwen2.5-VL-7B (also Qwen3-VL-4B); inherits the pretrained Qwen ViT, visual projector and language backbone; interleaves reasoning with VideoCrop tool calls and learns from SFT plus trajectory-guided RL.	long-video reasoning, visual tool use
VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding	arXiv 2026	Paper Code	4B; I3D-ViT initialized from image-pretrained MoonViT, with a lightweight projector and Qwen3-4B text backbone; spatiotemporal visual compression supports generalist offline and proactive streaming understanding.	video understanding, temporal grounding, proactive streaming
TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs	arXiv 2026	Paper Code	2B / 4B / 8B; inherits the pretrained Qwen3-VL ViT, visual projector and language backbones; verified SFT and temporal-grounding GRPO adapt them to generalist interval localization.	video temporal grounding
OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction	arXiv 2026	Paper Code	4B; initialized from Qwen3-VL-4B-Instruct with its pretrained vision encoder frozen and the multimodal projector and language model fine-tuned; hierarchical caption memory and control states support proactive streaming interaction.	streaming video understanding, memory, proactive response
Ling-3.0-flash-VL	Model release 2026	HF	Open-weight VLM (MIT) extending Ling-3.0-flash with a ViT and a two-layer MLP projector; supports image and video inputs with text outputs. BF16 and quantized checkpoints belong to the same model entry	image/video understanding, document understanding, visual reasoning, GUI agents
LLaDA-UI	Model release 2026	Code HF	SigLIP-initialized native-resolution ViT connects to LLaDA2.0-mini-base through spatial 4-to-1 feature grouping and a two-layer MLP; block-wise diffusion generates text reasoning and GUI actions. Code and weights are available; checkpoint license is unspecified in the model card as of 2026-09-15	screenshot understanding, GUI grounding, mobile/desktop/web agents
Muse Glimmer-30B	Model release 2026	HF Implementation	Open-weight 30B model (Apache 2.0), distilled from Muse Spark; a dedicated Perception Encoder connects to the language backbone through an MLP adapter and linear projection; text and image inputs, text outputs	visual understanding, visual reasoning, coding, agentic tool use
DiffusionGemma	Google DeepMind, 2026	Model Card HF Implementation	Open-weight Gemma 4-based model (Apache 2.0); normalized visual features enter the language model through a linear projection, while an encoder-decoder architecture uses block-wise discrete diffusion for text generation. Image/video understanding with text output; no native image generation	visual understanding, video understanding, reasoning, efficient text generation
Mistral Medium 3.5	Mistral AI, 2026	Announcement HF Implementation	Open-weight 128B dense VLM under a Modified MIT license; a from-scratch vision encoder connects through a patch merger and MLP projector to the language backbone. Supports text/image inputs and text outputs; whole-model from-scratch multimodal training is not established	visual understanding, reasoning, coding, long-horizon agents
Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model	arXiv 2026	Paper Project Model	4B codec-native streaming VLM combining a from-scratch Mage-ViT visual encoder, two-layer MLP projector, and pretrained Qwen3-4B-Instruct-2507 decoder; codec-guided token selection reduces visual tokens by over 75%	image/video understanding, long video, proactive streaming
StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design	arXiv 2026	Paper	0.9B on-device UI VLM with UI-aware layered visual encoding and a progressive dimensionality projection connector; quantized deployment is validated on Snapdragon 8 Gen5	UI understanding, OCR, grounding, on-device deployment
MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe	arXiv 2025	Paper Code Model	Efficient 8B MLLM built with Qwen3-8B, SigLIP2-400M and a unified 3D-Resampler for compact image-video encoding	visual understanding, document/OCR, video understanding, edge deployment
Moondream 3.1	Model Release 2026	Project Model	Efficient sparse MoE VLM with 9B total and 2B active parameters; supports visual reasoning, query, caption, detection, pointing and segmentation	visual understanding, grounding, segmentation, edge deployment
LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training	arXiv 2025	Paper Code HF	4B / 8B; 85M mid-training data + 22M instruction data; ViT–MLP–LLM architecture with pretrained RICE-ViT, a two-layer MLP projector, and Qwen3 backbone; supports image, multi-image, and video	visual understanding, video
SAIL-VL2 Technical Report	arXiv 2025	Paper	Open-suite 2B/8B vision-language foundation model with SAIL-ViT, progressive multimodal training, SFT-RL thinking fusion, and strong image/video reasoning across 106 datasets	visual understanding, video, reasoning
Kwai Keye-VL 2.0 Technical Report	arXiv 2026	Paper	achieves state-of-the-art performance among models of similar scale, particularly excelling in fine-grained temporal localization	video understanding
Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders	arXiv 2026	Paper	LLM-initialized vision encoder (non-CLIP); text-to-vision weight reuse, generative-aligned visual features, optimized for dense perception.	visual understanding
Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision	arXiv 2026	Paper	SigLIP2 vision encoder with 2×2 Spatial Merge and a two-layer MLP projector to Youtu-LLM; unified autoregressive supervision over visual and text targets supports fine-grained understanding and dense prediction.	visual understanding, grounding, segmentation, depth estimation
STEP3-VL-10B Technical Report	arXiv 2026	Paper	10B-scale foundation multimodal; unified unfrozen pre-training + PaCoRe test-time scaling, frontier-level reasoning with compact footprint.	visual understanding
GLM-OCR	arXiv 2026	Paper	GLM-OCR is an efficient 0.9B-parameter compact multimodal model designed for real-world document understanding.	OCR, structured extraction
Kimi K2.5	arXiv 2026	Paper	joint text-vision pretraining, Agent Swarm framework; coding, vision, reasoning, agentic tasks; reduces latency by up to 4.5x	visual agentic intelligence, agentic, reasoning
Kwai Keye-VL 1.5 Technical Report	arXiv 2025	Paper	Adaptive Slow-Fast encoding; 8B parameter scale with 128K long-context; SOTA video reasoning & human-preference aligned.	visual understanding
olmOCR / olmOCR-2	arXiv 2025	Paper	Efficient low-VRAM OCR model based on Qwen2.5-VL fine-tune; excels at preserving semantic structure and markdown output	OCR, structured extraction
PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model	arXiv 2025-10-16	Paper Code HF	Paper first submitted: 2025-10-16. Original 0.9B document VLM with a NaViT-style dynamic-resolution encoder, adaptive MLP connector and ERNIE-4.5-0.3B; document-image/text inputs produce multilingual text, table, formula and chart representations. Code and weights available; later 1.5 and 1.6 releases are listed separately.	OCR, multilingual document parsing
DeepSeek-OCR	arXiv 2025	Paper HF	Lightweight ~3B MoE vision model optimized for high-volume OCR, document digitization, charts and formulas; efficient inference	OCR, document
Kimi-VL	arXiv 2025	Paper HF	Projector + MoE backbone; long video/PDF/GUI, agentic capabilities, chain-of-thought vision reasoning	visual understanding, agentic, video
Seed1.5-VL Technical Report	arXiv 2025	Paper	20B MoE + 532M ViT; native-resolution vision-language foundation model; efficient asymmetric architecture.	visual understanding
Qwen3-VL	arXiv 2025	Paper HF	Frontier-grade vision/OCR (32+ languages), video analysis, agentic capabilities, strong multimodal reasoning; includes large MoE variants (e.g., 235B)	visual understanding, video, omni
SmolVLM: Redefining small and efficient multimodal models	arXiv 2025-04-07	Paper Announcement HF 256M	First public: 2024-11-26 (original release); family paper submitted 2025-04-07. Compact image-first SigLIP/SmolLM family using pixel-shuffle compression and projection; text/image/multi-image inputs produce text, with sampled-frame video explored in the original release. Code and Apache-2.0 checkpoints available; SmolVLM2 separately records dedicated video training and release.	visual understanding, efficiency, on-device deployment
LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning	arXiv 2025	Paper	Diffusion LLM as LLM backbone; vision encoder: SigLIP	visual understanding
jina-vlm	arXiv 2025	Paper HF	SigLIP2 + Qwen backbone with custom projector; optimized for semantic VQA, diagrams, scans and document semantics	visual understanding, VQA, document
Phi-4-Multimodal	arXiv 2025	Paper HF	Small-parameter (LoRA + projectors) multimodal; vision + speech support, efficient on-device deployment	visual understanding, on-device
Molmo / PixMo	CVPR 2025	Paper Code	Strong open-data/open-weight VLM pipeline	visual understanding
FastVLM: Efficient Vision Encoding for Vision Language Models	CVPR 2025	Paper	efficient multimodal visual encoding for on-device deployment	visual understanding, on-device
Qwen2.5-VL: Technical Report	arXiv 2025	Paper HF	Stronger document, grounding, and video capabilities	visual understanding
General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model	arXiv 2024	Paper HF/Code	Specialized end-to-end OCR model with grounding (boxes + points); strong on scientific papers, slides, and mixed visual-text docs	OCR, grounding
LLaVA-OneVision: Easy Visual Task Transfer	arXiv 2024	Paper Code	Single model for image, multi-image, and video transfer	visual understanding
MiniCPM-V: A GPT-4V Level MLLM on Your Phone	arXiv 2024	Paper Code	On-device efficient MLLM	visual understanding
NVILA: Efficient Frontier Visual Language Models	CVPR 2025	Paper	Efficient general purpose multimodal LLM; spatial and temporal "Scale then compress" design; vision encoder: SigLIP	visual understanding
GLM-4V	arXiv 2024	Paper	ViT-based vision encoder (EVA-02-CLIP-L); high-resolution input support (up to 1120x1120) via image tiling and a visual interface to the language model, optimized for document and OCR tasks	visual understanding
xGen-MM (BLIP-3)	arXiv 2024	Paper	Open training recipe, datasets, and safety-tuned variants	visual understanding
DeepSeek-VL2: Mixture-of-Experts Vision-Language Models	arXiv 2024	Paper Code	MoE VLM with dynamic tiling and efficient inference	visual understanding
Pixtral	arXiv 2024	Paper HF	12B open-weight model with strong instruction following, image+text understanding; competitive with larger open VLMs	visual understanding
Qwen2-VL	arXiv 2024	Paper HF	Dynamic resolution; native video	visual understanding
Cambrian-1: A Fully Open, Vision-Centric Exploration	NeurIPS 2024	Paper Code	Spatial Vision Aggregator	visual understanding
PaliGemma: A Versatile 3B VLM for Transfer	arXiv 2024	Paper HF	SigLIP encoder + Gemma backbone; strong transfer model	visual understanding
InternLM-XComposer2	arXiv 2024	Paper Code	Compositional visual grounding	visual understanding
Phi-3-Vision	arXiv 2024	Paper HF	Small but capable	visual understanding
LLaVA-HR: High Resolution MLLMs	CVPR 2024	Paper	Mixture-of-Resolution Adaptation	visual understanding
InternVL2	Model release 2024	HF	Instruction-tuned InternVL family release with strong multilingual and OCR capabilities	visual understanding
InternVL: Scaling up Vision Foundation Models	CVPR 2024	Paper Code	Progressively aligned ViT + LLM	visual understanding
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training	arXiv 2024	Paper	Large-scale proprietary recipe study for multimodal LLM pretraining	visual understanding
Ovis: Structural Embedding Alignment for Multimodal Large Language Model	arXiv 2024	Paper Code	Structural embedding alignment between visual tokens and LLM token space	visual understanding
TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document	arXiv 2024	Paper Code	OCR-free document MLLM emphasizing text-heavy images	document understanding, OCR
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models	arXiv 2024	Paper Code	Sparse MoE extension of LLaVA-style visual instruction tuning	visual understanding
MobileVLM: A Fast, Strong and Open Vision Language Assistant for Mobile Devices	arXiv 2023	Paper Code	Lightweight VLM for mobile deployment	efficient visual assistant
Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models	arXiv 2023	Paper Code	Expands visual vocabulary for dense OCR/document-style perception	document understanding, OCR
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models	arXiv 2023	Paper Code	Compresses each frame/image into compact context tokens for efficient video MLLMs	video understanding
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection	arXiv 2023	Paper Code	Unified image-video representation before projection into LLM	image/video understanding
GLaMM: Pixel Grounding Large Multimodal Model	arXiv 2023	Paper Code	Pixel-level grounding and phrase-region reasoning	grounded visual understanding
Ferret: Refer and Ground Anything Anywhere at Any Granularity	arXiv 2023	Paper Code	Referring and grounding across points, boxes, and free-form regions	grounded visual dialogue
LLaVA-1.5: Improved Baselines with Visual Instruction Tuning	arXiv 2023	Paper Code	Strong simple baseline with CLIP visual encoder, MLP projector, and instruction tuning	visual instruction tuning
ImageBind-LLM: Multi-modality Instruction Tuning	arXiv 2023	Paper Code	Connects ImageBind-aligned modalities to an LLM for multi-modality instruction following	multi-modal instruction tuning
PointLLM: Empowering Large Language Models to Understand Point Clouds	arXiv 2023	Paper Code	Extends LLM-based multimodal understanding to 3D point clouds	3D understanding
LISA: Reasoning Segmentation via Large Language Model	arXiv 2023	Paper Code	Couples MLLM reasoning with segmentation mask output	reasoning segmentation
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest	arXiv 2023	Paper Code	Region-of-interest instruction tuning for fine-grained visual reasoning	region-level understanding
3D-LLM: Injecting the 3D World into Large Language Models	arXiv 2023	Paper Code	Projects 3D scene features into LLMs for 3D reasoning and dialogue	3D understanding
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic	arXiv 2023	Paper Code	Referential dialogue with natural-language coordinates and grounding	grounded visual dialogue
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day	arXiv 2023	Paper Code	Biomedical adaptation of LLaVA-style visual instruction tuning	biomedical visual assistant
DetGPT: Detect What You Need via Reasoning	arXiv 2023	Paper Code	Uses LLM reasoning to orchestrate detection tools and visual grounding	detection reasoning
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks	arXiv 2023	Paper Code	Treats vision-centric tasks as open-ended decoding with LLMs	vision-centric decoding
MultiModal-GPT: A Vision and Language Model for Dialogue with Humans	arXiv 2023	Paper Code	Instruction-tuned image-text dialogue model	visual dialogue
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model	arXiv 2023	Paper Code	Parameter-efficient visual instruction tuning via adapters	visual instruction tuning
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention	arXiv 2023	Paper Code	Early adapter-based multimodal instruction tuning on LLaMA	visual instruction tuning
LLaVA	arXiv 2023	Paper Code	7B / 13B+ CLIP vision encoder (frozen/pretrained) + linear projection to LLM (Vicuna/LLaMA); common modular pretrained-backbone baseline	visual instruction tuning, VQA, image captioning

Q-Former

Paper	Venue	Links	Notes	Task
M-MiniGPT4: Multilingual VLLM Alignment via Translated Data	arXiv 2026	Paper	Q-Former based (inherits from MiniGPT-4 / BLIP-2)	vision-language understanding
Video Q-Former: Multimodal Large Language Model with Spatio-Temporal Querying Transformer	OpenReview	Paper	Spatio-temporal Q-Former (learnable queries for video spatial-temporal feature extraction)	video understanding
HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding	arXiv 2025	Paper	Hierarchical Q-Former (multi-level learnable queries with memory bank for long video)	long video understanding
Towards Efficient Visual-Language Alignment of the Q-Former	arXiv 2024	Paper	PEFT-tuned Q-Former (parameter-efficient fine-tuning on InstructBLIP-style Q-Former)	visual reasoning
Matryoshka Query Transformer (MQT) for Large Vision-Language Models	NeurIPS 2024	Paper	Matryoshka Query Transformer (elastic learnable queries, variable token count)	vision-language understanding
Semantically Grounded QFormer for Efficient Vision Language Understanding	arXiv 2023	Paper	Improved Grounded QFormer (direct latent conditioning, bypass input projection)	vision-language understanding
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding	arXiv 2023	Paper Code	BLIP-2/Q-Former-style visual and audio query transformers for video dialogue	audio-video understanding
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning	arXiv 2023	Paper Code	Instruction-aware Q-Former trained over diverse vision-language tasks	visual instruction tuning
VideoChat: Chat-Centric Video Understanding	arXiv 2023	Paper Code	Video-centric MLLM for temporal dialogue and understanding	video understanding
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality	arXiv 2023	Paper Code	Modular visual abstractor connected to a pretrained LLM	visual instruction tuning
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models	arXiv 2023	Paper Code	Uses BLIP-2 visual encoder/Q-Former and aligns visual features to Vicuna	visual dialogue
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models	ICML 2023	Paper Code	Foundational Q-Former architecture bridging frozen vision encoders and frozen LLMs	vision-language pretraining

Cross-Attention

Paper	Venue	Links	Notes	Task
CASA: Cross-Attention over Self-Attention	arXiv 2025	Paper	Efficient cross-attention via self-attention reformulation; competitive with token insertion on image benchmarks, strong for long video	efficient vision-language fusion, video captioning
LLaMA 3.2 Vision	arXiv 2024	Paper HF	Adapter-based vision addition to Llama 3.2; strong OCR, document VQA, 128K context	visual understanding, document
Qwen-VL: A Versatile Vision-Language Model	arXiv 2023	Paper HF	High-res, multi-lang, bounding box	visual understanding
Kosmos-2: Grounding Multimodal Large Language Models to the World	arXiv 2023	Paper Code	Adds grounded phrase-region modeling to multimodal language modeling	grounded visual understanding
Kosmos-1: Language Is Not All You Need: Aligning Perception with Language Models	arXiv 2023	Paper Code	Perception-language model aligning images and language for multimodal reasoning	multimodal reasoning
Flamingo: a Visual Language Model for Few-Shot Learning	NeurIPS 2022	Paper	Perceiver Resampler + gated cross-attention layers for few-shot multimodal prompting	few-shot visual understanding
IDEFICS	—	Hugging Face	80B Flamingo-inspired model; Perceiver resampling and gated cross-attention connect visual features to the language model	open-source multimodal understanding

Hybrid Adaptor

Paper	Venue	Links	Notes	Task
MiniCPM-V 4.7	Model release 2026-10-12	Code 35B-A3B 1B Architecture	First public: 2026-10-12. 35B-A3B MoE and 1B VLMs combining SigLIP, a window-attention merger, MLP merger and Qwen3.5 language backbone; canvas M-RoPE and mixed 4x/16x compression support image/multi-image/video inputs with text outputs. Code and Apache-2.0 weights available; Lookahead Expert Scheduling supports on-device MoE inference; DSpark heads released 2026-10-15.	visual understanding, document/OCR, video understanding, on-device deployment
MolmoPoint: Better Pointing for VLMs with Grounding Tokens	arXiv 2026-03-30	Paper Code HF	Paper first submitted: 2026-03-30. Extends Molmo2 with grounding tokens that select image patches/subpatch locations instead of spelling coordinates; retains the attention-pooling plus MLP visual input connector. Image/video/text inputs produce text, points and tracks; includes GUI/video specializations. Code, checkpoints and grounding data available as of 2026-10-19. Output pointing attention is distinct from the visual input bridge.	visual grounding, pointing, tracking, GUI grounding, video understanding
Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding	arXiv 2026-01-15	Paper Code HF	Paper first submitted: 2026-01-15. 4B/8B and OLMo-based 7B family with pretrained SigLIP2 and language backbones, spatial multi-head attention pooling and a shared MLP connector. Text/image/multi-image/video inputs produce text, pointing coordinates and object tracks. Open training code, checkpoints and video/multi-image data available; packing and message-tree training support multiple QA contexts.	image/video understanding, multi-image reasoning, pointing, tracking
mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding	arXiv 2024-09-05	Paper Code HF	Paper first submitted: 2024-09-05. Adds a global-feature-guided DocCompressor after the convolution/projection H-Reducer, retaining Modality Adaptive Modules in the language backbone; compresses a high-resolution page to 324 visual tokens. Document-image/multi-page/video-frame and text inputs produce textual answers. Code and checkpoints available; three-stage training covers single-page and multi-page documents.	document understanding, multi-page document QA, document video
CogVLM: Visual Expert for Pretrained Language Models	arXiv 2023	Paper Code	Pretrained ViT + two-layer MLP adapter + pretrained Vicuna; visual tokens use separate QKV and FFN experts within shared causal self-attention for deep fusion.	visual understanding
MiniCPM-V 4.6	Model release 2026	HF Architecture	Open-weight VLM (Apache 2.0) combining a SigLIP vision encoder, window-attention merger, MLP merger, and Qwen3.5 language backbone; supports 4x and 16x visual downsampling for efficient image/video understanding	visual understanding, document/OCR, video understanding, on-device deployment
EXAONE 4.5 Technical Report	arXiv 2026	Paper Code	Integrates a dedicated visual encoder with the EXAONE 4.0 framework for multimodal pretraining, with strong document understanding and Korean contextual reasoning	visual understanding, document
Phoenix-VL 1.5 Medium Technical Report	arXiv 2026	Paper	123B multilingual multimodal model continued-pretrained from Mistral Medium 3.1 on localized multimodal and long-context corpora	visual understanding, multilingual
DeepSeek-OCR-2	arXiv 2026	Paper HF	Optimized for high-volume OCR, document digitization, charts and formulas; efficient inference	OCR, document
Ovis2.5	arXiv 2025	Paper	Following VET architecture; excellent document understanding and fine-grained quantization	visual understanding, document
Ovis2	arXiv 2025	HF	Embedding table / projector architecture; excellent document understanding and fine-grained quantization	visual understanding, document
mPLUG-Owl3	arXiv 2024	Paper Code	Long visual sequences	visual understanding
NVLM 1.0: Open Frontier-Class Multimodal LLMs	arXiv 2024	Paper HF	Hybrid multimodal design with strong OCR and reasoning	visual understanding
Idefics2	arXiv 2024	Paper HF	SigLIP vision encoder → learned Perceiver pooling → MLP projection; the resulting visual sequence is concatenated with text embeddings in a pretrained Mistral backbone.	visual understanding, document understanding
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding	arXiv 2024-03-19	Paper Code	Paper first submitted: 2024-03-19. OCR-free document understanding with convolution/projection H-Reducer and Modality Adaptive Modules separating visual/text key-value projections inside the pretrained language backbone. Image/text inputs produce textual answers and structured document parsing; unified structure learning uses DocStruct4M and DocReason25K. Code, models and data available; DocOwl2 is a separate entry.	document understanding, OCR, layout understanding

Interface Unspecified

These understanding-model references have confirmed multimodal capabilities, but inspected primary sources do not establish the visual-to-language interface well enough to assign an adapter category. Placement is provisional: it neither establishes pretrained-language-model initialization nor excludes NMM membership. Initialization, fusion depth and training nativity remain independent evidence questions.

Paper	Venue	Links	Notes	Task
Muse Spark Open	Model release 2026-10-15	Announcement HF	Open-weight 816B-total / 42B-active MoE under the custom Muse Model License, with a 1.8B Perception Encoder. Text/image/audio/video input, text output. Initialization, multimodal training-entry stage, exact connector, and fusion depth are undisclosed; native branding alone does not establish them. Released separately from proprietary Muse Spark 1.3.	image/video/audio understanding, multimodal reasoning, coding, agents
Nemotron 3.5 Super VL	Model release 2026-10-15	Announcement Report HF Code	First public: 2026-10-14 (announcement); weights released 2026-10-15. 120B total/12B active, pretrained Nemotron language backbone with a C-RADIO vision encoder and VL connector; precise connector design not established here. Text/image/video inputs produce text and tool calls; BF16/NVFP4 checkpoints and recipes available. Mamba/Transformer hybrid describes the language backbone; Lightning is text-only.	image/video understanding, document understanding, visual tool use, agents
Infinity-Parser2 Technical Report	arXiv 2026-07-08	Paper Code Pro Flash	First public: 2026-05-11 (models); paper first submitted 2026-07-08. Flash and Pro fine-tune Qwen3.5-2B and Qwen3.5-35B-A3B with 5M bilingual document examples and joint RL across eight tasks; the visual connector is not independently verified here. Document-image/text inputs produce Markdown, HTML, LaTeX and SMILES. Code, checkpoints and Infinity-Doc2-5M available.	document parsing, OCR, layout grounding, table/formula/chart recognition

3.2 Omni MLLMs

Paper	Venue	Links	Notes	Task	Adaptor
TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming	arXiv 2026	Paper Code	4B/9B; pretrained Qwen3.5 language and vision backbone with a grafted AuT audio encoder and lightweight MLP aligner; Per-vGrid aligns audio-video tokens; up to 256K context; text output	image/video/audio understanding, live-commerce QA	MLP Projector
MiMo-V2.5	Model release 2026	HF	Open-weight MoE model (MIT) with a dedicated MiMo ViT and a MiMo-Audio-initialized audio encoder; text pretraining is followed by visual/audio MLP projector warmup and multimodal pretraining. Supports image, video, and audio understanding with text outputs	image/video/audio understanding, multimodal reasoning, long-context agents	MLP Projector
MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction	arXiv 2026	Paper Code Model	9B edge-oriented omni model using Omni-Flow for simultaneous visual/audio perception and speech response; supports proactive full-duplex interaction with less than 12GB memory	vision-language understanding, audio understanding, speech generation, full-duplex live interaction	Hybrid
Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence	arXiv 2026	Paper Code	Efficiency-optimized omni-modal backbone using Hybrid Mamba2-Transformer MoE; supports massive multi-modal contexts (10k+ tokens) for long-video reasoning and agentic GUI navigation on edge devices	omni-modal understanding & reasoning	Hybrid
OmniVideo-R1: Reinforcing Audio-visual Reasoning with Query Intention and Modality Attention	arXiv 2026	Paper	Reinforced audio-visual reasoning framework with query intention grounding and modality attention fusion	audio-visual reasoning	Hybrid
Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data	arXiv 2025	Paper Code	MoE-based scaling for omnimodal understanding and generation	omni-modal understanding & generation	MLP Projector
Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models	arXiv 2025	Paper Code	Unified audio-visual speech recognition using LLMs	audio-visual speech recognition	Hybrid
LongCat-Flash-Omni Technical Report	arXiv 2025	Paper Code	Long-context omni-modal model supporting text and audio generation	long-context omni-modal	Hybrid
OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM	arXiv 2025	Paper Code	Architecture and data enhancements for omni-modal understanding	omni-modal understanding	Hybrid
InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialogue	arXiv 2025	Paper Code	Unified model for audio-visual multi-turn dialogue	audio-visual dialogue	Hybrid
OneLLM: One Framework to Align All Modalities with Language	CVPR 2024	Paper	Unified framework aligning eight modalities to language through modality tokenizers and lightweight projectors	all-in-one LLM	Hybrid
MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition	NeurIPS 2025	Paper	Mixture of Matryoshka experts for efficient audio-visual speech recognition	audio-visual speech recognition	Hybrid
Qwen3-Omni Technical Report	arXiv 2025	Paper Code	Omni-modal model with text and audio capabilities (Alibaba/Qwen series)	omni-modal	Native
Qwen2.5-Omni Technical Report	arXiv 2025	Paper Code	Omni-modal technical report with text and audio support (Alibaba/Qwen series)	omni-modal	Hybrid
MiniCPM-o 2.6: A GPT-4o Level MLLM for Vision, Speech, and Multimodal Live Streaming on Your Phone	2025	Paper Code	On-device GPT-4o level MLLM for vision, speech and multimodal live streaming (OpenBMB)	on-device multimodal live streaming	Hybrid
Baichuan-Omni Technical Report	arXiv 2024	Paper Code	Technical report for Baichuan-Omni (Baichuan Inc.)	omni-modal	Hybrid
Baichuan-Omni-1.5 Technical Report	arXiv 2025	Paper Code	Technical report for Baichuan-Omni 1.5 (Baichuan Inc.)	omni-modal	Hybrid
VITA: Towards Open-Source Interactive Omni Multimodal LLM	arXiv 2024	Paper Code	Open-source interactive omni multimodal LLM	interactive omni multimodal	Hybrid
VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction	NeurIPS 2025	Paper Code	Real-time vision and speech interaction model	real-time multimodal interaction	Hybrid
Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities	NeurIPS 2024	Paper Code	Open-source GPT-4o style model with vision, speech and duplex capabilities	vision-speech duplex	Hybrid
Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive Modality Alignment	arXiv 2025	Paper Code	Progressive modality alignment for omni-modal language model	omni-modal alignment	MLP Projector
MIO: A Foundation Model on Multimodal Tokens	arXiv 2024	Paper Code	Foundation model based on multimodal tokens	multimodal tokens	Native
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions	CVPR 2025	Paper Code	Multimodal model supporting seeing, hearing and emotional speech	emotional multimodal	Hybrid
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model	arXiv 2025	Paper Code	Simultaneous multimodal interactions with language-vision-speech model	simultaneous multimodal	Hybrid
ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding	arXiv 2025	Paper Code	Native multimodal LLM focused on 3D generation and understanding	3D multimodal	Native
Pengi: An Audio Language Model for Audio Tasks	arXiv 2023	Paper Code	Audio-language model using audio representations with a frozen language model for audio captioning, QA, and retrieval-style tasks	audio-language understanding	Hybrid
LTU: Listen, Think, and Understand	arXiv 2023	Paper Code	Audio-oriented multimodal instruction-following model for open-ended audio understanding	audio-language understanding	Hybrid

Back to Top

4. Unified Multimodal Models (UMMs)

In this section: 4.1 Taxonomy by Generation Paradigm · 4.2 Any-to-Any / Omni UMMs · 4.3 Related Generators (U+G Unverified)

Models that unify multimodal understanding and visual generation within one framework. The defining property is U+G unification, not necessarily training from scratch.

Boundary with NMMs: if a unified model's central contribution is native end-to-end multimodal pretraining from scratch, we document its architectural details primarily in §5 NMMs and keep §4 focused on the unified U+G perspective.

Overview of representative paradigms and architectures of Unified Multimodal Models (UMMs). Source: https://github.com/AIDC-AI/Awesome-Unified-Multimodal-Models

4.1 Taxonomy by Generation Paradigm

Subtopics: Diffusion-Based UMMs · Autoregressive (AR) UMMs · Hybrid (AR + Diffusion) UMMs

This taxonomy first identifies how visual content is sampled, then records whether the described system also provides an autoregressive understanding/text-output path:

Autoregressive (AR): visual content tokens or embeddings are predicted autoregressively (including masked or next-scale variants). A separately trained image reconstruction decoder may use diffusion; the visual-content prediction stage remains AR.
Diffusion-Based: visual content is sampled through diffusion, masked denoising, or flow matching, and the described system has no separately established AR understanding/text-output path. An MLLM used only to encode conditions, or a prompt rewriter used only to prepare generation inputs, does not establish that output path.
Hybrid (AR + Diffusion): the described system provides both an AR language-output path for understanding/reasoning and diffusion/flow-based visual sampling. The paths may share a backbone or use a query, hidden-state, or prompt connector. The entry must establish both paths; the presence of an AR-capable condition encoder alone is insufficient.

For example, Qwen-Image uses its Qwen-VL component as a condition encoder for MMDiT image sampling and is placed in Diffusion-Based. Qwen-Image-2.0 uses the same generation paradigm, but its U+G membership is unverified; it is retained separately as a related generator (§4.3). UniWorld (Fig. 3 and Table 1) explicitly retains an AR VLM understanding branch, evaluates image-to-text understanding, and pairs it with a flow-matching image generator, so it is placed in Hybrid. Merely inheriting a pretrained VLM does not prove that the resulting system exposes its standalone outputs.

Pixel, semantic, learnable-query, and hybrid encoding describe a separate representation dimension. Subsections retain these distinctions within a generation paradigm; entries using diffusion reconstruction decoders state that component explicitly.

Diffusion-Based UMMs

Model	Venue	Links	Paradigm	Notes	Task
Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation	arXiv 2026-10-14	Paper Code Model	Discrete masked diffusion; codebook-depth AR head	Paper first submitted: 2026-10-14; official inference-code announcement is dated 2026-08 (day unspecified). Pretrained LLaDA2.0-mini masked-diffusion MoE with pretrained AToken eight-codebook visual tokens. Spatial positions are denoised in parallel; a local AR head predicts only the codebook depth within each position. This is not an AR visual-content sequence or whole-model scratch training. As of 2026-10-19: inference/SFT code and NVIDIA/Ascend checkpoints released.	image/video/3D understanding, text generation, image generation and editing
Multimodal Flow: Unified Flow Modeling of Language and Vision in Embedding Spaces (MF-1)	arXiv 2026-09-30	Paper Code Model	Continuous flow matching for text and vision	First public: 2026-09-30. Models continuous text/image embeddings with a shared flow field, chunk-causal attention and modality-specific FFNs. Sequential chunk continuation does not establish a separate AR next-token text-output objective. Reuses modality representations/decoders; whole-model scratch initialization is not established. As of 2026-10-19: code and 1.6B pretraining/SFT checkpoints released.	visual understanding, text/image generation, mixed-sequence continuation
Qwen-Image Technical Report	arXiv 2025	Paper Code	MMDiT diffusion	Frozen Qwen2.5-VL conditions an MMDiT image generator using semantic and VAE representations; supports text rendering, editing and image-form dense perception tasks.	image generation, editing, dense visual perception
LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model	arXiv 2026-04-22	Paper Code Model FP8	Unified discrete masked diffusion	First public: 2026-04-22. LLaDA2.0-based MoE predicts masked text/visual tokens; SigLIP-VQ produces discrete semantic tokens and a distilled diffusion reconstruction decoder produces images. As of 2026-10-19: inference code/checkpoints released 2026-04-23; FP8 released 2026-05-06, Diffusers/ComfyUI support 2026-05-12, SGLang-Omni support 2026-05-29.	visual understanding, text generation, image generation and editing, interleaved reasoning
Dual Diffusion	arXiv 2025	Paper Code	Dual Diffusion	Unified image generation + understanding via bidirectional diffusion	visual understanding, visual generation
UniDisc	arXiv 2025	Paper Code	Unified Discrete Diffusion	Discrete diffusion for multimodal U+G	visual understanding, visual generation
MMaDA	arXiv 2025	Paper Code	Multimodal Large Diffusion LM	Diffusion LM for unified understanding/generation	visual understanding, visual generation
FUDOKI	arXiv 2025	Paper	Discrete Flow-based Unified	Kinetic-optimal velocities for U+G	visual understanding, visual generation
Muddit	arXiv 2025	Paper Code	Unified Discrete Diffusion	Liberating generation beyond T2I	visual understanding, visual generation
Lavida-O	arXiv 2025	Paper Code	Elastic Large Masked Diffusion	Elastic masked diffusion for U+G	visual understanding, visual generation
UniModel	arXiv 2025	Paper	Visual-Only MMDiT Framework	Visual-only unified multimodal U+G	visual understanding, visual generation

Autoregressive (AR) UMMs

Pixel Encoding

Model	Venue	Links	Modalities	Notes	Task
LWM	arXiv 2024	Paper	video + language	World model on million-length video and language with blockwise ring attention	visual understanding, visual generation
Chameleon	arXiv 2024	Paper Code	image + text	Mixed-modal early-fusion foundation models; token-by-token generation	visual understanding, visual generation
ANOLE	arXiv 2024	Paper Code	image + text	Open autoregressive native LMM for interleaved image-text generation	visual understanding, visual generation
MMAR	arXiv 2024	Paper	image + text	Lossless multi-modal auto-regressive probabilistic modeling	visual understanding, visual generation
Orthus	arXiv 2024	Paper Code	image + text	Autoregressive interleaved image-text generation with modality-specific heads	visual understanding, visual generation
SynerGen-VL	arXiv 2024	Paper	image + text	Synergistic image understanding and generation with vision experts and token folding	visual understanding, visual generation
Liquid	arXiv 2024	Paper Code	image + text	Language models are scalable and unified multi-modal generators	visual understanding, visual generation
UGen	arXiv 2025	Paper	image + text	Unified autoregressive multimodal model with progressive vocabulary learning	visual understanding, visual generation
Harmon	arXiv 2025	Paper Code	image + text	Shared MAR encoder for semantic + fine-grained harmony; SOTA GenEval	visual understanding, visual generation
TokLIP	arXiv 2025	Paper Code	image + text	Marry visual tokens to CLIP for U+G	visual understanding, visual generation
Selftok	arXiv 2025	Paper Code	image + text	Discrete visual tokens for AR / Diffusion / Reasoning	visual understanding, visual generation
OneCat	arXiv 2025	Paper Code	image + text	Pure decoder-only unified U+G	visual understanding, visual generation
Uni-X	arXiv 2025	Paper Code	image + text	Two-end-separated architecture mitigating modality conflict	visual understanding, visual generation
Emu3	arXiv 2024	Paper Code	image + video + text	Early-fusion native autoregressive model; see NMM / Early Fusion for architectural classification	visual understanding, visual generation

Semantic Encoding

Title	Venue	Links	Focus	Task
Ming-UniVision: Joint Image Understanding and Generation with a Unified Continuous Tokenizer	arXiv 2025	Paper Code	Unified continuous tokenizer for joint understanding and generation	visual understanding, visual generation
Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents	arXiv 2025	Paper Code	Masked autoregression predicts patch-level CLIP image embeddings; a latent-ControlNet-conditioned diffusion decoder reconstructs the final image.	visual understanding, visual generation
X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again	arXiv 2025	Paper Code	RL-enhanced discrete autoregressive unified modeling	visual understanding, visual generation
UniCode²: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation	arXiv 2025	Paper	Cascaded large-scale codebooks for unified modeling	visual understanding, visual generation
Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations	arXiv 2025	Paper Code	Text-aligned discrete semantic representations	visual understanding, visual generation
UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation	arXiv 2025	Paper Code	Y-shaped architecture for modality alignment	visual understanding, visual generation
Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation	arXiv 2025	Paper	Auto-regressive foundation model	visual understanding, visual generation
DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies	arXiv 2025	Paper	Dual visual vocabularies	visual understanding, visual generation
UniTok: A Unified Tokenizer for Visual Generation and Understanding	arXiv 2025	Paper Code	Unified tokenizer	visual understanding, visual generation
QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive Multimodal Understanding and Generation	arXiv 2025	Paper Code	Text-aligned visual tokenization	visual understanding, visual generation
MetaMorph: Multimodal Understanding and Generation via Instruction Tuning	arXiv 2024	Paper	Instruction tuning for unified multimodal	visual understanding, visual generation
ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance	arXiv 2024	Paper	Self-enhancing unified see-and-draw	visual understanding, visual generation
PUMA: Empowering Unified MLLM with Multi-granular Visual Generation	arXiv 2024	Paper Code	Multi-granular visual generation	visual understanding, visual generation
VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation	ICLR 2025	Paper Code	Unified foundation model	visual understanding, visual generation
VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation	arXiv 2023	Paper	Generative pre-trained transformer	visual understanding, visual generation
Generative Multimodal Models are In-Context Learners (Emu2)	CVPR 2024	Paper	In-context learning generative multimodal model	visual understanding, visual generation
LaVIT: Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization	ICLR 2024	Paper Code	Dynamic discrete visual tokenization	visual understanding, visual generation
Emu: Generative Pretraining in Multimodality	ICLR 2024	Paper	Generative pretraining in multimodality	visual understanding, visual generation

Learnable Query Encoding

Title	Venue	Links	Focus	Task
Nexus-Gen: A Unified Model for Image Understanding, Generation, and Editing	arXiv 2025	Paper Code	Prefilled autoregression predicts visual embeddings in a shared space; a FLUX diffusion decoder converts them into images for generation and editing.	visual understanding, visual generation, editing
Making LLaMA SEE and Draw with SEED Tokenizer	ICLR 2024	Paper Code	SEED tokenizer enabling LLaMA for scalable multimodal autoregression (see and draw)	visual understanding, visual generation
Planting a SEED of Vision in Large Language Model	arXiv 2023	Paper Code	SEED image tokenizer with 1D causal dependency and high-level semantics for LLM vision	visual understanding, visual generation

Hybrid Encoding (Pseudo)

Title	Venue	Links	Focus	Task
Skywork UniPic: Unified Autoregressive Modeling for Visual Understanding and Generation	arXiv 2025	Paper Code	Unified autoregressive modeling with decoupled encoding for image understanding, generation and editing	visual understanding, visual generation
UniFluid: Unified Autoregressive Visual Generation and Understanding with Continuous Tokens	arXiv 2025	Paper	Unified autoregressive framework using continuous visual tokens	visual understanding, visual generation
OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models	arXiv 2025	Paper Code	Efficient linear-time unified multimodal model based on Mamba (state space models)	multimodal understanding, visual generation
Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling	arXiv 2025	Paper Code	Scaled-up version of Janus with improved training strategy, more data and larger model size	multimodal understanding, visual generation
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation	arXiv 2024	Paper Code	Decoupling visual encoding to enable unified understanding and generation in an autoregressive framework	multimodal understanding, visual generation

Hybrid Encoding (Joint)

Title	Venue	Links	Focus	Task
Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation	arXiv 2026-07-28	Paper	First public: 2026-07-28. A frozen pretrained VLM vision encoder supplies continuous understanding features and quantized generation codes; an LLM predicts visual codes autoregressively and a reconstruction decoder produces images. Reuses pretrained VLM components; studies compact models including a 0.5B language backbone. As of 2026-10-19: paper public; official code/weight release not confirmed by inspected primary sources.	image understanding, image generation
UniAR: Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification	ICML 2026	Paper	Shared context-visual tokenizer bridging understanding and generation; AR visual-token prediction with diffusion-based decoding for high-fidelity generation and editing	visual understanding, visual generation, editing
AToken: A Unified Tokenizer for Vision	arXiv 2025	Paper Code	AToken unified visual tokenizer achieving high-fidelity reconstruction and semantic understanding for images, videos and 3D	visual understanding, visual generation
UniWeTok: An Unified Binary Tokenizer with Codebook Size 2128 for Unified Multimodal Large Language Model	arXiv 2026	Paper	UniWeTok unified binary tokenizer with 2^{128} codebook, pre-post distillation and generative-aware prior for MLLMs	visual understanding, visual generation
Towards Scalable Pre-training of Visual Tokenizers for Generation	arXiv 2025	Paper Code	VTP unified visual tokenizer pre-training framework with joint image-text contrastive, self-supervised and reconstruction losses	visual understanding, visual generation
The Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding	arXiv 2025	Paper Code	Prism Hypothesis and unified autoencoding (UAE) harmonizing semantic and pixel representations across modalities	visual understanding, visual generation
UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding	CVPRW 2025	Paper Code	Unified visual encoding combining discrete and continuous representations for autoregressive multimodal models	multimodal understanding and generation
VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning	arXiv 2025	Paper Code	Enhanced visual autoregressive unified model with iterative instruction tuning and DPO reinforcement learning	visual understanding, generation and editing
ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinement	arXiv 2025	Paper Code	Dual visual tokenization and diffusion refinement for unified multimodal large language model	multimodal understanding and generation
SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation	arXiv 2025	Paper	Semantic-guided hierarchical codebook for unified image tokenization supporting understanding and generation	multimodal understanding and generation
VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model	arXiv 2025	Paper Code	Visual autoregressive framework unifying understanding and generation in a single MLLM	visual understanding and generation
TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation	CVPR 2025	Paper Code	Unified image tokenizer with dual-codebook architecture bridging understanding and generation	multimodal understanding and generation
MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding	arXiv 2024	Paper	Semantic discrete encoding for unified vision-language model enabling efficient multimodal understanding and generation	multimodal understanding and generation

Hybrid (AR + Diffusion) UMMs

Pixel Encoding

Paper	Venue	Links	Notes	Task
PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation	arXiv 2026-09-29	Paper Project Code Model	First public: 2026-09-29. Encoder-free MoT directly projects raw image patches/video tubelets into shared multimodal attention; AR text modeling and pixel-space flow matching provide understanding and visual generation. Architecture-native, initialized with Qwen3-8B rather than all components from scratch. As of 2026-10-19: code and checkpoints released; checkpoint use is limited to non-commercial research/evaluation under the NVIDIA One-Way Noncommercial License.	image/video understanding, image/video generation
Tuna: Taming Unified Visual Representations for Native Unified Multimodal Models	arXiv 2025	Paper Code	Native unified multimodal model with cascaded VAE + representation encoder for unified continuous visual representations	multimodal understanding and generation
LMFusion: Adapting Pretrained Language Models for Multimodal Generation	arXiv 2024	Paper	Adapting pretrained LLMs (Llama) for multimodal generation by adding parallel diffusion modules while keeping autoregressive text modeling	multimodal understanding and generation
MonoFormer: One Transformer for Both Diffusion and Autoregression	arXiv 2024	Paper Code	Single shared transformer backbone that handles both autoregressive modeling and diffusion for unified multimodal tasks	visual understanding and generation
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation	ICLR 2025	Paper Code	Unified transformer combining autoregressive and discrete diffusion modeling to flexibly handle mixed-modality inputs/outputs	multimodal understanding and generation
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model	ICLR 2025	Paper	7B-scale report; single multimodal model combining next-token text prediction with image diffusion over mixed discrete/continuous sequences	visual understanding, visual generation

Hybrid Encoding

Paper	Venue	Links	Notes	Task
Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation	arXiv 2026-10-12	Paper	First public: 2026-10-12. MoT combines AR language/scene-state outputs with flow-matching image synthesis. Persistent visual anchors, scene-aware memory retrieval and self-rollout stabilize long interleaved trajectories; reports 256K multimodal context and 50+ generated images. Whole-model scratch initialization is not established. As of 2026-10-19: paper public; official code/weight release not confirmed by inspected primary sources.	visual understanding, long-horizon interleaved text/image generation
Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States	arXiv 2026-09-03	Paper Project Code Model	First public: 2026-09-03. Geometry-aligned visual encoding and an AR language model produce camera/scene understanding; learnable queries connect language hidden states to a diffusion generator with appearance/depth and camera conditions. Native 3D state terminology does not establish whole-model scratch training. As of 2026-10-19: code, weights and Puffin-16M data released.	spatial understanding, camera-controlled image generation, 3D reconstruction
Object-Uni: A Unified Model for Object-Centric Spatial Understanding and Controllable Generation	arXiv 2026-08-24	Paper	First public: 2026-08-24. Adapts pretrained Ming-Lite-Uni with PoseAnchor object tokens and orientation heads for AR spatial reasoning; multi-scale queries and CNOCS conditions guide a diffusion transformer. Most pretrained backbone parameters are frozen during understanding adaptation. As of 2026-10-19: paper public; official code/weight release not confirmed by inspected primary sources.	object pose perception, spatial reasoning, pose-controlled image and novel-view generation
MedUAG: Unified Understanding and Generation for Medical Multimodal Models	arXiv 2026-08-19	Paper	First public: 2026-08-19. Initializes from BAGEL-7B-MoT with semantic ViT and VAE latent paths, AR text outputs and flow-matching image synthesis. Domain alignment and instruction tuning cover 14 medical imaging modalities; end-to-end adaptation does not mean training from scratch. As of 2026-10-19: paper public; official code/weight release not confirmed by inspected primary sources.	medical visual QA/reporting, image synthesis, translation, reconstruction and prediction
Rosetta: Composable Native Multimodal Pretraining	NeurIPS 2026	Paper Project Code Model	First public: 2026-07-01. Composable experts and momentum-anchored gradient projection preserve prior capabilities as modalities are added; shared attention combines AR understanding with flow-matching visual generation. Appendix C specifies Qwen3-0.6B sparse upcycling, frozen Qwen3-VL ViT and FLUX.2 VAE, so native does not mean all components from scratch. As of 2026-10-19: code and checkpoints released; NeurIPS venue is author-reported.	language/visual understanding, image generation, composable multimodal training
ILLUME-X: Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation	ECCV 2026 (author-reported)	Paper Code Model	First public: 2026-06-29. ViT semantic and VAE latent paths feed a unified transformer with causal text/control tokens and bidirectional visual segments; text cross-entropy and rectified-flow image losses support free-form interleaving. Distinct from ILLUME/ILLUME+. As of 2026-10-19: inference code and checkpoints released; training data marked coming soon.	visual understanding, free-form interleaved text/image generation and editing
Lance: Unified Multimodal Modeling by Multi-Task Synergy	arXiv 2026-05-18	Paper Project Code Model	First public: 2026-05-18. 3B active parameters; shared interleaved context with specialized AR understanding and flow-matching generation pathways. Paper §3.2 explicitly initializes the dual-expert architecture from Qwen2.5-VL and reuses its text embeddings/ViT plus a Wan2.2 3D causal VAE; whole-model scratch training is not supported by these details. As of 2026-10-19: inference code/weights released 2026-05-18; fine-tuning code released 2026-06-17.	image/video understanding, generation and editing
Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training	CVPR 2026 (Oral)	Paper Project	First public: 2026-03-26. MoT planner/visualizer uses a pretrained Qwen2.5-VL AR planner for understanding answers and dense prompts, plus a diffusion transformer visualizer trained from scratch. Decoupled training avoids requiring real interleaved training data; only the visualizer has the stated scratch initialization. As of 2026-10-19: paper/project demos public; official code/weight release not confirmed by inspected primary sources. Oral venue is author-reported.	visual understanding, interleaved text/image generation
ChatUMM: Robust Context Tracking for Conversational Interleaved Generation	arXiv 2026-02-06	Paper Project	First public: 2026-02-06. BAGEL-style MoT uses shared attention, task-specific FFNs, semantic ViT tokens and VAE image latents; AR text/control prediction and flow matching produce context-dependent interleaved replies. Whole-model scratch initialization is not established. As of 2026-10-19: paper revised 2026-06-01 and project examples public; official code/weight release not confirmed by inspected primary sources.	conversational visual understanding, history-dependent image editing, interleaved text/image generation
Vision as Unified Multimodal Generation (SenseNova-Vision)	arXiv 2026-07-07	Paper Code Model Collection	First public: 2026-07-07. Fine-tunes pretrained BAGEL-7B-MoT to express computer-vision tasks through AR text, diffusion-generated images or mixed responses without task-specific prediction heads. As of 2026-10-19: inference code, model weights and corpus public; checkpoint license is CC BY-NC 4.0 (non-commercial), distinct from repository code licensing.	detection, OCR, keypoints, segmentation, depth, normals, point maps, camera pose
S1-Omni-Image: A Unified Model for Scientific Image Understanding, Generation, and Editing	arXiv 2026	Paper	Builds on S1-VL-32B and injects reasoning hidden states into an image generation module for scientific image understanding, generation and editing	scientific image understanding, generation and editing
Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE	arXiv 2026-05-04	Paper Code	First public: 2026-02-15 (inference code); paper first submitted 2026-05-04. AR multimodal understanding coupled to a DiT-MoE generation backbone (25B total / 3B active in DiT), supporting image/video generation and editing, including few-step video editing. As of 2026-10-19: paper and inference code public; official repository still lists Mamoda2.5 weights under internal review. Released Mamoda2 checkpoints are a separate version.	multimodal understanding, image/video generation, image/video editing
InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing	arXiv 2026	Paper Code	4B unified multimodal model integrating a strong MLLM with an MMDiT-based visual generation head for understanding, reasoning, generation and editing	multimodal understanding, reasoning, generation and editing
Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editing	arXiv 2026	Paper Code	Connects pretrained MLLMs with video diffusion models through lightweight adapters for unified video generation and editing	video generation and editing
EMMA: Efficient Multimodal Understanding, Generation, and Editing with a Unified Architecture	arXiv 2025	Paper Code	Efficient unified architecture with autoencoders, channel-wise concatenation, shared-decoupled networks and MoE for understanding, generation and editing	multimodal understanding, generation and editing
HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation	arXiv 2025	Paper	Asymmetric H-shaped architecture bridging heterogeneous experts with symmetric dense mid-layer connections for unified multimodal modeling	multimodal understanding and generation
LightFusion: A Light-weighted, Double Fusion Framework for Unified Multimodal Understanding and Generation	arXiv 2025	Paper Code	Light-weighted double fusion framework that efficiently integrates pretrained vision-language and diffusion models	multimodal understanding and generation
Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Model	arXiv 2025	Paper	Qwen2.5-VL connects through MetaQueries to an SD3.5-Medium DiT; unified understanding, diffusion-based generation and editing with online RL.	visual understanding, visual generation, editing
TBAC-UniImage: Unified Understanding and Generation by Ladder-Side Diffusion Tuning	arXiv 2025	Paper	A pretrained diffusion generator receives conditions from multiple MLLM layers through ladder-side connections.	visual understanding, visual generation
UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing	arXiv 2025	Paper	A continuous CLIP-based tokenizer supports reconstruction; MLLM hidden states and learnable queries jointly condition a diffusion transformer for generation and editing.	visual understanding, visual generation, editing
Ovis-U1 Technical Report	arXiv 2025	Paper Code	Ovis understanding backbone coupled to an MMDiT visual decoder and bidirectional token refiner; diffusion samples images from LLM conditions.	visual understanding, visual generation
OmniGen2: Exploration to Advanced Multimodal Generation	arXiv 2025	Paper Code	Autoregressive MLLM with a separate diffusion transformer; text hidden states and VAE features condition rectified-flow image generation and editing.	visual generation
Show-o2: Improved Native Unified Multimodal Models	NeurIPS 2025	Paper Code	Autoregressive language modeling and flow matching for image/video generation share unified visual representations in a 3D causal VAE space.	multimodal understanding and generation
UniWorld: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation	arXiv 2025	Paper Code	Retains a frozen autoregressive Qwen2.5-VL understanding/text-output branch (Fig. 3; Table 1); SigLIP2 reference features and an MLP connector condition a FLUX generator trained with flow matching.	visual understanding, visual generation
OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation	arXiv 2025	Paper Code	Learnable queries and a lightweight transformer connector bridge an autoregressive MLLM to a diffusion image generator.	visual understanding, visual generation
BAGEL: Emerging Properties in Unified Multimodal Pretraining	arXiv 2025	Paper Code	Open-source foundational decoder-only model pretrained on trillions of interleaved multimodal tokens supporting native understanding and generation	multimodal understanding and generation
MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO	arXiv 2025	Paper Code	Autoregressive multimodal reasoning conditions a diffusion decoder trained with rectified-flow objectives; RGPO improves reasoning-guided generation.	multimodal understanding, reasoning, image generation, editing
BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset	arXiv 2025	Paper	Autoregressive multimodal understanding with a diffusion transformer that generates CLIP image features for image synthesis.	visual understanding, visual generation
Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation	arXiv 2025	Paper	Causal interleaved multi-modal generation framework with deep-fusion, dual vision encoders and multi-modal classifier-free guidance	interleaved multimodal generation
Ming-Lite-Uni: Advancements in Unified Architecture for Natural Multimodal Interaction	arXiv 2025	Paper	A fixed autoregressive MLLM connects through multi-scale learnable tokens to a trainable diffusion model for image generation and editing.	visual understanding, visual generation
Transfer between Modalities with MetaQueries	arXiv 2025	Paper Code	Learnable MetaQueries connect autoregressive MLLM latents to a diffusion decoder; the MLLM can remain frozen while diffusion learns image generation.	visual understanding, visual generation
JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation	arXiv 2024	Paper Code	Minimalist framework harmonizing autoregressive LLMs with rectified flow for efficient unified understanding and generation	multimodal understanding and generation
SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation	arXiv 2024	Paper Code	Learnable image queries reconstruct ViT features; an SDXL-initialized visual decoder samples images and supports high-precision manipulation.	visual understanding, visual generation
Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models	arXiv 2024	Paper Code	Multimodal understanding with an autoregressive language model; generated text prompts drive a pretrained SDXL image generator.	visual understanding, visual generation
MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer	arXiv 2024	Paper Code	Autoregressive text modeling with a diffusion image decoder, connected through resampling and a multi-scale multi-image feature synchronizer.	visual understanding, visual generation
DreamLLM: Synergistic Multimodal Comprehension and Creation	ICLR 2024	Paper	Autoregressive text and interleaved structure modeling; learnable dream queries condition a Stable Diffusion image decoder.	visual understanding, visual generation

4.2 Any-to-Any / Omni UMMs

Models that extend unified understanding and visual generation to additional modalities such as audio, video, and speech. “Omni” does not imply that every input modality can also be generated; supported outputs are specified per entry. Models restricted to text/speech output, including Qwen3-Omni and LongCat-Flash-Omni, belong in Omni MLLMs.

Model	Paper	Links	Notes	Task
Kling-Omni Technical Report	arXiv 2025	Paper	Unified Diffusion Transformer (DiT) framework with Prompt Enhancer for high-fidelity video generation and reasoning-based editing	Multi-modal visual language (MVL) for unified generation and understanding
Ming-flash-omni 2.0	Model release 2026-02-11	HF	Open-weight 100B-total / 6B-active MoE model (MIT) based on Ling-2.0; unifies image, text, video, and audio inputs with image, text, and audio outputs, including image editing and controllable speech/audio/music synthesis	multimodal understanding, image generation/editing, audio generation, streaming video conversation
Ming-Flash-Omni	arXiv 2025	Paper Code	Sparse unified MoE architecture (100B total, 6.1B active) for efficient multimodal perception and generation	any-to-any multimodal perception and generation
Ming-Omni	arXiv 2025	Paper Code	Unified multimodal architecture for perception + generation (images, text, audio, video)	any-to-any multimodal tasks
M2-Omni	arXiv 2025	Paper	Extends Omni-MLLM with broader modality support and competitive performance to GPT-4o	any-to-any multimodal modeling
Spider	arXiv 2024	Paper Code	Any-to-many multimodal LLM with flexible output heads for arbitrary modality combinations	multimodal understanding and generation
MIO	arXiv 2024	Paper	Token-level unified multimodal foundation model on discrete multimodal tokens	any-to-any multimodal token modeling
X-VILA	arXiv 2024	Paper	Cross-modality alignment for LLM-based multimodal systems (image/video/audio)	multimodal understanding
AnyGPT	arXiv 2024	Paper Code	Discrete token modeling for unified multimodal generation	any-to-any multimodal generation
OmniFlow	CVPR 2025	Paper	Uses multi-modal rectified flows for any-to-any generation across modalities	any-to-any generation across modalities
Video-LaVIT	ICML 2024	Paper Code	Decoupled visual-motion tokenization for video-language modeling	video understanding and generation
Unified-IO 2	CVPR 2024	Paper Code	Scales autoregressive multimodal models across modalities	any-to-any multimodal tasks (vision, language, audio, action)
NExT-GPT	arXiv 2023	Paper Code	Any-to-any; encoder+LLM+diffusion decoders	visual understanding, visual generation, omni

4.3 Related Generators (U+G Unverified)

These boundary references support architecture comparisons and are not verified UMM members. The linked report establishes generation/editing but does not establish a separate image-understanding task; move an entry into the UMM taxonomy only when that evidence is available. Reference rows count toward the website’s catalog-entry totals, not a count of verified UMMs.

Model	Venue	Links	Paradigm	Notes	Task
Qwen-Image-2.0 Technical Report	arXiv 2026	Paper	MMDiT diffusion	Frozen Qwen3-VL conditions an MMDiT image generator with a high-compression VAE. The report establishes generation and editing; separate image-understanding task evidence is unverified, so this is a boundary reference rather than a verified UMM.	image generation, editing

Back to Top

5. Native Multimodal Models (NMMs)

In this section: 5.1 Design Analyses & Scaling Laws · 5.2 Early Fusion NMMs · 5.3 Encoder-Based NMMs · 5.4 Any-to-Any / Omni NMMs · 5.5 Training-Native Models

Two dimensions of nativity: architectural integration and multimodal optimization. Architecture-native entries may inherit pretrained backbones; training-native entries are distinguished from claims about fusion depth. From-scratch initialization is stated only where supported. Models may also retain their UMM classification.

What recent arXiv work emphasizes: native multimodality is increasingly defined by end-to-end multimodal pretraining, tokenizer/representation co-design, and scaling strategies that explicitly address the asymmetry between vision and language.

5.1 Design Analyses & Scaling Laws

Recent arXiv papers sharpen the definition of NMMs and identify the main bottlenecks in native multimodal pretraining.

Paper	Venue	Links	Insights
Can Visual Generation Expand What Unified Multimodal Models Know?	arXiv 2026-10-15	Paper	Knowledge Transfer Probing supplies knowledge through image-generation supervision and tests access through text-only QA across seven categories and three UMMs. Transfer is uneven; textual and interleaved alignment improve access, without establishing a universal benefit from generation.
More Than Seeing: Vision as Scalable Anchors in Native Multimodal Pre-Training	arXiv 2026-10-14	Paper	Controlled from-scratch raw-patch MoE pretraining compares visual content and multimodal mixtures, including transfer to text-only spatial tasks. Shared decoder experiments span 71M–3B active parameters and 325B input tokens; this is a pretraining study rather than a separately released frontier model.
Scaling Laws for Multimodal Data Mixtures	NeurIPS 2026 Spotlight (author-reported); arXiv 2026-10-10	Paper	Studies text/vision/audio mixtures through 268 MoE runs (457M–8.3B parameters, up to 150B tokens). Finds modality-specific compute optima and asymmetric interference within the tested mixtures; results and extrapolations retain this experimental scope.
Multimodal Co-Generation With Continuous Bitstream Diffusion (CoBit)	arXiv 2026-10-09	Paper Code	A common bitstream-diffusion objective is tested with separately trained image–text, speech–text, and protein sequence–structure models. Denoisers train from scratch, while some experiments use frozen pretrained codecs. The image experiment uses MNIST-Sum; this is not one general-purpose omni checkpoint.
How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining	arXiv 2026-09-28	Paper	Matched encoder-based and encoder-free pretraining compares 11 MoE decoder scales (1.1B–44B total; 71M–2.4B active parameters). Raw patches enter the shared decoder directly in the encoder-free branch. The projected crossover near 10²² FLOPs is an extrapolation beyond measured runs.
Uncovering Understanding–Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System	arXiv 2026-09-01	Paper	Controlled pixel-in/pixel-out experiments study representation, task mixtures, routing, and planner/executor synergy. The backbone initializes from Qwen3-1.7B, so native describes architectural integration rather than whole-model training from scratch.
Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes	arXiv 2026-08-05	Paper Project	Compares when vision enters pretraining, joint versus sequential schedules, representations, and shared versus modality-specific computation; scales to 13.5B MoE and 2T tokens. Default visual representations use a frozen pretrained SigLIP2, so the study does not establish all-components-from-scratch training.
Scaling Native Multimodal Pre-Training From Scratch	arXiv 2026	Paper	Derives compute and allocation laws for transformer-based VLMs trained from scratch, showing distinct language and multimodal scaling behavior, data-mixture-sensitive multimodal allocation, and positive cross-modal transfer
Toward Native Multimodal Modeling: A Roadmap	arXiv 2026	Paper	The end-to-end pipeline from industrial perspectives from architectural coordination, massive data curation, to full-stack training recipes, inference & deployment, and the comprehensive evaluation for truly native modeling
Beyond Language Modeling: An Exploration of Multimodal Pretraining	ICML 2026; arXiv 2026-03-03	Paper Proceedings	Controlled Transfusion-style multimodal pretraining studies unified visual representations, understanding/generation and world-modeling transfer, MoE, and vision–language scaling asymmetry.
NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints	arXiv 2025	Paper Code	End-to-end native MLLM scaling shows positive correlation between visual encoder and LLM size under data constraints; optimal meta-architecture balances cost and performance
Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training	arXiv 2025	Paper	reveals that LLMs develop latent visual priors during text-only pre-training, where reasoning-centric data (code and math) builds transferable visual reasoning skills while broad corpora foster perception, enabling models to 'see' before ever processing an image.
Scaling Laws for Native Multimodal Models	arXiv 2025	Paper	Early-fusion NMMs match or outperform late-fusion at low compute; early-fusion needs fewer params; MoE with modality-agnostic routing boosts sparse NMM scaling
The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models	arXiv 2024	Paper	Native models often funnel image-to-text communication through a single post-image token

5.2 Early Fusion NMMs

Modality tokens or embeddings enter shared backbone computation before its first block. Separate input encoders, tokenizers, experts, and decoders are compatible with early fusion. Architecture-native entries are explicitly qualified where pretrained initialization prevents an equally strong from-scratch claim.

Recent scaling-law evidence suggests early-fusion NMMs are often stronger at lower parameter counts and simpler to deploy when paired with sufficiently strong visual representations.

Model	Paper	Links	Training Scale	Notes	Task
NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference	arXiv 2026-08-31	Paper HF	260M / 800M; 16K context	Open-weight bidirectional single-tower encoder (Apache 2.0), trained from random initialization on text and raw image patches without a separate vision tower. Multimodal examples participate from backbone pretraining; masked text prediction supports representations and retrieval, not image generation or general chat.	multimodal representations, visual document retrieval, multilingual understanding
DeepSeek-V4.1-Flash	Model release 2026-09-10	Announcement HF	552B backbone; 45T multimodal pretraining tokens; 1M context	Open-weight model (MIT) trained from scratch on multimodal data; DeepSeek-ViT and a two-layer MLP produce image embeddings jointly processed with text from the start of language-model pretraining. Shared computation uses a Causal Encoder-Decoder architecture	image/text understanding, visual reasoning, coding, long-context agents
Qwen3.8-Flash-Next	Model release 2026-08-26	Code HF Implementation	125B backbone / 6B active; 51B n-gram embeddings + 4B MTP	Open-weight experimental architecture preview under Qwen Community License 1.0; architecture-native early fusion inserts image/video embeddings into the shared language-backbone input. Combines Gated DeltaNet, Qwen Sparse Attention, and gated residuals; all-components-from-scratch initialization is unverified	image/video understanding, visual reasoning, coding, long-horizon agents
Qwen3.8-27B	Model release 2026-08-14	Code HF	27B dense; 262K native context, extensible to 1M	Open-weight architecture-native VLM built on Qwen3.5's early-fusion vision-language architecture; image/video embeddings and text share the language backbone. Training stage is pretraining plus post-training; all-components-from-scratch initialization is unverified	image/video understanding, document understanding, visual reasoning, agentic work
Inkling-Small	Model release 2026-07-30	Announcement Model Card HF	276B total / 12B active	Open-weight MoE model with a hierarchical image-patch encoder and discrete audio encoding; text, image, and audio representations enter a shared hidden space and are jointly processed by one decoder. Outputs text; fusion classification does not assert all-components-from-scratch initialization	text/image/audio understanding, visual and audio reasoning, coding, agentic tool use
Inkling	Thinking Machines Lab, 2026	Announcement Model Card HF	975B total / 41B active; 45T multimodal pretraining tokens; 1M context	Open-weight MoE model trained from scratch with encoder-free vision and audio inputs; images use 40×40 patches with a lightweight four-layer hMLP, while audio uses dMel spectrograms, jointly processed with text tokens	text/image/audio understanding, visual and audio reasoning, speech transcription, coding, agentic tool use
Gemma 4 12B Unified	Model release 2026-06-03; report 2026-07-02	Paper Announcement HF	12B	Open-weight encoder-free architecture (Apache 2.0): lightweight linear layers project raw image patches and audio waveforms into the shared decoder’s embedding space. Listed separately from encoder-based Gemma 4 variants; Unified refers to input architecture, with text-only output. The shared technical report first appeared 2026-07-02.	image/video/audio understanding, multimodal reasoning, on-device agents
HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers	arXiv 2026	Paper	7B dense model	Native unified multimodal model with a holistic image-video ViT tokenizer, covering image/video understanding, image/video generation, and image editing	image/video understanding, image/video generation, editing
NEO-OV	arXiv 2026	Paper	—	Native one-vision scaling from pixels to words, extending the NEO line toward stronger native visual primitives	vision-language understanding
SenseNova-U1.5-8B-MoT	Model release 2026-08-20	Paper Code HF	—	NEO-unify-based encoder-free MoT model with improved patchify layers, native 4K generation, text rendering, and image editing; grouped with U1 by architecture, with from-scratch initialization unverified	visual understanding, image generation, editing, interleaved generation
SenseNova-U1	arXiv 2026	Paper Code	—	NEO-Unify-based native unified multimodal model handling understanding, generation, and reasoning in one model	Unified Understanding & Generation
HiDream-O1-Image	arXiv 2026	Paper	—	Natively unified image generative foundation model with pixel-level unified transformer	image generation, unified modeling
Tuna-2	arXiv 2026	Paper	—	Native unified multimodal model that discards traditional vision encoders in favor of direct pixel embeddings for end-to-end understanding and generation	Unified Understanding & Generation
NEO	arXiv 2025	Paper	—	Native vision-language primitives at scale; paired with reusable components for cost-effective native VLM development	vision-language understanding
NEO-Unify	Blog 2025	Blog	—	Native unified extension of NEO for understanding, generation, and reasoning	Unified Understanding & Generation
Emu3.5	Nature 2026	Paper Code	Large-scale (trillion+ tokens)	Native world model; next-state prediction on interleaved video/text; Discrete Diffusion Adaptation for efficiency	interleaved generation, world modeling, any-to-image
LongCat-Next	arXiv 2026	Paper	—	Early fusion: DiNA represents text, vision, and audio as discrete tokens processed by a shared modality-agnostic autoregressive backbone; Omni capabilities	text/image/audio understanding and generation
Llama4	arXiv 2026	Paper Blog	Scout/Maverick: 17B active / ~109B–400B total; Behemoth: ~2T total	Architecture-native early fusion of text and vision tokens in a shared MoE backbone; separately trained vision encoder, so early fusion does not imply all components were trained from scratch	vision-language understanding
Emu3	arXiv 2024	Paper Code	8B	Early-fusion autoregressive backbone jointly models text, image, and video tokens; also indexed under UMM / AR / Pixel Encoding	visual understanding, visual generation
Chameleon	arXiv 2024	Paper Code	—	Early-fusion backbone trained from scratch on interleaved image-text tokens; separate visual tokenizer; also listed under UMM / AR / Pixel Encoding	visual understanding, visual generation
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model	ICLR 2025	Paper	7B	Early-fusion shared Transformer jointly models text tokens and continuous image latents with next-token prediction and diffusion; also listed under Hybrid UMMs	visual understanding, visual generation
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation	ICLR 2025	Paper Code	—	Architecture-native early fusion with a pretrained language-model initialization; shared Transformer combines autoregressive text modeling and masked image-token diffusion; UMM entry	multimodal understanding and generation
Show-o2: Improved Native Unified Multimodal Models	NeurIPS 2025	Paper Code	—	Architecture-native early fusion with autoregressive text modeling and visual flow matching; uses a pretrained language backbone, not an all-components-from-scratch model; UMM entry	multimodal understanding and generation
OneCat	arXiv 2025	Paper Code	—	Architecture-native early fusion with a Qwen2.5-initialized decoder and modality-specific experts; unified autoregressive understanding and generation; UMM entry	visual understanding, visual generation, editing

5.3 Encoder-Based NMMs

Encoder-based systems discussed in the native multimodal literature. A vision encoder may be pretrained or trained from scratch; its presence alone establishes neither strict late fusion nor whole-model training from scratch. The notes qualify the available training evidence independently, and leave the first cross-modal interaction layer unspecified unless supported by a source.

Model	Paper	Links	Training Scale	Notes	Task
Mistral Large 4	API preview 2026-10-06	Announcement Docs	1.05T total / 52B active; 1.6B vision encoder; 1M context	Officially described as a natively multimodal model trained from scratch, with text/image understanding and text output. The exact encoder-to-backbone fusion depth is undisclosed. API public preview is available; weights are promised by end-October 2026 and were not released as of 2026-10-19.	visual understanding, grounding, document reasoning, coding, agents
Kimi K3: Open Frontier Intelligence	arXiv 2026	Paper HF	2.8T total / 104B active; 1M context	Joint language-vision training from the outset; MoonViT-V2 is trained from scratch and connected to the language backbone through pixel-shuffle downsampling and an MLP projector	image/video understanding, visual reasoning, long-horizon agents
Kimi K2.6	Moonshot AI 2026	Blog	MoE Architecture: 32B active / 1T total parameters; supports 256K context	Native multimodal MoE with MLA (Multi-head Latent Attention) and MoonViT encoder. Encoder initialization and strict fusion depth are not established in this entry; native training terminology does not imply all components were trained from scratch.	Unified multimodal understanding, long-horizon coding, and agent swarms
GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents	arXiv 2026	Paper	—	Native multimodal foundation model featuring CogViT vision encoder, Multimodal Multi-Token Prediction (MMTP), and joint RL for agentic GUI/Design2Code tasks. Encoder initialization and strict fusion depth are not established in this entry; native training terminology does not imply all components were trained from scratch.	multimodal agentic reasoning
LLaVA-OneVision-2	arXiv 2026	Code HF	8B	Native multimodal training with OneVision-Encoder and Qwen3 text backbone; codec-aligned visual encoding for image, long-video, spatial, document/OCR/chart understanding. Encoder initialization and strict fusion depth are not established in this entry; native training terminology does not imply all components were trained from scratch.	visual understanding, video, spatial reasoning
Qwen3.5	-	Blog	—	Discrete Native Any-resolution Visual Transformer. Encoder initialization and strict fusion depth are not established in this entry; native training terminology does not imply all components were trained from scratch.	vision-language understanding
Gemma 4 (encoder-based variants)	Model release 2026-04-02; report 2026-07-02	Paper Announcement Model Card	—	First public: 2026-04-02. Technical report first submitted 2026-07-02. Open-weight encoder-based Gemma 4 variants (Apache 2.0) with multimodal pretraining and text output; the public evidence cited here does not establish exact encoder initialization or strict fusion depth. The encoder-free 12B Unified variant is listed separately above.	vision-language understanding

5.4 Any-to-Any / Omni NMMs

The latest arXiv-native multimodal papers increasingly blur the boundaries between omni understanding, any-to-any generation, world modeling, and RL-enhanced post-training.

Model	Paper	Links	Training Scale	Notes	Task
HyperCLOVA X 8B Omni	arXiv 2026	Paper HF	8B	Any-to-any omnimodal model unifying text, audio, and vision through a shared next-token prediction interface over interleaved multimodal sequences	text/audio/vision understanding and generation
MiniMax-M3	MiniMax 2026	Blog Code	~428B total / ~23B active; 1M context	Native multimodal model with MiniMax Sparse Attention (MSA), trained for text, image, and video understanding, long-context coding, agentic workflows, and computer-use tasks	native multimodal understanding, video, coding, agentic computer use
The Design Space of Tri-Modal Masked Diffusion Models	NeurIPS 2026; arXiv 2026-02-25	Paper Apple Research	3B; 6.4T tokens	From-scratch tri-modal masked-diffusion pretraining over text, image–text, and audio–text data; analyzes scaling, modality mixing, noise schedules, batch size, and inference. Apple’s October NeurIPS listing updates the venue, not the first-paper date.	text generation, text-to-image, text-to-speech
Qwen3.5-Omni	Qwen Blog 2026	Blog	—	Discrete native any-resolution visual transformer with omni-modal extension	vision-language understanding, omni
ERNIE 5.0 Technical Report	arXiv 2026 (Late fusion)	Paper	—	Natively autoregressive foundation model designed for unified multimodal understanding and generation across text, image, video, and audio	vision-language understanding, omni

5.5 Training-Native Models

These entries record a native multimodal training route separately from fusion topology. This qualification does not claim that every component is trained from scratch; no Early, Mid, or Late Fusion label is assigned without architectural evidence.

Model	Paper	Links	Training Scale	Notes	Task
InternVL3	arXiv 2025	Paper	—	Training-native route: native multimodal pretraining within a ViT–MLP–LLM architecture; pretrained components are retained. This training qualification does not establish strict late or mid fusion; fusion depth is left unspecified here.	vision-language understanding
InternVL3.5	arXiv 2025	Paper	—	Training-native route: native multimodal pretraining within a ViT–MLP–LLM architecture; pretrained components are retained. This training qualification does not establish strict late or mid fusion; fusion depth is left unspecified here.	vision-language understanding

Back to Top

6. Closed-Source Multimodal Models

Year 2026

Model	Venue	Links	Notes	Task
Claude Haiku 5.5	Anthropic; release 2026-10-07	Announcement Models	Efficient model for focused, high-volume work; proprietary Claude/API release supporting text/image inputs and text output. A distinct 5.5-family release rather than a rename of Claude 5 or Fable/Mythos 5.1.	visual understanding, reasoning, coding, computer use, agents
GPT-6.1 Sol	OpenAI; API release 2026-09-29	Model Changelog	Proprietary reasoning model accepting text/image inputs and producing text, available through the Responses and Chat Completions APIs. Supports visual reasoning, computer use and coding; image_generation is an external tool capability, not evidence of a native image-generation backbone.	visual understanding, reasoning, coding, computer use, agents
Claude Sonnet 5.5	Anthropic; release 2026-09-28	Announcement Models	Balanced model for coding and agentic work; proprietary Claude/API release supporting text/image inputs and text output. A distinct 5.5-family release rather than a rename of Claude 5 or Fable/Mythos 5.1.	visual understanding, reasoning, coding, computer use, agents
Sarvam Vision 2.1	Model release 2026-09-24	Announcement Digitize API Extract API	First public: 2026-09-24. Proprietary document VLM with semantic-layout and pointer reading-order harnesses; document-page/text inputs produce structured text, tables and key-value fields, including Indic handwriting. Digitize and Extract APIs available; open checkpoints not established as of 2026-10-19. Backbone initialization and vision-to-language interface are undisclosed; benchmark claims are author-reported.	multilingual document understanding, Indic OCR, handwriting, structured extraction
GPT-6 Sol / GPT-6 Luna	OpenAI; API release 2026-09-22	Sol Luna Changelog	Proprietary reasoning models accepting text/image inputs and producing text through the Responses and Chat Completions APIs. Sol targets complex coding and agent workflows; Luna targets efficient focused tasks. Retained separately from GPT-6.1 Sol, GPT-6 Astra, and the GPT-5.6 family.	visual understanding, reasoning, coding, computer use, agents
Claude Opus 5.5	Anthropic; release 2026-09-22	Announcement Models	Flagship model for complex coding, computer use, and professional work; proprietary Claude/API release supporting text/image inputs and text output. A distinct 5.5-family release rather than a rename of Claude 5 or Fable/Mythos 5.1.	visual understanding, reasoning, coding, computer use, agents
Gemini 3.8 Live / Live Extended Thinking	Google DeepMind; release 2026-09-15	Announcement Live Avatar	Proprietary live-dialogue models combine audio dialogue, near-real-time visual grounding, and asynchronous tools; Extended Thinking reasons while speaking. Available through Gemini Live/API services. The 2026-09-24 Live Avatar extension adds streamed speech/video responses in Gemini Enterprise; its announcement does not establish one shared native video-generation backbone.	audio-visual understanding, speech generation, live dialogue, tool use
GPT-6 Astra	OpenAI, 2026	Announcement System Card	September 2026 proprietary model with text/image input and text output, supporting visual reasoning, coding, computer use, and long-horizon agentic work	visual understanding, multimodal reasoning, coding, computer use, agents
Claude Fable 5.1 / Mythos 5.1	Anthropic, 2026	Announcement Models	September 2026 release; Fable 5.1 and Mythos 5.1 are the same underlying model with different safeguards. Fable is generally available, while Mythos is restricted to trusted access programs; grouped as one entry	visual understanding, reasoning, coding, computer use, long-horizon agents
Muse Spark 1.3	Meta; September 2026 release	Announcement Muse Spark Open	Proprietary Muse Spark update for multimodal perception, coding, and agentic work through Muse Code and Meta Model API. Meta separately released Muse Spark Open weights on 2026-10-15; this does not establish that the proprietary 1.3 API model has identical weights or license. Status rechecked 2026-10-19.	visual understanding, multimodal reasoning, coding, agents
Gemini 3.8 Flash	Google DeepMind, 2026	Model Card	September 2026 Gemini Flash model supporting text, image, audio, and video inputs with text output; up to 1M input context, with configurable reasoning effort	multimodal understanding, video/audio understanding, reasoning, coding, agents
Grok 4.6	xAI, 2026	Announcement Model Docs	August 2026 proprietary model with text/image input and text output; extends the Grok family with visual work, coding, and long-running agent workflows	visual understanding, multimodal reasoning, coding, agents
Claude 5 Family (Opus 5 / Sonnet 5)	Anthropic, 2026	Opus 5 Sonnet 5 Models	Opus 5 (July 2026) and Sonnet 5 (June 2026) support text/image input and text output, with vision, reasoning, tool use, and agentic coding; retained separately from the Fable/Mythos family	visual understanding, reasoning, coding, computer use, agents
Seed2.1 Pro / Turbo	ByteDance Seed, 2026	Announcement Models	June 2026 proprietary Seed2.1 family; Pro and Turbo provide different model sizes for visual/video understanding, reasoning, software engineering, and cross-tool agent workflows	visual/video understanding, multimodal reasoning, coding, agents
GPT-5.6 Sol / Terra / Luna	OpenAI, 2026	Announcement	July 2026 general-availability release of the GPT-5.6 family: Sol is the flagship model, Terra balances capability and cost, and Luna prioritizes speed and affordability.	Multimodal Reasoning + Computer Use + Agentic Work
GPT-5.4 / GPT-5.5	OpenAI Blog	GPT-5.4 GPT-5.5	2026 GPT-5-series updates with improved reasoning, multimodal capability, and deployment efficiency.	Omni-Modal + Professional/Agentic Workflows
Claude 4.6 Family (Opus 4.6 / Sonnet 4.6)	Anthropic Blog	Opus 4.6 Sonnet 4.6	Claude 4.6 proprietary model family with vision, coding, tool use, and computer-use workflows.	Multimodal + Agentic/Coding/Computer-Use
Gemini 3.1 Pro / Gemini 3.5	Google Blog	Gemini 3.1 Pro Gemini 3.5	2026 Gemini 3-series updates with stronger multimodal reasoning, long-context, and action-oriented capabilities.	Frontier Multimodal (text/image/audio/video + reasoning)

Year 2025

Model	Venue	Links	Notes	Task
Gemini 3 / Gemini 3 Pro	Google Blog	Gemini 3	Native multimodal Gemini 3 generation with advanced reasoning, coding, and long-context capabilities.	Frontier Multimodal + Reasoning
GPT-5 / GPT-4.5	OpenAI Blog	GPT-5 GPT-4.5	Proprietary OpenAI GPT-series updates with multimodal reasoning and tool-use support.	Omni-Modal + Advanced Reasoning/Agentic
Claude 4 Family (Opus 4 / Sonnet 4)	Anthropic Blog	Claude 4	Claude 4 model family for coding, agentic tasks, vision, and extended tool use.	Vision + Advanced Reasoning/Agentic Workflows
Grok 3 / Grok 4 / Grok 4.1	xAI Announcement	Grok 3 Grok 4 Grok 4.1	Proprietary xAI models with multimodal input, real-time platform integration, and reasoning-focused releases.	Multimodal Reasoning + Real-Time Integration
Gemini 2.0 / 2.5 (Pro / Flash)	Google Blog	Gemini 2.0 Gemini 2.5 Pro Gemini 2.5 Updates	Native multimodal Gemini 2-series models with long context, video/audio/image understanding, and agentic features.	Advanced Native Multimodal + Agentic
Mistral Medium 3	Mistral AI	Announcement	Proprietary Mistral model offering text and vision capabilities through API/platform deployments.	General Multimodal Tasks

Year 2024

Model	Venue	Links	Notes	Task
Gemini 1.5 (Pro / Flash)	Google DeepMind Blog	Gemini 1.5 Announcement	Released February 2024. Massive context (>1M tokens), strong long-context multimodal (video, audio, images). Proprietary.	Long-Context Multimodal (video/audio/image/text)
Claude 3 Family (Opus / Sonnet / Haiku)	Anthropic Blog	Claude 3 Family	Released March 2024. Strong native vision for images, charts, diagrams, and documents. Proprietary API + Claude.ai.	Vision-Language + Reasoning
Claude 3.5 Sonnet	Anthropic Blog	Announcement	Stronger proprietary Claude vision/reasoning model used for images, documents, charts, and coding workflows.	Vision-Language + Reasoning
GPT-4o / GPT-4o mini	OpenAI Blog	GPT-4o GPT-4o mini	GPT-4o introduced real-time omni interactions; GPT-4o mini provided a lower-cost multimodal model.	Real-Time Omni-Modal (text/vision/audio)
Grok-1.5V / Grok-2 Vision	xAI Announcement	Grok-1.5V Grok-2	Vision-capable Grok models for image understanding, diagram reasoning, and X platform integration.	Vision-Language
Amazon Nova (Pro / Lite / Canvas / Reel)	Amazon Announcement	AWS Blog	Amazon Bedrock model family covering multimodal understanding and image/video generation variants.	Multimodal Understanding + Generation
Pixtral Large	Mistral AI	Announcement	Proprietary frontier multimodal Mistral model for image-text understanding through Le Chat/API deployments.	Vision-Language

Year 2023

Model	Venue	Links	Notes	Task
GPT-4V (Vision)	OpenAI Announcement	GPT-4V System Card	Released September 2023. First widely available multimodal GPT-4 variant. Image + text input, text output. API/ChatGPT access only.	Vision-Language (image understanding, VQA, OCR, document analysis, captioning)
Gemini 1.0 (Ultra / Pro / Nano)	Google DeepMind Blog	Gemini Announcement	Released December 2023. Native multimodal from training (text + image + audio + video). Proprietary API + Gemini chatbot.	Native Multimodal Understanding (text/image/audio/video)

7. Resources

In this section: 7.1 Related Awesome Lists · 7.2 Slides & Survey Papers · 7.3 Code Repositories & Tools · 7.4 Benchmarks & Evaluation

7.1 Related Awesome Lists

Repository	Focus	Author
awesome-multimodal-ml	General multimodal ML	pliang279
Awesome-Multimodal-Large-Language-Models	MLLMs + evaluation	BradyFU
Awesome-Multimodal-Research	Broad multimodal research	Eurus-Holmes
Awesome-Unified-Multimodal-Models	UMMs	ShowLab
Awesome-Multimodal-Large-Language-Models	MLLMs	yfzhang114
awesome-foundation-and-multimodal-models	Foundation + multimodal	SkalskiP
Awesome-Multimodality	General multimodality	Yutong-Zhou-cv
Awesome-Unified-Multimodal	Unified models	Purshow
Awesome-Unified-Multimodal	Unified models	AIDC-AI

7.2 Slides & Survey Papers

Type	Resource	Notes
Slides	Native LMM Slides	Ziwei Liu (NTU); concise framing for native multimodal models
Survey	A Survey on Multimodal Large Language Models	Broad survey of MLLM architectures, data, and evaluation
Report	The Dawn of LMMs: Preliminary Explorations with GPT-4V	Early capability analysis around GPT-4V
Survey	Multimodal Foundation Models: From Specialists to General-Purpose Assistants	Broader foundation-model view across multimodal systems

7.3 Code Repositories & Tools

Tool	Description	Link
TorchUMM	Unified evaluation, analysis and post-training toolkit for heterogeneous unified multimodal model architectures, tasks and datasets	Paper Code
LMMs-Eval	Unified evaluation harness for multimodal models	Code
LAVIS	Library for Language-Vision Intelligence (Salesforce)	Code
OpenFlamingo	Open reproduction of DeepMind Flamingo	Code
xtuner	Efficient fine-tuning for multimodal LLMs	Code
LLaMA-Factory	Multimodal instruction tuning framework	Code
MMEngine	Foundation for perception research (OpenMMLab)	Code
DeepSpeed-VisualChat	Scalable multimodal chat training	Code

7.4 Benchmarks & Evaluation

Evaluation artifacts are catalogued separately from model architectures. Dataset coverage, agentic baselines and model-native input/output capabilities are distinct. The companion website exposes these rows in its Evaluation collection.

Unified Understanding & Generation

Paper	Venue	Links	Type	Notes	Task
Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation	ICML 2026	Paper Code Project	Benchmark	First public: 2026-06-25. Provides joint and decoupled tracks for internal consistency, understanding-guided generation, generation-guided understanding and mutual enhancement; 2,169 task samples plus Unison-Judge. As of 2026-10-19: code, benchmark and judge public.	joint understanding/generation, synergy evaluation
UniG2U-Bench: Do Unified Models Advance Multimodal Understanding?	arXiv 2026-03-03	Paper	Benchmark	First public: 2026-03-03. Seven regimes and 30 subtasks test when visual generation benefits understanding, including implicit/explicit transformations and multi-step reasoning. Improvements are task-dependent; generating before answering is not uniformly beneficial. As of 2026-10-19: paper public; code/data release not confirmed by inspected sources.	generation-to-understanding evaluation
UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark	arXiv 2025-12-19	Paper Project	Benchmark	First public: 2025-12-19. Assesses understanding, generation and editing in one process, with decoupled tracks for diagnosis; covers 13 domains and over 200 concepts. As of 2026-10-19: paper and project public; resource downloads require checking the project.	understanding, generation, editing evaluation
MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models	ICLR 2026	Paper Proceedings Project	Benchmark	First public: 2025-04-04. Standardizes understanding/generation tasks drawn from 12 datasets and introduces five mixed-modality tasks for joint reasoning. Also called MME-U in the proceedings. As of 2026-10-19: official project links evaluation code and data.	understanding, generation, mixed-modality reasoning evaluation

Interleaved & Any-to-Any Evaluation

Paper	Venue	Links	Type	Notes	Task
LongMIG-Bench: Long-Horizon Interleaved Generation Evaluation	arXiv 2026-10-12	Paper	Benchmark	First public: 2026-10-12. Benchmark introduced with Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation; evaluates text-to-multi-image and image-to-multi-image trajectories, character/scene consistency and long-range continuity. As of 2026-10-19: paper public; standalone benchmark download not confirmed.	long-horizon interleaved generation evaluation
IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation	arXiv 2026-06-08	Paper	Benchmark	First public: 2026-06-08. Multi-turn interleaved image-text evaluation with static spatial, temporal causal and hybrid tasks; 3,113 samples and 12,034 turns expose understanding/generation errors and exposure bias. As of 2026-10-19: paper public; code/data release not confirmed by inspected sources.	multi-turn understanding/generation, exposure-bias evaluation
UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark	CVPR 2026	Paper Project Code Data	Benchmark	First public: 2026-03-05. Contains 31K instances across 30 domains and seven modalities; measures semantic/generation quality, response structure and interleaved coherence. UniMA is its agentic baseline, not evidence of a single backbone generating every modality. CVPR venue is author-reported. As of 2026-10-19: code and CC BY 4.0 data public; official repository records data release 2026-04-18 and evaluation scripts 2026-05-21.	any-to-any interleaved multimodal evaluation

General Multimodal & Robustness

Paper	Venue	Links	Type	Notes	Task
FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models	NeurIPS 2026 (author-reported); arXiv 2026-10-15	Paper	Evaluation method	PaRMA audits pathway-conditioned robustness using pixel perturbations and rendered-text insertion across 18 open-weight VLMs; FusionAudit instantiates diagnostic-guided attacks. An auditing method, not a foundation model; native categories and conclusions follow the authors’ tested taxonomy.	robustness auditing, pixel attacks, rendered-text attacks
OmniGAIA: Towards Native Omni-Modal AI Agents	arXiv 2026-02-26	Paper Code	Benchmark	First public: 2026-02-26. Benchmark for omni-modal agents with multi-hop queries across video, audio and images; includes the separate OmniAtlas tool-using agent baseline. This is an evaluation artifact, not a visual-adapter category.	omni-modal agent evaluation
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning	NeurIPS 2021	Paper	Benchmark	Benchmark suite for multimodal representation learning and cross-modal robustness across tasks and modalities; retained as a general evaluation reference rather than a model architecture.	multimodal representation evaluation

Image Generation Evaluation

Paper	Venue	Links	Type	Notes	Task
ImageBench	Live benchmark	Site Methodology	Benchmark	Live text-to-image evaluation using VLM judges, with generated images published for inspection. Ranking/model counts change over time; consult the linked methodology and current results. This evaluates image generation and does not establish U+G membership.	text-to-image evaluation

Back to Top

How to Contribute

In this section: Validation Rules · Entry Format

We welcome contributions! Please follow these guidelines:

Validation Rules

For NMM submissions:

Record architectural integration and multimodal training history separately
Identify pretrained backbones, encoders, tokenizers, and the trainable parameters at each stage
Qualify architecture-native or training-native entries; claim from-scratch training only with supporting evidence
Assign fusion depth only with architectural evidence. Use Encoder-Based (§5.3) when the input architecture is established but strict fusion depth is unknown; use Training-Native (§5.5) when only the training route is established, leaving fusion unspecified

For UMM submissions:

Confirm the model handles both image understanding AND image generation
Note whether pretrained components are used (annotate accordingly)

For MLLM submissions:

Identify the pretrained vision and language components and the visual-to-language interface for adapter-based classification
Use Interface Unspecified only as a provisional understanding-model reference when those details are unavailable; do not infer initialization or native training

For evaluation submissions:

Use §7.4 and state whether the artifact is a benchmark or an evaluation method
Describe evaluated tasks separately from any model or agent baseline
Distinguish paper, code and data availability, licenses and dates

Entry Format

Match the column headers of the destination table. For a typical five-column model table:

| Paper | Venue | Links | Notes | Task |
|---|---|---|---|---|
| <!-- entry:REPLACE_WITH_UNIQUE_ID --> Model Name | arXiv 2026 | [Paper](https://arxiv.org/abs/PAPER_ID) [Code](https://github.com/ORG/REPO) [HF](https://huggingface.co/ORG/MODEL) | Model size; architecture and training qualifications | visual understanding |

For a six-column NMM table:

| Model | Paper | Links | Training Scale | Notes | Task |
|---|---|---|---|---|---|
| <!-- entry:REPLACE_WITH_UNIQUE_ID --> Model Name | arXiv 2026 | [Paper](https://arxiv.org/abs/PAPER_ID) [Code](https://github.com/ORG/REPO) | 7B | Architecture, initialization and trainable components | visual understanding, image generation |

Every catalog row has an immutable hidden entry ID in its first cell. For a new row, replace REPLACE_WITH_UNIQUE_ID with a fresh value generated by:

node -e "console.log(require('node:crypto').randomUUID())"

Keep the existing ID when correcting a title, venue, link, or category. A cross-listed row gets its own ID; copied templates must receive a new one. Missing or duplicated IDs fail validation. When merging duplicate rows or removing an incorrect cross-listing, add an explicit old-ID → surviving-ID alias in data/entry-identities.json so existing saved entries and detail URLs remain usable.

Submit a PR with:

The paper/model entry in the correct section
A one-line justification for the chosen category
Links to paper, code, and/or weights

Back to Top

Citation

If this list is useful in your research, please consider citing:

@misc{awesome-multimodal-modeling-2026,
  title     = {Awesome Multimodal Modeling: From Traditional to Native & Unified},
  author    = {{OpenEnvision Lab}},
  year      = {2026},
  url       = {https://github.com/OpenEnvision/Awesome-Multimodal-Modeling},
  note      = {GitHub repository}
}

Back to Top

⭐ Star Growth of Awesome-Multimodal-Modeling

License

This list is released under the CC0 1.0 Universal license.

Maintained by the community for the multimodal research community.

Back to Top

Companion Website

The companion research library is designed for openenvision.github.io/Awesome-Multimodal-Modeling/, under the OpenEnvision organization site.

📄 Survey paper — Coming soon. A survey of multimodal foundation models is in preparation.
💻 GitHub repository. Explore the list, suggest entries, and contribute.

The website derives its catalog and category memberships from this README. It supports search, nested category filters, year and resource filters, sorting, pagination, entry details, shareable filter URLs, and a browser-local reading list. The dedicated Evaluation collection derives from §7.4; other resource tables retain their README format. Counts represent entries, including cross-listed models, analysis papers, and explicitly unverified boundary references, rather than unique or verified models.

Files ready to publish

index.html lives in the repository root, beside README.md. It is a complete static page, with its styles, scripts, icons, diagrams, and catalog in assets/site/. The legacy website/index.html entry redirects to the root page; edit the root index.html and assets/site/ when maintaining the website.

Awesome-Multimodal-Modeling/
├── index.html              # Website entry point, ready to serve
├── README.md               # Source catalog and project documentation
├── .nojekyll               # Serve static files directly on GitHub Pages
├── robots.txt
├── sitemap.xml
├── assets/
│   ├── site/               # Website styles, scripts, icons, diagrams, and catalog.json
│   └── ...                 # Existing README illustrations
├── scripts/                # Catalog synchronization and local preview tools
└── .github/workflows/pages.yml

Publish index.html together with assets/; keep their relative paths intact. No server, package installation, or API key is required to host the prepared files.

Publish from the repository root

Commit the root index.html, README.md, assets/, .nojekyll, robots.txt, and sitemap.xml to main.
Open Settings → Pages → Build and deployment.
Choose Deploy from a branch, select main and /(root), then save.

The site will be available at openenvision.github.io/Awesome-Multimodal-Modeling/. The custom workflow detects branch publishing and skips its own deployment.

Keep the website synchronized

After editing the model list, run:

npm test
npm run build

Node.js 22 or later is sufficient. The build refreshes the generated catalog blocks inside the root index.html, writes assets/site/catalog.json, and exports a clean copy to dist/. Commit the refreshed root files when publishing from a branch. Page content outside the marked catalog blocks can be edited directly in the root index.html.

For automatic catalog synchronization on every push, choose GitHub Actions as the Pages source and keep the included workflow. It checks pull requests, rebuilds the catalog from README changes, and deploys successful main builds. A first deployment can also be started with Run workflow. The generated dist/ directory is only used as the Actions deployment artifact and is excluded from version control.

See GitHub's publishing-source documentation.

Local preview

Build the prepared files before starting production preview:

npm run build
npm run preview

Open http://127.0.0.1:4173/Awesome-Multimodal-Modeling/. This serves the prepared files directly from the repository root, including the .nojekyll marker generated by the build. Use npm run dev for an in-memory preview that watches README.md, index.html, and assets/; reload the browser after edits.

The project uses relative asset URLs. It needs no custom domain, CNAME file, or change to the OpenEnvision organization's main website.

The brand mark comes from the OpenEnvision site. The GitHub icon comes from GitHub Octicons; its MIT license is included in assets/site/OCTICONS-LICENSE.txt. Model membership, release annotations, and availability qualifications remain governed by the README.

Name		Name	Last commit message	Last commit date
assets		assets
data		data
dist		dist
scripts		scripts
website		website
.DS_Store		.DS_Store
README.md		README.md
index.html		index.html
package.json		package.json
robots.txt		robots.txt
sitemap.xml		sitemap.xml

Latest commit

History

Folders and files

Repository files navigation

Awesome Multimodal Modeling

📢 News

Table of Contents

About This List

At a Glance

Curation Principles

1. Introduction & Definitions

1.1 Multimodal Model Evolution Stages

Traditional Multimodal Models

Multimodal Large Language Models (MLLMs)

Unified Multimodal Models (UMMs)

Native Multimodal Models (NMMs)

NMM — Early Fusion

NMM — Late Fusion

1.2 Scope & Taxonomy

1.3 Architecture Diagrams

2. Traditional Multimodal Models

2.1 Multimodal Representations & Alignment

Multimodal Representations

Multimodal Fusion

Multimodal Alignment

2.2 Multimodal Pretraining

3. Multimodal Large Language Models (MLLMs)

3.1 Taxonomy Based on Vision Adapter

MLP/Others Projector

Q-Former

Cross-Attention

Hybrid Adaptor

Interface Unspecified

3.2 Omni MLLMs

4. Unified Multimodal Models (UMMs)

4.1 Taxonomy by Generation Paradigm

Diffusion-Based UMMs

Autoregressive (AR) UMMs

Pixel Encoding

Semantic Encoding

Learnable Query Encoding

Hybrid Encoding (Pseudo)

Hybrid Encoding (Joint)

Hybrid (AR + Diffusion) UMMs

Pixel Encoding

Hybrid Encoding

4.2 Any-to-Any / Omni UMMs

4.3 Related Generators (U+G Unverified)

5. Native Multimodal Models (NMMs)

5.1 Design Analyses & Scaling Laws

5.2 Early Fusion NMMs

5.3 Encoder-Based NMMs

5.4 Any-to-Any / Omni NMMs

5.5 Training-Native Models

6. Closed-Source Multimodal Models

Year 2026

Year 2025

Year 2024

Year 2023

7. Resources

7.1 Related Awesome Lists

7.2 Slides & Survey Papers

7.3 Code Repositories & Tools

7.4 Benchmarks & Evaluation

Unified Understanding & Generation

Interleaved & Any-to-Any Evaluation

General Multimodal & Robustness

Image Generation Evaluation

How to Contribute

Validation Rules

Entry Format

Citation

⭐ Star Growth of Awesome-Multimodal-Modeling

License

Companion Website

Files ready to publish

Publish from the repository root

Keep the website synchronized

Local preview

About