Multimodality & The Great Consolidation:
How AI Models Are Merging Into One

For a decade, AI was built modality-by-modality: a language model here, an image classifier there, a separate speech recognizer, a bolted-on captioning module stitched together with glue code. That era is ending. The frontier labs — OpenAI, Google DeepMind, Meta, and a widening circle of Chinese labs — are converging on a single bet: one model, trained jointly across text, image, audio, video, and increasingly physical action, will out-perform any collection of specialist models stitched together. This is "the Great Consolidation" — and it's rewriting how AI products get built, priced, and researched. This article explains why it's happening, how it actually works under the hood, who's ahead, and where the unresolved research problems still are.

🕰️ From Bolted-On to Born-Multimodal

The path from "caption this image with a separate model" to "one model natively perceives and generates everything" took just over a decade. Here's every landmark moment, in order:

📊 Why Consolidation, Why Now

Four forces are pulling the industry toward consolidation simultaneously:

🧩 A Taxonomy of Multimodality

"Multimodal" gets used loosely to describe wildly different architectures. Understanding the taxonomy is the key to reading model announcements critically — a model that can merely "look at an image" and a model that was jointly pretrained on image-audio-video-text from token one are fundamentally different engineering achievements, even if both check the "multimodal" marketing box.

By Capability Direction

By Fusion Architecture

How Unified Tokenization Actually Works

The hardest engineering problem in early fusion is representing fundamentally different signal types — discrete word pieces, continuous pixel grids, continuous audio waveforms — in one shared vocabulary a single Transformer can process uniformly.

Each modality gets its own tokenizer (a VQ-VAE codebook for images, a neural audio codec for speech, spatio-temporal patch embeddings for video) — but critically, all of them map into one shared discrete or continuous vocabulary that the same Transformer layers process identically, regardless of what modality a given token originally came from. This is what "early fusion" means in practice: the model doesn't have an "image brain" and a "text brain" — it has one brain that has learned to read many alphabets.

🗂️ The Model Landscape

Here's how the frontier models actually stack up on consolidation — what's genuinely unified versus what's still composed of specialist parts under a shared brand:

⚙️ How Consolidated Models Are Actually Built

Building a genuinely unified model requires solving four distinct engineering problems that don't exist when training single-modality models.

1. Balanced Multimodal Pretraining

Naively mixing training data (e.g., 90% text, 8% image, 2% audio) causes the model to become "text-dominant" — it under-learns rarer modalities. Frontier labs use careful data-mixture curricula, sometimes up-weighting rare modalities early in training and annealing toward the target ratio, plus modality-balanced loss weighting so gradient updates aren't swamped by whichever modality has the most tokens.

2. Mixture-of-Modality-Experts (MoE Routing)

Rather than every token flowing through identical dense feedforward layers, some architectures (Llama 4's MoE design, rumored components of Gemini) route tokens through different expert sub-networks depending on the input modality — letting the model specialize computation per modality while still sharing the attention mechanism and overall context. This gets some of early fusion's benefits with better compute efficiency than a fully dense shared backbone.

3. Generative Decoding Across Modalities

Understanding is architecturally easier than generation — feeding an image in for the model to reason about doesn't require the model to invert its representation back into pixels. Generation requires the backbone to drive a modality-specific decoder (a diffusion head for images, a codec decoder for audio) while remaining coherent with everything the model has reasoned about in-context. This is why "any-to-any" generation lagged "any-to-text" understanding by roughly two years across the industry.

4. Preventing Catastrophic Interference

Jointly training on many modalities risks each new modality degrading performance on the others — a phenomenon directly analogous to catastrophic forgetting in continual learning. Labs mitigate this with careful learning-rate schedules per modality, replay of earlier-modality data throughout training, and architectural isolation (like MoE) that gives each modality dedicated capacity even within a shared backbone.

🏢 Major Players & Their Consolidation Strategy

🚀 Startups & Funding in Multimodal Infrastructure

Unlike the image/video and voice categories, few startups compete to build frontier omni-models directly — that requires frontier-lab-scale compute. Instead, the multimodal startup opportunity concentrates in infrastructure, evaluation, and vertical application layers built on top of the frontier omni-models.

Investor thesis patterns: Twelve Labs and Reka target the "multimodal retrieval/search" layer — indexing video and audio the way text search was indexed a decade ago. Physical Intelligence and Skild AI represent the embodied/robotics extension of consolidation, both backed heavily by frontier-lab-adjacent capital (OpenAI directly invested in Physical Intelligence) — a signal that labs see robotics as the next modality to fold in, not a separate category. Adept's acqui-hire by Amazon shows how quickly "multimodal agent" startups get absorbed once frontier labs ship equivalent native capability.

📐 How Consolidation Is Measured

Benchmarking a genuinely unified model is harder than benchmarking a specialist — you need to test not just per-modality competence, but whether cross-modal reasoning is actually happening, or whether the model is secretly still solving each modality independently.

🔨 How to Build a Multimodal Foundation Model

Full frontier-scale omni-model pretraining is only realistic for a handful of labs with billions in compute. But understanding the pipeline — and knowing which steps are viable to replicate at startup scale via fine-tuning — is essential for anyone building multimodal products.

💰 Monetization: How Consolidation Changes Pricing

The strategic shift: consolidation collapses vendor lock-in risk from "many single-purpose vendors" to "one or two omni-model providers" — which is precisely why the frontier labs are racing to consolidate first. Whoever ships the best any-to-any model captures the entire multimodal spend that used to be split six ways across specialist vendors. This is also why open-weight omni models (Qwen2.5-Omni, Moshi) matter disproportionately: they're the only check against full vendor lock-in at the consolidated layer.

🔬 Open Research Frontiers

🌐 Applications Unlocked by Consolidation

⚖️ The Verdict: Is Full Consolidation Inevitable?