Foundation Models for Robot Manipulation Policy Learning

Robot manipulation has spent decades solving the same problem over and over, one task at a time, because the field lacked a way to make what a robot learns in one setting carry over to the next. Imitation learning changed the economics of building a policy: instead of hand-coding perception and control for each task, engineers could show a robot arm what to do and let it learn from demonstration, which opened the door to dexterous, hard-to-model behaviors that classical pipelines never handled well. But the gains had a ceiling. Policies trained this way stayed fragile against anything cosmetic: swap the color of an object, change the background, add clutter to the table, and a policy could fail a task it had otherwise mastered. Nothing about the task itself had changed. The policy's grip on what mattered versus what didn't failed, and a model trained on narrow, task-specific data has no way to learn that distinction. The pipelines encoded neither a broad visual understanding of the world nor any grasp of language, so each new object, each new room, each new phrasing of an instruction meant collecting fresh data and retraining from close to scratch. What manipulation research lacked wasn't more demonstrations or better tuning. It lacked a representation that could travel.

What foundation models bring to manipulation

The bet driving the current wave of robotics research is that a model pretrained on internet-scale vision and language data already carries the priors that task-specific pipelines were missing, that a comparatively small amount of robot-specific training can adapt those priors into motor commands, and that this is a different kind of claim than "more data helps. It says the representations a model builds from looking at billions of images and reading enormous amounts of text already contain most of what a robot needs to recognize objects, parse instructions, and reason about scenes, so the robot-specific training only has to teach the last mile: turning that understanding into actions.

Google's RT-1 and RT-2 supplied the first real evidence for the idea, showing that combining larger robot datasets with pretrained vision-language representations produced meaningful gains in manipulation performance across a range of tasks. OpenVLA, trained on the Open X-Embodiment large-scale robot manipulation dataset, became a widely used open-source baseline, showing the approach was reproducible outside Google. What made the transfer work wasn't just scale. Pretrained backbones hand a policy four kinds of signal that narrow robot datasets can't easily generate on their own: visual representations built from far more variety than any robot dataset contains, latent codes describing what an action is doing, signals about how much progress a task has made, and priors about how objects can be grasped and how contact behaves. VLA publications grew faster year-over-year than any other robotics sub-field, per one literature analysis, making it a research area that expanded faster than almost anything else in robotics.

The three architectural paths now competing to extend that foundation

Once the field accepted that pretrained backbones were worth building on, the question was which prior to build around, and by mid-2026, three distinct architectural lineages had formed, each staking out a different answer.

As of mid-2026, the first of three converging but distinct architectural paths stays closest to the original VLA formula, attaching an action head to a pretrained vision-language backbone. The live debate inside this camp is about how the action head should generate motor commands, through autoregressive token prediction or diffusion-based decoding, given that motor commands are continuous and often multimodal in a way that text tokens are not. HybridVLA, presented at ICLR 2026, hedges by combining both decoding strategies in a single backbone, while CogACT commits to a diffusion-only action head conditioned on the vision-language model's output, treating the two techniques as complementary tools rather than rivals. ChainVLA takes a different angle within the same camp, carrying task progress and unfinished motion plans across successive queries, and its strong results on RMBench and LIBERO suggest that how a model carries context between steps matters as much as how large its backbone is. Qwen-RobotManip pushes on a related question borrowed from language modeling: whether the alignment techniques that made large language models more useful during post-training can do the same for manipulation foundation models at scale.

The second path, world-action models, addresses a different weakness. Rather than reacting to a single frame, WAMs adapt large video-generation backbones so the policy predicts future observations jointly with the actions that produce them, giving it a kind of foresight that purely reactive VLAs don't have. The motivating argument, drawn from a recent world-model survey, is that reactive policies struggle specifically with long tasks: they have trouble assigning credit across long time horizons and their small errors compound as a task drags on, problems that explicit prediction of what happens next is designed to catch. Because these models start from video-generation pretraining, they inherit strong priors about how scenes evolve over time and become more data-efficient by learning to predict actions and future frames together. FLEX-π, out of the University of Washington and AI2, extends the paradigm with a finding that's almost incidental in how useful it turns out to be: the frozen video-generation VAE it builds on already encodes 3D pointmaps nearly losslessly, without any pointmap-specific training. The model can be supervised on 3D geometry and object-centric semantic features without adding new sensors, new pretraining, or any hit to inference speed. FLEX-π also runs on a Mixture-of-Transformers backbone with per-stream dropout and cross-modality forcing, so one trained checkpoint can operate in a fast action-only mode or a fuller joint-generation mode depending on what a deployment needs, letting practitioners pick their own trade-off between speed and performance after training is done. NVIDIA's Cosmos, released across 2025 and 2026, works this same territory from another direction, treating video generation itself as a simulator for training and evaluating policies and changing what it costs to generate robot data in the first place.

VLAs and WAMs mostly operate on 2D image frames, leaving 3D geometry implicit, even though contact-rich manipulation, the physical business of grasping, pushing, and placing, depends on geometry that a flat image doesn't fully capture. The Geometric Action Model, or GAM, takes a pretrained geometric foundation model and repurposes it as a single shared substrate for perception, prediction, and action decoding. It does this by splitting the geometric backbone at an intermediate layer: the shallow layers handle observation encoding, a causal predictor inserted at the split point forecasts future latent tokens conditioned on language, proprioception, and action history, and the remaining layers decode those tokens back into both future geometry and action commands. Across simulation and real-robot benchmarks, GAM is reported to beat foundation-model-scale baselines on accuracy, robustness, speed, and model size at the same time. A different group, at MBZUAI, works the geometric angle at a smaller scale: rather than building one large geometric backbone, their framework coordinates pretrained vision models, SAM3 for segmentation and Depth Anything 3 for monocular depth, with a small policy model through semantic repainting, constructing task-relevant visual observations that strip out appearance variation while keeping the underlying geometry intact, all without touching the policy's architecture.

Three converging but distinct architectural paths are active as of mid-2026, each addressing a different limitation of the base VLA idea. Each is a bet on which prior, semantic, temporal, or geometric, does the most work for generalization, and each is actively being extended by multiple labs at once.

Data infrastructure as the constraint on scaling these paths

Architecture is not the constraint that will decide which of these approaches actually reaches deployment at scale. A 2026 survey argues the real bottleneck is data infrastructure itself, and that future progress depends less on backbone design than on building high-fidelity data engines alongside structured ways to evaluate them, calling for data infrastructure to be treated as a first-class research problem rather than plumbing.

The trouble is that combining data across different robot embodiments often backfires: a 2026 study on rethinking VLA scaling found that pooling datasets from other robots can make a model perform worse on the specific robot it's meant to run on, a pattern known as negative transfer. That finding cuts against the instinct to just pool everything and let scale sort it out. Human video looks like an obvious way around the problem, since it's cheap and there's an enormous amount of it available, but human hands and bodies differ enough from robot manipulators, in appearance and in how they move, that a policy can't just watch people and directly copy what it sees. The H2R data engine tackles that mismatch head-on: it detects 3D hand poses in human video, retargets the motion onto robot kinematics, and composites robot arms into the footage using segmentation and inpainting, and using it to pretrain visual encoders improves manipulation success in both simulation and on real robots. LAP, or Language-Action Pre-Training, approaches the same underlying problem, reusing a single policy across different robot bodies, from a different direction, using language-action pretraining to get zero-shot transfer across embodiments without finetuning for each one.

All of this has an access consequence: building datasets at the scale foundation models need takes resources that only well-funded labs currently have, putting smaller academic groups and startups at a structural disadvantage before they've written a line of model code. OpenVLA and Xiaomi-Robotics-0 stand out as open-source efforts to lower that barrier, though several VLA papers report a real cost to the strategy: when the action-learning objective pulls too far from the model's original pretraining signal, the pretrained vision-language capabilities it started with start to degrade.

The three failure modes that architectural scaling alone has not resolved

Three failure modes appear across all three architectural paths, and none of them yields to simply adding more data or building a bigger backbone.

The first is long-horizon task failure. Reactive VLA policies have trouble reasoning over long sequences of steps, struggle to assign credit correctly across a long horizon, and accumulate small errors until they compound into task failure. MPVI documents this directly: long-horizon failures persist even after fine-tuning on large teleoperated datasets, which points to a limit that more data doesn't fix. The Latent Reasoning VLA, presented at ICML 2026, responds architecturally rather than through data, borrowing the chain-of-thought mechanics of large language models to give a policy explicit latent thinking and prediction steps before it commits to an action.

The second is the embodiment gap. A 2026 paper, "The Embodiment Gap in Robot Foundation Models," formalizes the structural mismatch between human video and robot morphology as something data scale cannot resolve on its own, arguing instead for domain-invariant representations, explicit correspondence mappings between human and robot motion, and stronger data augmentation. Several current frameworks have taken up cross-embodiment transfer as a central design goal in response, learning skill or world-model representations that abstract away the details specific to any one robot body.

The third is missing physical priors. A June 2026 position paper, "Robots Need More Than VLAs and World Models," argues that even combining VLAs with world models is insufficient without structured physical reasoning layers, because the models lack grounded understanding of contact, force, and physical constraint. Sim-to-real transfer compounds all three failure modes at once, because any approach that tries to pre-train reactions to every condition a robot might encounter runs into an open-ended space of conditions that training can never fully cover.

Where the converging evidence points next

Setting the three architectural paths, the data constraints, and the three failure modes side by side reveals a single conclusion: no one prior, semantic, temporal, or geometric, is enough by itself to produce manipulation that generalizes, and the systems making the most progress combine priors without letting latency or data requirements blow up. FLEX-π offers a concrete demonstration of the principle. Adding geometric and semantic supervision to a WAM backbone during training improves performance even when those extra modalities are dropped at inference. Richer training signals pay off even when the deployed model has to run leaner than it trained.

A related structural shift is underway in how world models relate to embodiment. Methods that separate an action-free video world model from an embodiment-specific inverse-dynamics model, connected through Jacobian-based translation, achieve zero-shot policies across different robot bodies. The video model no longer needs any robot data at all during its own pretraining. That is a genuine decoupling, not just an efficiency trick, and it changes what counts as the expensive part of the pipeline. The 2026 survey's call to treat data infrastructure as a first-class problem is already showing up in the work itself rather than staying an abstract recommendation: the H2R data engine, LAP, and the MBZUAI semantic-repainting framework all represent perception and data pipelines designed together, not model architectures bolted onto whatever data happened to be available.

The position paper "Robots Need More Than VLAs and World Models" points toward what comes next: structured physical reasoning layers that sit alongside the foundation-model backbone rather than folded inside it, a modular design rather than one monolithic model trying to do everything. For anyone building on top of this research, that argues for a specific set of decisions rather than a bet on a single winning architecture: matching backbone type to the task's horizon and latency budget, checking whether geometric priors can come from off-the-shelf depth models instead of new sensors, auditing training data for negative transfer before pooling it across embodiments, and weighing whether cross-embodiment transfer is worth the architectural complexity it adds for the task at hand.