Imitation Learning Data Collection Pipelines for Manipulation Tasks

Large language models scaled by reading text that already existed. Robotic manipulation has no equivalent archive to draw on: every trajectory a robot needs to learn from has to be physically performed, recorded, and structured before it teaches the policy anything. That constraint makes the collection pipeline, not the model architecture downstream of it, the central engineering problem in imitation learning for manipulation.

Two failure modes follow directly from learning off a fixed set of recorded demonstrations, and neither is an incidental bug to patch later. The first is covariate shift: once a robot's actions drift even slightly from the trajectories it was trained on, it reaches states the policy never saw, and small errors compound into task failure. The second is causal confusion, where a policy locks onto whatever feature happened to correlate with the right action in training rather than the feature that actually caused it. Qi et al. (2026) show this in concrete terms: across six foundation policies, color consistently dominates as the shortcut cue, while verb and size are the factors the policies ground least, in a hierarchy that holds as color ≥ object ≥ spatial ≥ verb ≥ size.

Both failure modes are manageable only to the degree that the data pipeline that produced the training set anticipated them. A pipeline built without that awareness bakes assumptions about generalization into the dataset long before a single parameter gets trained. Everything that follows in a manipulation data pipeline, which demonstrations get collected, how they're recorded, how the dataset is composed, when collection happens relative to training, is a design decision with direct consequences for what the resulting policy can and cannot do.

How collection modality shapes what the data can teach

The most upstream decision in any manipulation pipeline is how the demonstrations get physically produced, and the four dominant modalities each come with a structural ceiling on what they can encode rather than a simple cost tradeoff.

VR and hardware teleoperation remains the source most able to produce high-quality demonstrations. The DROID dataset shows what coordinated teleoperation looks like at scale: tens of thousands of trajectories gathered across hundreds of scenes in dozens of buildings, covering dozens of tasks, built by a team of operators working over 12 months on a standardized Franka Panda arm with synchronized RGB stereo and wrist-camera streams. The quality this produces is hard to match by other means, but the method is slow and tied to wherever the hardware and operators physically are.

Robot-free and portable collection removes the hardware constraint instead. Chi et al. (2024) built a universal manipulation interface around handheld grippers so demonstrators don't need access to a robot. But removing the hardware barrier exposes a different one: Bai et al. note that human demonstrators still need active incentives to keep producing good data, so the bottleneck moves from hardware access to human motivation. RoboCade (Mirchandani et al.) takes that problem on directly, gamifying remote teleoperation through leaderboards, badges, progress bars, visual feedback, sound effects, and goal visualizations, built on a low-cost GELLO controller. Co-training on this gamified data improved success rates even on target tasks that had nothing to do with the game.

Egocentric and human video collection aims at a different kind of scale: passive data gathered from people simply doing things, with a camera watching. EgoMimic (Kareer et al., ICRA 2025) builds imitation learning on top of this approach. The limiting factor is morphology mismatch: human hands move differently than robot grippers, people move faster than teleoperated arms, and the video stream from a human head-mounted camera doesn't match what a robot's own egocentric camera sees.

Simulation-generated synthetic data removes the need for a physical demonstration, so data collection no longer needs to scale with the number of real-world demonstrations gathered. One system is the clearest illustration of how far this can go: real robot data totaled only a handful of hours, while a simulation engine generated 780,000 synthetic episodes. That scale was only possible once 3D Gaussian Splatting solved a rendering bottleneck that had sabotaged sim-to-real transfer for years. Mesh rasterization rendered fast but looked poor enough that policies trained on it transferred badly; NeRF rendered convincingly but too slowly to generate data at scale; 3DGS does both at once, so it's what unlocked synthetic datasets of this size.

Contact-rich manipulation is where simulation still struggles. Replacing expensive real demonstrations with synthetic ones is attractive precisely because it's cheap, but the sim-to-real gap is worst exactly where contact dynamics, friction, deformation, force feedback, matter most to the task. FreeTacMan (Wu et al., ICRA 2026) and ForceMimic (Liu et al.) respond to this directly by building force-centric sensing into robot-free collection hardware rather than trying to simulate contact physics convincingly. Each modality, in other words, defines in advance which tasks the resulting dataset can actually teach a policy to do.

Data structure and action representation as a hidden pipeline variable

Beyond how a demonstration gets physically collected sits a second question: what exactly gets recorded during it. The schema of a trajectory, which sensors are logged, how actions are represented, how tasks are broken into pieces, determines the expressiveness of any policy trained on it before a single architectural choice gets made downstream.

DROID's sensor schema makes the stakes concrete. Each trajectory synchronizes three RGB stereo streams (two external ZED 2 cameras plus one wrist-mounted ZED Mini), depth maps, camera calibration, low-level robot state, and crowd-sourced language instructions. Every one of those modalities is a decision about what the policy will be allowed to condition its behavior on later. If depth is left out, the policy can't learn depth-dependent behavior no matter how good the model is. Force and tactile sensing are starting to get folded into collection pipelines the same way: ForceMimic introduces force-centric imitation learning specifically because force-related skills are missing from what most robot learning algorithms can currently learn. This remains an early, developing part of the field rather than a settled practice, but skipping force data at collection time forecloses an entire category of policy later.

Task decomposition carries the same weight. HD-Space (Yang et al., 2025) segments a long-horizon manipulation task, say, opening a box, picking up a cup, placing it inside, closing the box, into atomic subtasks, each with its own state and action space, rather than recording the whole sequence as one undifferentiated trajectory. If you record flat, full-sequence demonstrations instead, the dataset has to cover a much larger combinatorial space of task variation, and any model trained on it is more exposed to compounding error across the full task length. Structuring the recording around atomic subtasks gives the policy cleaner, more composable units to learn from.

Vision-Language-Action models raise the cost of getting this wrong. These models train on heterogeneous action data pulled from many different robot setups, and without a carefully designed schema, aggregating datasets from multiple sources produces action representations that contradict each other. Aggregating datasets this way makes the resulting policy perform worse than it would have on a single clean dataset, not better.

Diversity of environments and objects over trajectory count

Once the recording schema is fixed, the next pipeline decision is how to compose the dataset as a whole, and the empirical evidence here points in one direction: generalization scales with the number of distinct environments and objects represented in training data, not with the raw count of demonstrations collected.

Lin et al. (ICLR 2025 Oral) establish this directly. Past a certain threshold of demonstrations per environment or object, adding more demonstrations in that same setting barely moves performance. What keeps paying off is adding new environments and new objects. The team demonstrated this with four data collectors working a single afternoon: that effort alone produced policies with high success rates in novel environments and on unseen objects across two separate tasks. The result reframes what a collection budget should buy. Spending it on more repetitions of a task already well covered in the dataset returns little. Spending it on a new room, a new object, a new arrangement returns far more.

Most existing manipulation datasets were built inside a small number of lab settings. Policies trained on them tend to fail once deployed somewhere new. DROID was built explicitly to break that pattern: it targets an order-of-magnitude jump in scene diversity over earlier datasets by spanning 564 unique scenes across 52 buildings. Cross-embodiment datasets extend the same logic to the level of the entire field rather than a single dataset. Open X-Embodiment aggregates demonstrations from dozens of platforms across institutions on multiple continents, standardizing more than a million trajectories across hundreds of distinct manipulation skills. Qwen-RobotManip (2026) folds in nine open-source datasets totaling thousands of hours of demonstration, spanning single-arm, bimanual, dexterous, mobile, and humanoid loco-manipulation setups. A collection strategy that keeps filling its budget with repeated demonstrations of one task in one scene stops producing returns once that scene is already well covered. New collection effort belongs in new scenes and new objects, not more passes through old ones.

Instruction factor bias in the data distribution and its limits on language-conditioned policies

Diversity of scenes and objects addresses one axis of generalization. A second axis, independent of it, is what the language instructions in the dataset actually say, and a dataset can be rich in scenes and objects while still being badly imbalanced in how it describes tasks. That imbalance teaches a language-conditioned policy to rely on shortcuts instead of the instruction itself.

Qi et al. (2026) name this instruction factor bias and built two metrics to measure it. Factor Dominance Rate (FDR) captures the pairwise bias between any two instruction factors, color versus verb, object versus spatial term, and so on. Factor Dominance Hierarchy (FDH) aggregates those pairwise scores into a single global ranking. Tested across six different foundation policies, that ranking came out strikingly consistent: color ≥ object ≥ spatial ≥ verb ≥ size. Color acts as the dominant shortcut a policy reaches for first, while verb and size are the factors every architecture tested grounds the least, regardless of how the policy itself is built.

The finding points to a specific fix rather than a vague caution. A compositionally-guided data collection strategy, one that deliberately balances factor combinations rather than letting them fall out naturally from whatever tasks get recorded, achieves a 77.5% success rate transferring to entirely new environments with unseen combinations of factors. A separate bias-aware strategy takes a fixed demonstration budget and reallocates it toward the factors FDH ranks lowest, verb and size, rather than spreading it evenly or following whatever's convenient to collect. The "Scale Up Strategically" study found that approach outperforms standard baselines in both simulation and on a real robot. Shaping what the demonstrations say, in other words, can do more for the resulting policy than simply collecting a larger pile of demonstrations under the same instruction imbalance. This is a case where the collection pipeline has to encode knowledge of the model's likely failure modes before collection starts, diagnosing bias in advance and routing new data collection toward exactly the factors that diagnosis flags as weak.

Collecting interactively rather than offline

Everything above assumes demonstrations get collected once, in full, before training begins. That default, offline collection followed by batch training, has a structural weakness: it only ever records expert trajectories, so it systematically misses the states a partially trained policy will actually wander into once it starts making its own mistakes.

DAgger (Ross et al., 2011) is the foundational method here. The policy executes its own actions, an expert labels the correct action at each state the policy actually visits, and the dataset grows to cover the distribution the policy itself produces rather than only the distribution an expert would produce. The taxonomy built by Khojasteh et al. (2025) classifies DAgger under both imitation learning and reinforcement learning, reflecting how it sits between the two paradigms.

RoboCopilot (Wu et al., 2025) turns this into a working bimanual teleoperation system using HG-DAgger. A human teacher decides, in real time, when to take control away from the robot through a leader-follower setup, gathering corrective data as problems arise rather than waiting for a failure to be flagged first. That design fixes the passivity built into standard human-in-the-loop collection, where the system only gathers a correction after something has already gone wrong.

HD-Space (Yang et al., 2025), introduced earlier for its task decomposition, also functions as a structural answer to covariate shift. By collecting demonstrations separately within each atomic subtask rather than across one long undifferentiated trajectory, it anticipates the states where long-horizon tasks tend to fail rather than discovering them only after a trained policy stumbles into them.

Moving to interactive collection changes what the pipeline needs operationally. It requires hardware built for fast, seamless handoff between robot and human control, leader-follower rigs rather than simple teleoperation stations, and operators trained to recognize and correct failures as they happen rather than to simply demonstrate a task from scratch. It also changes the shape of the collection effort itself: instead of a single batch of data gathered once and handed off to training, collection becomes an ongoing loop tied directly to how the policy is currently performing.

Automated and simulation-assisted scaling strategies

Automated and simulation-based generation can multiply a collection budget by orders of magnitude, but how reliably that multiplication translates into a usable policy depends heavily on the task and on how faithful the underlying simulation is.

DexMimicGen (ICRA 2025) provides automated data generation purpose-built for bimanual dexterous manipulation, extending imitation learning into a task category, two coordinated hands, that is notoriously hard to collect by hand at scale. One system shows what this looks like at the extreme end: a system trained on only a small number of hours of real robot data, supplemented by 780,000 synthetic episodes generated in a simulation engine. That scale of synthetic generation was only reliable once 3D Gaussian Splatting closed the rendering gap discussed earlier: it's fast enough to generate data at scale and visually faithful enough for the resulting policy to transfer to a real robot.

Where this breaks down is exactly where contact dynamics dominate the task. Friction, deformation, and force feedback are hard to simulate with enough fidelity for a policy trained on synthetic contact-rich trajectories to transfer reliably to a real gripper touching a real object. FreeTacMan and ForceMimic exist because simulation alone hasn't closed that gap, so they capture force data directly from real-world robot-free hardware instead. Scale, in other words, extends collection capacity substantially, but it does not by itself resolve the question of whether the data it produces is trustworthy enough to train on, which is the question any pipeline still has to answer before it decides how much to lean on simulation and how much to keep in the real world.