Humans have spent decades preparing training data for language models without knowing it. Every Wikipedia edit, Reddit argument, recipe blog, and unnecessarily long LinkedIn post contributed to the cause. Even your embarrassing fan fiction may have found a higher purpose.
We have been considerably less generous to robots. You’ve probably picked up thousands of objects in your life, but you haven’t recorded the grip pressure, wrist movement, or tiny corrections that stopped them from slipping. You just picked them up. Very inconsiderate of you, from a research perspective. That missing experience is a huge problem for physical AI. The internet can show a robot what people do. Teaching it how to do those things with its own body requires data that is much harder to collect and much more expensive to get wrong.
So where does that data come from? And can simulation, synthetic experience, and robots practising on their own ever make up the difference?
Why Robots Can’t Learn From the Internet Alone
Large language models trained on trillions of words that humans wrote for their own reasons. Nobody commissioned this dataset. It just piled up over thirty years. Now ask: where's the internet's archive of "motor torque at millisecond 14 while gripping a mug"? It doesn't exist. Nobody uploads their joint angles. To train a robot you need triples of what it saw, what it did, and what happened next, and every one of them has to be produced by an actual machine, in actual physical time, at the speed of actual physics.
There's a second problem: text is forgiving. A language model that picks a slightly wrong word gives you an awkward sentence. A robot that picks a slightly wrong action gives you a shattered glass.
Real-World Data: Slow, Pricey, and Weirdly Fragile
The standard way to collect robot data is teleoperation: a human straps on VR gear or grabs a controller and puppeteers the robot through a task, again and again. Hours of tedious human labor yield minutes of usable data. Google's RT-1 project, for instance, spent roughly 17 months and 13 robots to gather about 130,000 demonstrations. That's a heroic effort, and it's still a rounding error next to the trillions of tokens an LLM sees.
And quality is a headache of its own:
Humans are inconsistent. Operator #3 grasps the cup from the left after lunch and from the right before it. The model has to make sense of both.
Bodies differ. Data from one arm doesn't transfer neatly to another with different joints, grippers, or cameras. This is the "cross-embodiment" problem, and it's why pooling datasets is so hard. Efforts like Open X-Embodiment combined data from 20+ robot types, which helped, but it's still modest next to text.
Hardware wears down. Motors drift, cameras shift, and the lab's lighting changes when someone opens the blinds.
Resets are brutal. After every attempt, someone has to put the blocks back. Robot learning is, in large part, an unpaid-intern problem.
Failures go unrecorded. The most educational data, what went wrong and why, is the data people tend to throw away.
Simulation: Infinite Practice, One Tiny Catch
So why not let robots practice in a simulator? You can run thousands of virtual robots in parallel, faster than real time, with zero broken hardware and zero resets. It sounds like the cheat code.
The catch is the sim-to-real gap. Simulators approximate physics, and approximations leak: friction is off, contacts are too clean, sensors are suspiciously noiseless, and anything squishy, like cloth, food, or cables, is a nightmare to model. A policy that moves like a gymnast in simulation steps into reality and behaves like a newborn on a wet floor. It's like learning to swim from a video game and then being surprised by water.
The most popular patch is domain randomization: if you can't make the simulator accurate, make it chaotic. Randomize friction, mass, lighting, textures, camera angles, and delays. The robot learns that the world is unreliable, builds a policy that survives the whole mess, and reality becomes just one more variation. OpenAI's robot hand that solved a Rubik's Cube was trained entirely this way, and legged robots now learn rugged walking in simulation before trotting off across real hills.
But randomization has limits. Randomize too much and the policy turns timid and mediocre, hedging against every possible universe. You can also only randomize what you thought to model; the thing you forgot about is exactly what will bite you. And that cube-solving hand, impressive as it was, still dropped the cube fairly often on the hardest scrambles. Rigid bodies like walking legs are manageable. The squishy, contact-heavy world of hands and household objects is where simulation still struggles.
Self-Play, Synthetic Data, and the LLM
Self-play is a good idea. AlphaGo and its successors became superhuman by playing themselves millions of times. But Go has a luxury robotics lacks: the rules are the simulator. There's no gap between the game and the model of the game. The real world doesn't publish its rulebook, and it keeps adding house rules involving humidity.
Synthetic data is more practical. Teams now generate thousands of trajectories in simulation, then use generative models to vary scenes, objects, and appearances. Video models are even being explored as "world simulators" that imagine what happens next. It's promising, but the principle holds: synthetic data is only as good as the physics baked into it. Fake data with wrong physics teaches confident wrong habits, which is arguably worse than no habits.
So can robots learn like LLMs? Partially, and the cleverest trick is borrowing. Models like RT-2, OpenVLA, and Physical Intelligence's π0 start from vision-language models pretrained on internet-scale data. That gives the robot common sense for free: it understands that "something to hammer with" could mean a rock, without a single robot ever having seen a rock. Then a comparatively small pile of real robot data teaches the actual moving part.
Other sources are being tapped too: human videos of people cooking and cleaning, handheld gripper devices that let people collect demonstrations without a robot, and "data factories" where workers perform chores wearing cameras and sensors. The catch with plain video is that it shows what happened but not the forces or exact actions involved. It's like learning piano by watching someone's shoulders.
The long-term dream is the data flywheel: deploy robots, let them collect data from real work, improve them, deploy more. You need good-enough robots to get the data that makes robots good enough.
The Next Breakthrough Needs More Practice
There probably won’t be one magical dataset that teaches robots everything. Progress will come from combining sources: internet content to connect language with visual concepts, simulation to practise at scale, and real-world experience to discover everything the simulator conveniently forgot.
The important question is how much a robot learns from that experience. Can it apply yesterday’s lesson to a different object, a different room, or a task it hasn’t rehearsed? Collecting a million demonstrations is useful. Needing another million every time someone changes the gripper is an expensive habit.
For anyone learning robotics, this makes data collection, sensor calibration, and careful testing worth understanding alongside the models themselves. The impressive demo gets the attention; figuring out why attempt number 47 failed helps make the next one work.
We’ve spent years teaching AI using what humans put on the internet. Teaching robots will also require capturing what we’ve spent our lives doing without bothering to explain. Apparently, “just pick it up” leaves out quite a lot.
