Scaling Blind
We can't tell if the data is good. We're scaling it anyway.
Ask a robotics data company whether their egocentric dataset is good, and you’ll get a brag sheet about collection hours, broad task diversity, or magnitude of environments it spans. Ask how they know it’s good — what metric moved, what it was held out against, what predicted downstream policy performance before massive compute was spent and the hand waving begins. Nobody has an instrument. There’s no accepted metric that takes a raw dataset as input and returns a number that correlates with “this will make a better robot.” Every decision about what to collect, how much, and from whom is downstream of a guess. That includes the biggest guess in the field right now: that egocentric video scales for robotics the way text scaled for language models. That’s not a finding, but a prior. One that has been priced by labs, funded by VCs, and globally shipped.
The Assumption
Language is already compressed cognition1. Every sentence on the internet is the residue of someone’s perceptual work: saw the world, decided what mattered, discarded the rest, encoded what was left. When a language model pre-trains on a trillion tokens, it isn’t learning from raw experience. It’s distilling from a billion people who already abstracted it.
Video is the input side. Raw, pre-abstraction sensory data, the world before anyone decided what in it mattered. Not the same substance as text, however much both get called “data.”
The bet on video says: do the same thing one layer down. Skip the abstraction. Train directly on the sensory stream — hands, objects, contact. Let the model build its own abstractions instead of inheriting language’s. Scale hours of footage the way GPT scaled tokens, and the same curve should show up.
Every hyperscaler is behind this. Real capital is behind it.2 It is a bet, and not a result.
Where it Breaks
The wrong axis
What is actually in a unit of data?
A token carries about ten bits of semantic content, already stripped of ambiguity by whoever wrote the sentence.
A 1080p video frame carries maybe a few hundred task-relevant bits: hand pose, contact state, object ID. But it is buried in a million bytes of texture, lighting, and camera shake.
Same unit. Wildly different payload.
Language scaling counts tokens. A token is close to the unit of information the model needs. Video scaling counts hours, a metric that is easy to report.
One person folding shirts for an hour. A hundred people across a hundred kitchens for an hour. Same number on the x-axis. Not the same data.
Hours is the wrong axis. Diversity is the right one. Labs know this, but they still publish raw counts for diversity instead of measuring how diverse.
The wrong loss
Next-token loss is one of the best proxy objectives ever discovered on accident. Minimize for LLMs and models get better at answering questions, writing code, holding a chain of reasoning. The proxy and the target are almost the same thing.
Next-frame loss is a terrible proxy. It’s dominated by photometric prediction — will this pixel brighten, will that shadow move. Drive it toward zero and you might still learn nothing about which actions caused which outcomes.
Veo and Sora generate minutes of near-photorealistic footage of hands manipulating objects. None of that implies a policy. Perfect video. Zero manipulation competence. Same model, same time.
Scale the wrong loss and you get a better version of the wrong thing.
Identifiability
This one doesn’t get fixed with more data.
Video is observational. It shows what a person did, once, under intent the camera never recorded. It never shows what would have happened if they’d grasped an inch to the left, if the object had been heavier, if their hand were a different size.
A policy needs exactly that comparison — state, chosen action, outcome, evaluated against the actions that weren’t taken. That’s an interventional quantity. Video only ever samples the action someone actually took.
That’s a missing variable, not a small sample. More data doesn’t fix a missing variable. It just makes you more confident in the wrong answer.
Ten billion hours of ego video and ten thousand hours share the same property: neither one is an intervention.
What Annotation Can’t Fix
The obvious reply to all three breaks is “this is just for pretraining”. Nobody claims only video produces a policy. It produces a prior, the visual features, physics, object permanence. The policy comes later, from post-training on real robot data. LLMs work the same way: pretrain, then align.
Fair. And annotation does the real work. Language instructions get added after the fact. Hand and body pose gets estimated. Tasks get segmented, outcomes get scored, garbage gets filtered. Most of what makes ego datasets usable at all is layered on after the capture. The Physical AI industry has built an entire global annotation economy on this.
But there are two lists.
The line doesn't separate two kinds of data. It cuts through single signals. Annotation can tell you contact happened. It cannot tell you how hard. One side is inference from pixels. The other had to be sensed at the moment of capture, or it's gone. No annotator recovers muscle commands from pixels.
And the pretraining defense has a testable implication. If the prior is real, post-training cost should fall as pretraining scales. That number is checkable. Almost nobody reports it.
Annotation patches what’s missing. It can’t patch what was never measured.
The instrument, in the wild
Nobody has the full instrument. Pieces of it exist. You can see them in who measures what.
EgoVerse3 ran the study the counts imply. 1,362 hours, 240 scenes, 2,087 demonstrators — and a controlled experiment on top. Co-training with human data improved robot performance up to 30%. But the sharper finding: diverse ego data alone produced no scaling. A small amount of aligned human-robot data — same tasks, same scenes as the robot data — had to anchor it. Two hours of aligned data unlocked transfer from the diverse pile. Scale didn’t work until alignment made it work.
They also split diversity apart: scene diversity drives generalization to new environments, demonstrator diversity to new people. Different axes, different effects. A count can’t tell you that.
Last week, Danfei Xu, senior author on EgoVerse, sketched where each data-collection method tops out:
Note his y-axis: unique tasks × success rate. Not hours. Sr. researchers at Nvidia GR00T have already dropped the most reported metric.
ActiveMimic4 measured the other direction. Ego-pretrained models consistently underperform robot-data-pretrained ones until the authors identified a missing signal in the pipeline and recovered it. The gap between what the camera saw and what the policy needed was real, and it showed up as a number.
Sunday5 built the fullest version. Their UMI-style glove measures action at capture, signals no annotator can recover after the fact. Then the reporting: scope declared before evaluation, 99.1% zero-shot on unseen homes, and a generalization gap that fell from 82 points to zero as pretraining scaled. Meaning: at full scale, the model needed zero home-specific data to hit 99.1% in homes it had never seen. The prior was real, and they proved it with a curve, not a claim.
That generalization gap is the metric nobody else publishes today. It's adaptation cost, measured.
Three different answers, one pattern. EgoVerse measured what it collected. ActiveMimic measured what was missing. Sunday designed the capture so measurement was possible before the scale bet was placed.
The instrument is not a single metric but a practice: declare what the data should transfer to, measure the gap, report it.
The scaling bet is still priced, funded, and shipped as a finding. It’s still a prior. The difference now: you don’t have to guess. The field’s real lever isn’t more video. It’s building capture and measurement together, so the next billion dollars knows what it’s buying.
Identifiability is still broken. No amount of captured video fixes it. Simulation is the other half of this story — the interventional data video can't give you. Next piece.
P.S. — If you want to see hours, environments, and diversity counts across the open ego datasets side by side, I built a scorecard: Open Data Eval
Compression is Intelligence by 3Blue1Brown is a great watch that re-discovers entropy and finds that compression of information is synonymous with intelligence.
For the strongest published version of this bet, see Hu et al., Data Scaling Laws in Imitation Learning for Robotic Manipulation.
Punamiya, Kareer, et al., EgoVerse (2026). The largest action-labeled egocentric dataset for robot learning — and its authors state human-to-robot transfer ‘remains an open research problem, with unresolved questions around the embodiment gap and scaling behavior.’
ActiveMimic (2026). The paper’s own motivation: models pretrained on egocentric human video consistently underperform those pretrained on robot data — until a missing signal, active perception, is recovered from the capture.
Sunday Robotics, ACT-2 Preview (July 2026): pre-declared evaluation scope, 99.1% zero-shot on unseen homes, generalization gap falling from ~82 points to zero with pretraining scale. See also ACT-1 (Nov 2025) for the Skill Capture Glove design.
Learned a lot, thanks for putting this together Varun!
I like the framing around minimizing adaptation cost, but once you add more sensors for multimodal capture (e.g. tactile, force, finger config), how do you evaluate which data streams are actually useful vs. noise? It seems like the relevant of each modality is task-dependent.
Insightful read!