Catching requires generalizing to novel projectiles and adapting in real time to unpredictable contact, from noisy visual observations alone. Catch It All! is a sim-to-real catching framework for humanoid robots that generalizes to unseen objects with different geometry, deformability, and elasticity, for both underhand and overhand catches, using only one egocentric camera.
Grasps that adapt to the object's pose and shape come from a fingertip-to-surface distance reward. Safety-aware curricula let the policy act at high frequency near hardware limits, including wrist torque limits as low as 5 N·m. The policy is trained in simulation and deployed zero-shot on a Unitree G1 with a 20-DoF dexterous hand, where it catches 9 objects it never saw in training: low-rebound plushes, semi-deformable toys and chip bags, and rigid-elastic objects that bounce.
There is no third-person camera. The only sensor is a ZED 2i on the robot's head, and it moves with the robot. Each panel shows a side-by-side pair from that camera; the blue square marks the object position extracted by SAM2-Tiny, including camera and inference latency. No square means the prediction was dropped, which happens whenever the object leaves the frame.
Overhand
Underhand
Overhand is the harder case. The hand starts high with the palm facing the pitcher and has to intercept the object at the top of its arc, moving forward for short tosses and pulling back mid-flight for long ones. Every clip is a real toss at 0.33× speed, caught zero-shot by the same policy with no per-object tuning.
Takisdeformable
Green bowlrigid-elastic
Strawberrydeformable
Pandaplush
Turtleplush
Pineappledeformable
Bearrigid-elastic
Ellipsoidrigid-elastic
Sandbagplush
For underhand tosses the hand tracks the object through its arc and closes as it arrives, adjusting the grasp to whatever lands in it: a rigid bowl, a bag of chips, a plush turtle.
Pandaplush
Bearrigid-elastic
Green bowlrigid-elastic
Takisdeformable
Strawberrydeformable
Turtleplush
Pineappledeformable
Ellipsoidrigid-elastic
Sandbagplush
Each of the 9 objects was thrown four times per catching style: one long (over 1.8 m), two medium (1.4 to 1.8 m), and one short (1.0 to 1.4 m). Shorter tosses demand fast reaction; longer ones demand good tracking. Success rate over 36 tosses per style.
| Object class | Underhand | Overhand |
|---|---|---|
| Plush panda, turtle, sandbag | 50.0% | 33.3% |
| Deformable Takis, strawberry, pineapple | 25.0% | 41.7% |
| Rigid-elastic green bowl, ellipsoid, bear | 41.7% | 33.3% |
| All objects | 38.9% | 36.1% |
A human teleoperator with a bimanual rig at 15 Hz caught 3 of 44 tosses, which is why the policy is trained with online reinforcement learning in simulation rather than from demonstrations.
Not every toss is caught. In simulation, success drops most on long-axis objects with high angular velocity, which the state representation does not capture, and on very heavy or very large objects. On the real robot, stereo depth noise and dropped detections when the object leaves the frame add to the difficulty.
Bearrigid-elastic
Ellipsoidrigid-elastic
Pandaplush
Strawberrydeformable
Green bowlrigid-elastic
Ellipsoidrigid-elastic
Sandbagplush
Takisdeformable
@misc{shukla2026catchallgeneralizabledynamic,
title = {Catch It All!: Generalizable Dynamic Catching with a Dexterous Hand},
author = {Arth Shukla and Kehlani Fay and Huaze Liu and Darin Djapri and Alexiy Buynitsky and Nicklas Hansen},
year = {2026},
eprint = {2610.39044},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2610.39044},
}