Guiding intermediate features to be control-relevant improves robot zero-shot performance.

Robots need to turn camera pictures into physical movements. Standard robot visual training learns rich images, but it often misses key physical details and keeps useless background clutter. The researchers call this problem the action-sufficiency gapaction-sufficiency gapThe mismatch between rich visual features and their usefulness for robot control.. To fix it, they created GIFT, or Guided Intermediate Feature Training. GIFT trains the robot's middle visual layers to focus on three useful structures: physical geometry that controls feasible movement, affordance that marks the right objects, and goals that highlight target areas. It teaches these by aligning geometry, predicting affordances, and reconstructing goal regions during training. The team tested GIFT across three robot models. Under zero-shot transfer tests on LIBERO-Plus, the GIFT-enhanced models beat their original versions by 4.6, 12.6, and 5.2 points. On RoboCasa, they beat their counterparts by 12.6, 9.0, and 8.4 points, showing large gains on articulated-object tasks and real-world precision manipulation.

Why it matters

This matters for people building robots that need to work in unpredictable environments. By helping robots focus on essential visual cues, they can handle new tasks without retraining, which improves reliability and reduces setup time.

Questions answered