A robot hand can learn to avoid parts it was never taught to avoid.

Robotic hands can follow language commands to hold objects, but they struggle when told where not to touch. For example, if asked to hold a mug by its handle while avoiding its body, models often touch the forbidden body anyway. This happens because past training data only showed robots what to grab, tricking models into treating any mentioned part as a target. To fix this, researchers created a new method that guides the robot hand during its decision-making steps, without needing extra training. Their system pinpoints the forbidden zone with a frozen 3D vision model, steers hand positions toward the target part, and prunes away attempts that approach forbidden areas. They tested it on a new benchmark called NegGrasp. The approach dropped constraint violations from 57.9% down to 17.2% while improving overall grasping success.

Why it matters

This matters for tasks like surgery or manufacturing, where robots must handle objects carefully and follow nuanced commands like 'don't touch this.' Without it, robots would need exhaustive retraining for every possible avoidance instruction, which is impractical.

Questions answered