I experimented with using a hand tracker and voice-activated commands (start, pause, resume) to collect behavior cloning data in simulation. The webcam tracks my hands and the sim’s bimanual arms follow, while spoken commands control the recording.

It was too hard. Controlling two arms in 3D through a webcam, with no feedback in my hands, was far less precise than the leader arms.

It did help me understand why the wrist cams matter so much more than the ego cam for control. While I was teleoperating, the wrist views showed me what the gripper was doing at the moment of contact, and the ego view could not. The wrist cam sees the contact, and the ego cam mostly sees the scene around it.

That makes me wonder whether the biggest opportunity for improving robots is a touch-native foundation model. If vision close to the contact is what matters most, then the signal the robot lacks is touch itself.