The problem
People communicate with a mix of gestures, movement and words. The thesis asks how a collaborative manipulator can accept these channels together, interpret them with machine learning, and still behave predictably and safely.
Approach
- 01Perception: vision-based recognition of hand gestures and body pose.
- 02Language: LLM-based interpretation of spoken or written instructions, including multilingual input.
- 03Control: mapping recognised intent to trajectory generation and execution on a 7-DoF Franka Emika Panda.
- 04Evaluation on physical tasks such as pick-and-place, cleaning and beverage preparation.