The problem
Many gesture interfaces either rely on pre-scripted commands or on wearable sensors, and few close the loop all the way from a camera image to stable motion on a redundant 7-DoF arm. The question was whether vision-only gesture recognition could drive multistep tasks — not just jog the arm — reliably and in real time.
Approach
- 01GESTID, a CNN-based vision model, continuously recognises the operator's hand gestures from a camera stream.
- 02Each recognised gesture is mapped to a task definition — for example gesture 1 to a pick-and-place routine and gesture 2 to a cleaning routine — described in JSON.
- 03TAGEM generates and executes smooth, coordinated 7-DoF trajectories for the selected task using forward-kinematics-driven trajectory execution.
Main components
- GESTID
- Gesture identification — CNN-based visual recognition
- Gesture → task mapping
- Links each gesture to a JSON-defined manipulation task
- TAGEM
- Trajectory generation and execution on the 7-DoF arm
Evidence
< 70 ms
End-to-end latency reported in the paper
Source: CCPE journal paper, 2026
Publication
2026
Journal article
Published
Real-Time HRI: Human Hand Gesture Mapping for 7-DoF Robotic Arm Operation
Concurrency and Computation: Practice and Experience · Vol. 38, Issue 6, e70643 · Open access
AI-RTGM: an end-to-end framework that couples CNN-based gesture recognition (GESTID) with trajectory generation and execution (TAGEM) for real-time task control of a 7-DoF manipulator.