AUTHOR=Cognolato Matteo , Atzori Manfredo , Gassert Roger , Müller Henning 

TITLE=Improving Robotic Hand Prosthesis Control With Eye Tracking and Computer Vision: A Multimodal Approach Based on the Visuomotor Behavior of Grasping

JOURNAL=Frontiers in Artificial Intelligence

VOLUME=Volume 4 - 2021

YEAR=2022

URL=https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2021.744476

DOI=10.3389/frai.2021.744476

ISSN=2624-8212

ABSTRACT=The complexity and dexterity of the human hand make the development of natural and robust
4 control of hand prostheses challenging. Although a large number of control approaches were
5 developed and investigated in the last decades, limited robustness in real-life conditions prevented
6 their application in clinical settings and in commercial products. In this paper, we investigate a
7 multimodal approach that exploits the use of eye-hand coordination to improve the control of
8 myoelectric hand prostheses. The analyzed data are from the recently released MeganePro
9 Dataset 1, that includes multimodal data from transradial amputees and able-bodied subjects
10 while grasping numerous household objects with ten grasp types. A continuous grasp-type
11 classification based on surface electromyography served as both intent detector and classifier.
12 At the same time, the information provided by eye-hand coordination parameters, gaze data
13 and object recognition in first-person videos allowed to identify the object a person aims to
14 grasp. The results show that the inclusion of visual information significantly increases the
15 average offline classification accuracy by up to 15.61 ± 4.22 % for the transradial amputees
16 and of up to 7.37 ± 3.52 % for the able-bodied subjects, allowing trans-radial amputees to reach
17 average classification accuracy comparable (and in some cases higher) than intact subjects and
18 suggesting that the robustness of hand prosthesis control based on grasp-type recognition can
19 be significantly improved with the inclusion of visual information extracted by leveraging natural
20 eye-hand coordination behavior and without placing additional cognitive burden on the user