AUTHOR=Gu Yutong , Zheng Chao , Todoh Masahiro , Zha Fusheng 

TITLE=American Sign Language Translation Using Wearable Inertial and Electromyography Sensors for Tracking Hand Movements and Facial Expressions

JOURNAL=Frontiers in Neuroscience

VOLUME=Volume 16 - 2022

YEAR=2022

URL=https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2022.962141

DOI=10.3389/fnins.2022.962141

ISSN=1662-453X

ABSTRACT=A sign language translation system can break the communication barrier between hearing-impaired people and others. In this paper, a novel American sign language translation method based on wearable sensors is proposed. We leverage inertial sensors to capture signs and surface electromyography sensors to detect facial expressions. We apply convolutional neural network to extract features from input signals. Then, long short-term memory and transformer models are exploited to achieve end-to-end translation from input signals to text sentences. We evaluate two models on 40 American sign language sentences strictly following the rules of grammar. Word error rate and sentence error rate are utilized as evaluation standard. The long short-term memory model can translate sentences in testing dataset with 7.74% word error rate and 9.17% sentence error rate. The transformer model performs much better than long short-term memory by achieving 4.22% word error rate and 4.72% sentence error rate. Both models are suitable for sign language translation with high accuracy.