Translating Human Information into Robot Tasks: Action Sequence Recognition and Robot Control based on Human Motions

Taichi Obinata¹Kazutomo Baba^2,3Akira Uehara^3,4Hiroaki Kawamoto^3,4*Yoshiyuki Sankai^2,3,4

• ¹graduate school of science and technology, university of tsukuba, Tsukuba, Ibaraki, Japan
• ²Cyberdyne Inc., Tsukuba, Ibaraki, Japan
• ³Center for Cybernics Research, University of Tsukuba, Tsukuba, Ibaraki, Japan
• ⁴Institute of Systems and Information Engineering, University of Tsukuba, Tsukuba, Ibaraki, Japan

Long-term use and highly reliable batteries are essential for wearable cyborgs including Hybrid Assistive Limb and wearable vital sensing devices. Consequently, there is ongoing research and development aimed at creating safer next-generation batteries. Researchers, leveraging advanced specialized knowledge and skills, bring products to completion through trial-and-error processes that involve modifying materials, shapes, work protocols, and procedures. When robots can undertake the tedious, repetitive, and attention-demanding tasks currently performed by researchers within facility environments, it will reduce the workload on researchers and ensure reproducibility. In this study, aiming to reduce the workload on researchers and ensure reproducibility in trial-and-error tasks, we proposed and developed a system that collects human motion data, recognizes action sequences, and transfers both physical information (including skeletal coordinates) and task information to a robot. This enables the robot to perform sequential tasks that are traditionally performed by humans. The proposed system employs a non-contact method to acquire three-dimensional skeletal information over time, allowing for quantitative analysis without interfering with sequential tasks. In addition, we developed an action sequence recognition model based on skeletal information and object detection results, which operated independent of background information. This model can adapt to changes in work processes and environments. By translating the human information including the physical and semantic information of a sequential task performed by humans into a robot, the robot can perform the same task. An experiment was conducted to verify this capability using the proposed system. The proposed action sequence recognition method demonstrated high accuracy in recognizing humanperformed tasks with an average Edit score of 95.39 and an average F1@10 score of 0.951. In two out of the four trials, the robot adapted to changes in work processes without misrecognizing action sequences and seamlessly executed the sequential task performed by the human. In conclusion, we confirmed the feasibility of using the proposed system.