AUTHOR=Borges Elvis I. A. , Rieder Jonas S. I. , Aschenbrenner Doris , Scharff Rob B. N. 

TITLE=Framework for Armature-Based 3D Shape Reconstruction of Sensorized Soft Robots in eXtended Reality

JOURNAL=Frontiers in Robotics and AI

VOLUME=Volume 9 - 2022

YEAR=2022

URL=https://www.frontiersin.org/journals/robotics-and-ai/articles/10.3389/frobt.2022.810328

DOI=10.3389/frobt.2022.810328

ISSN=2296-9144

ABSTRACT=Due to the compliance of soft robots, their response to actuation inputs heavily depends on the environment in which they operate. Therefore, the ability to study soft robots' behavior while interacting with their environment is crucial for improving their design and control strategies. However, soft robots are often intended to operate in confined environments that impede direct observation of the robot. Although recent developments in proprioceptive sensors for soft robots have enabled accurate real-time capture of a soft robot’s configuration while operating in such environments, these complex three-dimensional configurations can be difficult to interpret using traditional visualization techniques. 

In this work, we present an open-source framework for real-time three-dimensional reconstruction of soft robots in eXtended Reality (Augmented and Virtual Reality). XR offers the opportunity to visualize the simulation as an overlay to the real environment and enhance the experience by displaying additional information on the soft robot that cannot be observed during direct observation. 
The approach is demonstrated in Augmented Reality using a Microsoft Hololens device and runs at up to 60~FPS. We explore the influence that system parameters such as mesh density, neural network design, and armature complexity have on the reconstruction (i.e., speed, scalability).

The open-source framework is expected to function as a platform for future research and developments on real-time control and investigation of soft robots operating in environments that impede direct observation of the robot.