PARTI-A Haptic Virtual Reality Control Station for Model-Mediated Robotic Applications

1 Introduction

As changing demographics lead to increasingly aging populations many recent research efforts focus on robotic applications to support elderly users in every-day tasks in order to increase the user’s autonomy and support care providers. One such effort is the humanoid robot assistant GARMI, which was specifically designed to support elderly users at home (cf. Tröbinger et al. (2021)). Indeed, bimanual manipulation tasks and skills of caregivers are crucial for elderly care scenarios. In this work, we introduce the haptic virtual reality control station and counterpart to GARMI, called PARTI. The PARTI system was designed to closely mimic the kinematic and dynamic configuration of a caregiver and mapping it to the humanoid’s two-arm system. By doing so, the system can function as a haptic input interface for bilateral teleoperation architecture, similar to an upper-body exoskeleton. Much beyond that the system integrates an advanced simulation framework and virtual reality user interfaces to facilitate immersive real-time interaction with digital twins. The digital twins in turn can be transparently coupled to their real-world counterparts. Such a system not only is beneficial for training the caregivers to use such a complex robotic system but also for sending/teaching nursing skills to GARMI.

Multimodal telepresence systems are gaining interest due to the demand for remote interaction with haptic feedback. There already exist a variety of different teleoperation-/telepresence systems such as NimbRo (Schwarz et al., 2021) and DLR’s HUG (Hulin et al., 2011), just to name a few. The systems were designed for different use case scenarios, NimbRo focusses on teleoperation of a humanoid avatar, while DLR’s HUG system also considers interaction with virtual environments for training purposes (Sagardia et al., 2016). Delay compensation techniques for NimbRo are neglected, since the distance between operator and teleoperator is assumed to be small. Balachandran et al. (2018) investigate HUGs feedback behaviour under delay.

As in Sagardia et al. (2016), we also develop a simulation environment with different use cases such as training medical and technical personal. Therefore realistic simulated haptic force feedback is required that also needs to consider the hardware on the simulated remote side. We opt MuJoCo’s (Todorov et al., 2012) convenient representation in generalized coordinates for kinematic chains, yielding haptic feedback in case of undesired (self-) collisions. A notable byproduct of the simulation of the systems is a digital twin, which can be used for system diagnostics. However the focus of our simulation lies on physical realism and real-time aspects for haptic force feedback. This system allows us to explore complex scenarios where various physical and virtual agents interact with each other in a simulated environment, not unlike a haptic meta-verse for robotic systems (see Figure 1). We refer to this approach as model-mediated robot control and present several experiments to highlight the potential of this paradigm. The main components of our system, which we explore further, are namely:

• State-of-the-art digital twins based on a multibody dynamics physical engine with real-time capability;

• Haptic real-time interface rendering constraint forces from the virtual physics engine in generalized coordinates;

• Hardware-in-the-loop interface for model-mediated closed-loop control.

FIGURE 1

FIGURE 1. System overview of our model-mediated approach to robot control. The simulation environment is populated with digital twins that may interact with each other and the environment. Hardware interfaces allow for transparent coupling of the digital twin to its real world counterpart to close the control loop in the physical world.

Our model-mediated robot control paradigm can be linked to the model-mediated teleoperation (MMT) approach (Xu et al., 2016). To our knowledge this is the first work that considers a physics engine as model for MMT. Virtual reality (VR)-based teleoperation is widely used for MMT to provide a non-delayed force and visual feedback and thus guarantee system stability and transparency. Many studies on VR-based MMT have focused on estimating the environment geometry. Valenzuela-Urrutia et al. (2019) have built a virtual workspace that is created directly from point cloud data. Similarly in Ni et al. (2017), the remote real world is modeled in a local virtual environment with the help of RGB information and a 3D registration method. Both virtual environments, however, do not provide sufficient physical properties. The gap between the virtual model and the real world is still large. Recently, the digital twin methodology is applied to a robot-environment interaction for enhancing task performance with the help of Unity (Li et al., 2021). A 3-D generic graphic model as well as the contact dynamics of the real entities is modeled to mirror a real interactive scene. Physical properties of the real environment are also extracted, simultaneously. This work, however, does not consider real teleoperation scenarios with communication unreliability. System control schemes to deal with various tasks and communication delays are not studied. As a contrast, in this paper, we have proposed a digital twin system with a physical engine to describe the full dynamics of the real environment. This minimizes the gap between visual-haptic rendering in virtual environment and the real-world interaction. In addition, our work considers teleoperation over real communication networks, where time delay and packet loss are inevitable. To address these issue, stability-ensuring control schemes such as the Time Domain Passisivity Approach (TDPA, Ryu and Preusche (2007)) and MMT are investigated on our system.

2 PARTI: System Overview

2.1 Hardware

PARTI is designed to emulate and mirror the same physical properties of humanoid avatar GARMI, whilst at the same time enabling a pleasant human-robot and haptic interaction. To this aim, the first design choice concerns the hardware layout. The robot arms were mounted horizontally, rather than vertically—with care consideration with respect to the reachability space of the human operator. This mode of operation requires a modified firmware on each of the robots master controllers for proper gravity compensation. Also precise adjustment of the main frame is mandatory, such that gravity compensation works as intended. Therefore, PARTI’s baseframe is supported by four height-adjustable wheels for leveling purposes. The seat as well as robotic arms are adjustable to fit the operators demands while mirroring the properties of the humanoid avatar GARMI. A second obvious design choice was to use the Panda robot from Franka Emika, a torque-actuated 7-DoF robotic system, for each arm. The robotic arm is embedded with torque sensors on every joint, which in turn properly emulated the GARMI robot and, most importantly, provided precise (cf. Kirschner et al. (2021)) wrench estimation on task-space on different levels of the robotic kinematics, i.e., in different parts of the serial chain. Such feature enabled real-time wrench rendering in our HIL framework at the custom end-effectors is shown in Figure 2. For haptic- and visual rendering, a performant computer (Intel^® Core™ i7-9700F, 16 GiB RAM, NVIDIA GeForce RTX 2080 Ti) is connected to both master controllers via a switch generating torque signals for the robotic arms or reading telemetry data in a 1 kHz control loop. Since teleoperation requires depth perception, we use a 3D-PluraView monitor for stereoscopic rendering. The PARTI hardware framework, as described above, is shown in Figure 2 with its base frame and backwall consisting of a robust construction of item-profiles enabling flexible prototyping.

FIGURE 2

FIGURE 2. Overview of the hardware components of the PARTI system.

2.2 Simulation and Control

Our framework integrates physically accurate digital twins (see Figure 3) directly into the real-time control loop of our hardware to provide a framework for model-mediated hardware-in-the-loop control. The hardware-in-the-loop requirements of our framework—that integrates virtual worlds and a bimanual-robot digital-twin to our haptic framework and real-world applications—calls for most recent developments in robotics’ hardware, as previously described, as well as for new control algorithms and state-of-the-art physics engines. Indeed, our proposed framework has only be made feasible due to separated developments from the past 5–10 years.¹ For the physics engine and core virtual environment, our system relies on MuJoCo ² as it is one of the modicum engines that satisfies our requirements, that is, it is capable of running highly constrained and contact rich simulations in real-time at a control frequency of 1 kHz (cf. Todorov et al. (2012)). Compared to other popular robot simulators such as Gazebo, MuJoCo has the additional advantage of simulating in generalized coordinates (cf. Erez et al. (2015)). We use the latter to define an interface between the physical system and its digital twin for haptic interaction with the simulation environment. Given the Lagrangian describing the system dynamics of an n-degree-of-freedom digital twin as

$\mathit{M}\left({\mathit{q}}_{\mathsf{d}\mathsf{t}}\right){\ddot{\mathit{q}}}_{\mathsf{d}\mathsf{t}}+\mathit{c}\left({\mathit{q}}_{\mathsf{d}\mathsf{t}},{\dot{\mathit{q}}}_{\mathsf{d}\mathsf{t}}\right)+\mathit{g}\left({\mathit{q}}_{\mathsf{d}\mathsf{t}}\right)={\mathit{\tau }}_{\mathsf{d}\mathsf{t},\mathsf{d}}+{\mathit{\tau }}_{\mathsf{constr}}+{\mathit{\tau }}_{\mathsf{bias}},(1)$

where $\mathit{M}\in {\mathbb{R}}^{n\times n}$ is the total inertia, $\mathit{c}\left(\mathit{q},\dot{\mathit{q}}\right),\mathit{g}\left(\mathit{q}\right)\in {\mathbb{R}}^n$ the Coriolis and centrifugal and gravity forces, $\mathit{q},\dot{\mathit{q}}\in {\mathbb{R}}^n$ position and velocity and ${\mathit{\tau }}_{\mathsf{d}\mathsf{t},\mathsf{d}},{\mathit{\tau }}_{\mathsf{constr}},{\mathit{\tau }}_{\mathsf{bias}}\in {\mathbb{R}}^n$ the applied, constraint and bias forces all in generalized coordinates. We couple the digital twin’s motion to the physical system using a PD control law in generalized coordinates

${\mathit{\tau }}_{\mathsf{d}\mathsf{t},\mathsf{d}}=\mathit{K}\left({\mathit{q}}_{\mathsf{p}\mathsf{h}\mathsf{y}\mathsf{s}}-{\mathit{q}}_{\mathsf{d}\mathsf{t}}\right)-\mathit{D}{\dot{\mathit{q}}}_{\mathsf{d}\mathsf{t}}(2)$

where $\mathit{K},\mathit{D}\in {\mathbb{R}}^{7\times 7}$ are gain matrices and the subscripts _phys and _dt are used to refer to the physical system and digital twin respectively. Note that while we could set the position of the digital twin directly, we use a tracking controller instead. This way the dsigital twin is subject to the entire simulation pipeline, including computation of the constraint force ${\mathit{\tau }}_{\mathsf{constr}}$ and bias force ${\mathit{\tau }}_{\mathsf{bias}}=\mathit{c}\left(\mathit{q},\dot{\mathit{q}}\right)+\mathit{g}\left(\mathit{q}\right)$. The constraint force computed by MuJoCo includes the forces produced by contacts and joint limits (cf. Todorov (2014)) and can be directly applied to the physical system as

${\mathit{\tau }}_{\mathsf{p}\mathsf{h}\mathsf{y}\mathsf{s},\mathsf{d}}=\mathit{\lambda }{\mathit{\tau }}_{\mathsf{constr}}(3)$

where $\mathit{\lambda }\in {\mathbb{R}}^{n\times n}$ is a gain matrix that usually comes down to a single scalar to adjust for the reality gap between the digital twin and the physical system. In addition, we usually apply a slight low-pass IIR filter to ${\mathit{\tau }}_{\mathsf{constr}}$ to make haptic interactions smoother for the user. We also implemented a hardware-in-the-loop (HIL) interface that allows us to run closed-loop controllers from within the simulator on the physical system. This is done by superseding the state of the digital twin with that of the physical system and applying the computed control signal on the physical system, thus closing the loop. Using the HIL interface, we can run experiments developed in the simulator on real hardware without any development overhead.

FIGURE 3

FIGURE 3. Visualization of our two primary digital twin models in MuJoCo. The model of the humanoid robot GARMI shown on the left includes fully actuated arms, head and a friction driven differential drive system with castor wheels and passive suspension. The model of PARTI to the right also features a fully actuated two-arm system. These models can run purely in a software environment or be coupled to the real hardware using the haptic or HIL interfaces.

3 Experiments

We performed a series of experiments to highlight the advantages provided by our model-mediated approach to robot control and the design of our control station. The first experiment uses the real-time interface to our digital twin to haptically render generalized constraint forces on the robot. Using this methodology, we can transfer physical behavior modelled in the simulation environment to reality with minimal effort. This is demonstrated by applying a safety margin to the collision meshes of the digital twin. The real robot consequently adheres to this updated model, which serves as a form of collision avoidance—and enables the design of geometric-based virtual walls for task-specific applications. In the second experiment, we evaluate the physical fidelity of our digital twin for the Franka Emika Robot System. We compare the tracking performance as well as the estimated contact wrenches of our model for the MuJoCo simulator to those of the real system as well as a model for the Gazebo simulation environment provided by Franka Emika. Finally, we explore the use of our system as a teleoperation station controlling a humanoid robot in a contact-rich scenario. We compare different approaches such as bilateral position-force computed teleoperation as well as model mediated control.

3.1 Collision Avoidance

We use the haptic interface introduced in Section 2.2 to connect the PARTI system to its digital twin. For this experiment, the collision model of the digital twin is extended with a safety margin in the form of two spherical meshes around the end effectors as seen in Figure 4. While our approach allows the use of arbitrary meshes ³, we chose spherical shapes for salient visualization and affordable computation of distances. The haptic interface will then render the generalized constraint forces as torques on the robot arms with regard to this updated model, effectively preventing any collisions between the end effectors and the rest of the system. To demonstrate the emerging collision avoidance behavior, an operator attempts to bring the end effectors into collision with each other and the main body of the PARTI system, while we record the shortest distance between the surfaces of the involved collision meshes during the motion. The compiled results in Figure 5 confirm that our approach successfully prevented the intersection of collision geometries.

FIGURE 4

FIGURE 4. Visualization of the digital twin of the PARTI system in MuJoCo (left) and the coupled real hardware (right) during the first experiment, where the operator actively attempts to bring the system into collision. The additional collision meshes around the end effectors to facilitate the collision avoidance behavior are seen in red.

FIGURE 5

FIGURE 5. Shortest distances between the collision surfaces of the PARTI system’s left and right end effectors as well as the main body during the first experiment. The constraint forces from the simulator applied to the physical system prevent the collision meshes from intersecting, i.e. the distances remain greater or equal to 0. Note that the sections highlighted in grey mark active attempts by the operator to bring the respective bodies into collision.

3.2 Digital Twin

The second experiment explores the physical fidelity of our digital twin. We compare our MuJoCo model of the Franka Emika Robot System as well as the Gazebo model, provided by Franka Emika, to the physical robot, which serves as a ground truth. For this purpose, we prepared a trajectory, which we taught kinesthetically using the PARTI system. The trajectory includes fast free motion as well as contacts with a solid plane below the base, which is perpendicular to the robot’s first axis. We execute this trajectory using a task space PD control law with additional nullspace control running at a frequency of 1 kHz, which reads

${\mathit{\tau }}_{\mathsf{d}}=\left.\mathit{J}{\left(\mathit{q}\right)}^{\mathsf{T}}\left({\mathit{K}}_{\mathsf{d}}\left({\mathit{x}}_{\mathsf{d}}-\mathit{x}\right)-{\mathit{K}}_{\mathsf{p}}\dot{\mathit{x}}\right)\right)+{\mathit{K}}_{\mathsf{null}}\left(\mathit{I}-\mathit{J}{\left(\mathit{q}\right)}^{\mathsf{T}}\mathit{J}{\left(\mathit{q}\right)}^{\mathsf{T}†}\right)\left({\mathit{q}}_{\mathsf{null,d}}-\mathit{q}\right)(4)$

where ${\mathit{\tau }}_{\mathsf{d}}\in {\mathbb{R}}^7$ is the control torque applied to the robot, $\mathit{J}\left(\mathit{q}\right)\in {\mathbb{R}}^{6\times 7}$, ${\mathit{x}}_{\mathsf{d}}\in {\mathbb{R}}^6$ and $\dot{\mathit{x}}\in {\mathbb{R}}^6$ are the end effector Jacobian, pose, and twist respectively, $\mathit{q}\in {\mathbb{R}}^7$ are the joint positions, and $\mathit{I}\in {\mathbb{R}}^{7\times 7}$ is the identity matrix. The trajectory is given to the controller as a list of Cartesian end effector poses ${\mathit{x}}_{\mathsf{d}}\in {\mathbb{R}}^6$. The gain matrices ${\mathit{K}}_{\mathsf{p}},{\mathit{K}}_{\mathsf{d}}\in {\mathbb{R}}^{6\times 6}$ and ${\mathit{K}}_{\mathsf{null}}\in {\mathbb{R}}^{7\times 7}$ as well as the desired nullspace configuration ${\mathit{q}}_{\mathsf{null,d}}$, which is equal to the initial joint positions⁴, are identical across the experiments. We present the resulting end effector positions and external force estimates in Figure 6 and a more detailed view of the model errors compared to the real hardware in Figure 7. The robot shows some characteristics that are difficult to model due to the proprietary nature of the control and signal processing system. Especially the estimates for external wrenches show an offset and sensitivity to configuration and regions of the workspace (cf. Petrea et al. (2021)).

FIGURE 6

FIGURE 6. The left column of graphs shows the resulting end effector positions as well as the tracked trajectory for the second experiment, while the right column displays the respective models’ estimated external forces acting at the end effector during the same time period. All quantities are expressed relative to the robot’s base frame. Periods of contact are highlighted by the grey vertical bars. Note that the robot is mounted 2 cm above the plane and contact is therefore established slightly below 0 on the z-axis while the reference trajectory reaches below the plane to facilitate solid contact wrenches given the controller’s impedance.

FIGURE 7

FIGURE 7. Linear and angular error norms for the pose (left) and external wrench estimate (right) as observed in the second experiment. The model error of the Gazebo implementation is compared to the error of our own digital twin based on MuJoCo, where telemetry from the real system serves as ground truth.

3.2.1 Gazebo Model

Franka Emika provides a simulation environment, which is well embedded in the ROS framework, granting the user all parameters for implementing custom control laws. Also the telemetry data for evaluation can be accessed easily. Within the ROS framework, we implement a simple ROS node, which publishes each point of the recorded trajectory. We then record the telemetry data with another node. Surprisingly, the Gazebo simulation behaved not as expected. The results diverged, both from the estimates provided by the MuJoCo environment and from the real system. The estimated forces showed a greater magnitude during movement and static contacts. A closer look to the code base revealed that it utilizes a type of direct estimator with further moving average filtering for computing external torques

$\begin{array}{c}\hfill {\mathit{\tau }}_{\mathit{ext}}^{\left(t\right)}={\mathit{\tau }}_d^{\left(t\right)}-{\mathit{\tau }}_m^{\left(t\right)}+\mathit{g}\left({\mathit{q}}^{\left(t\right)}\right),\\ \hfill {\widehat{\mathit{\tau }}}_{\mathit{ext}}^{\left(t\right)}=\gamma {\mathit{\tau }}_{\mathit{ext}}^{\left(t\right)}+\left(1-\gamma \right){\widehat{\mathit{\tau }}}_{\mathit{ext}}^{\left(t-1\right)}\end{array}(5)$

where superscript $\left(t\right)$ denotes the current timestep, and ${\mathit{\tau }}_{\mathit{ext}}\in {\mathbb{R}}^7$ and ${\widehat{\mathit{\tau }}}_{\mathit{ext}}\in {\mathbb{R}}^7$ the estimated torque and the filtered torque estimate, respectively. ${\mathit{\tau }}_m\in {\mathbb{R}}^7$ represents the measured torque and $\mathit{g}\left(\mathit{q}\right)\in {\mathbb{R}}^7$ denotes the gravitational induced generalized forces. The filter parameter γ should be chosen such that $\gamma \in \left(\mathrm{0,1}\right]$. In order to get more insight to the simulation, we inquired Franka Emika. The main reason for employing the estimation, stems from the underlying ODE engine, which enforces constraints for each joint and therefore does not know the external torques in joint space.

The implemented observer resembling the direct observer (Haddadin et al., 2017) uses commanded torque signals generated by some control law, while neglecting inertia and coriolis/centrifugal terms. We assume that these approximations have an significant effect for the external torque estimates especially during free end effector motion. During the first 10 s of the traversed trajectory in Figure 6, the forces of the Gazebo environment are notably overestimated, most likely due to the uncompensated inertia terms.

3.2.2 MuJoCo Model

The kinematic and geometric properties of our digital twin for the MuJoCo simulation environment are based on information provided by the manufacturer, specifically the Denavit–Hartenberg parameters and the visual meshes for the robot links, respectively. For the dynamic model, i.e. the links’ inertia tensors, mass and center of mass, and the friction in the joints, we use the parameters identified in Gaz et al. (2019). Mechatronic components like the motors, torque sensors, or the position encoders in the joints are modelled using MuJoCo’s equivalent native components. Given these definitions, the simulator’s computational engine provides us with the necessary dynamic model quantities for robot control, such as the Jacobian matrix for all the bodies, Coriolis and gravity forces, and inertia in generalized coordinates. While we can retrieve the generalized constraint forces from MuJoCo to numerical precision, we opted to implement a momentum observer for estimating the external torque at the joints ${\mathit{\tau }}_{\mathsf{ext}}$ to more closely model the behavior of the real robot (cf. Haddadin et al. (2017)). Given the definition of the robot’s momentum p as

$\mathit{p}=\mathit{M}\left(\mathit{q}\right)\dot{\mathit{q}},(6)$

the external torques can be estimated as

${\widehat{\mathit{\tau }}}_{\mathsf{ext}}={\mathit{K}}_{\mathsf{O}}\left(\mathit{p}\left(t\right)-{\int }_0^t{\mathit{\tau }}_{\mathsf{m}}-\left(\mathit{g}\left(\mathit{q}\right)+\mathit{c}\left(\mathit{q},\dot{\mathit{q}}\right)-\dot{\mathit{M}}\left(\mathit{q}\right)\dot{\mathit{q}}+{\widehat{\mathit{\tau }}}_{\mathsf{ext}}\right)d\mathsf{s}-\mathit{p}\left(0\right)\right)(7)$

where ${\mathit{\tau }}_{\mathsf{m}},\mathit{c}\left(\mathit{q},\dot{\mathit{q}}\right),\mathit{g}\left(\mathit{q}\right)\in {\mathbb{R}}^7$ are the applied motor force, centrifugal and Coriolis forces, and gravity force, respectively, as computed by MuJoCo in the robot’s generalized coordinates, and ${\mathit{K}}_{\mathsf{O}}=\mathrm{diag}\{k_{\mathsf{O,i}}\}>0$ is the observer’s gain matrix. Additionally, we compute $\dot{\mathit{q}}$ numerically using the Euler method for the observer instead, as joint velocities are not directly measured by the robot. The estimated external wrench acting at the end effector is then simply computed as

${\mathit{F}}_{\mathsf{ext}}=\mathit{J}\left(\mathit{q}\right){\mathit{\tau }}_{\mathsf{ext}}.(8)$

This estimate shares characteristics with the ground truth, such as a high frequency but low magntitude noise and sensitivity to robot movements, while the estimated wrench corresponds well with the robot’s own estimates during contacts (cf. Figure 6).

3.3 Two-Arm Teleoperation

This experiment demonstrates our system’s versatility as a teleoperation station by controlling the digital twin of the robot humanoid GARMI in a contact-rich cooperative two-arm manipulation scenario. The task consists of lifting a box out of a socket and putting it inside a shelve with only small clearance (cf. Figure 8). During this task, the box is only held in place by the resulting friction due to the pressure exerted by the arms. The operator controls both arms independently using the two robot arms of the PARTI system and receives haptic force feedback for both. The virtual scene is graphically rendered in stereo from the perspective of the humanoid’s head camera and displayed stereoscopically on the PARTI display to provide the operator with depth perception. For this experiment, we coupled the controlled velocity of the GARMI mobile base to the arms’ displacements along the humanoid’s sagittal axis ⁵. This way, the operator can focus on the manipulation task without the additional mental load of having to explicitly control the robot’s mobile drive system. We evaluated different control schemes to perform this task. First, we directly control the humanoid, i.e. we use the haptic interface introduced in Section 2.2 to connect the physical PARTI system to the digital twin of GARMI. This is possible as the systems were designed to have identical kinematic and dynamic configurations. We refer to this approach as model-mediated teleoperation (MMT), as the physical system interacts solely with a local representation of the remote environment and as such is not subject to communication delays (cf. Xu et al. (2016)). As an alternative approach, we implemented a more traditional bilateral position-force computed (PFC) architecture. Within this approach, the twists of the leader system’s end effector are applied to the follower system, while estimated wrenches at the respective follower’s end effectors are reflected as haptic force feedback. We complement this architecture with passivity controllers informed by energy observers to compensate for communication delays as introduced in Ryu and Preusche (2007). Our implementation of the latter method—commonly referred to as time domain passivity approach (TDPA)—uses a six degree of freedom two channel architecture for each arm. In total, our experiment consists of three teleoperator configurations, namely the MMT variant as well as the PFC architecture with and without TDPA and a round-trip delay of around 35 ms. For analysis, we compute the net external force of GARMI’s end effectors and the relative transform between the end effectors over time as depicted in Figure 9.

FIGURE 8

FIGURE 8. Visualized sequence of the digital twin of GARMI (bottom row) and the coupled real PARTI system (top row) during the third experiment. The operator is tasked with lifting the box from its socket, which has a clearance of only 2 mm and putting it into the shelve below with a clearance of 2 cm.

FIGURE 9

FIGURE 9. The graphs in the left column show the net force of the estimated external wrenches acting on GARMI’s end effectors during the teleoperatoin task of the third experiment. The column on the right displays graphs of the displacement between the end effectors. Note that the force and displacement induced by the grasp are primarily expressed around the y-axis of this experiment. The MMT method allowed for the successful completion of the task, while the oscillations and instability of the PFC architecuteres with and without TDPA respectively resulted in the grasp being released before the task could be completed.

4 Discussion

The results of our experiments show the benefits of a model-mediated approach to robot control in various applications, including scenarios with a human operator in the loop. The first experiment demonstrated how a simple change in the model can result in complex behavior, which can be transferred to the real hardware using the haptic interface. While the emerging collision avoidance behavior is useful in itself, it is worth noting that this approach allows us to focus on the physical modeling of a desired behavior without having to concern ourselves with the implementation of any controllers generating said behavior. We also showed that given identical dynamic parameters the behavior of our digital twin of the Franka Emika Robot System is closer to the real hardware than the provided Gazebo model, reinforcing our choice for MuJoCo as a simulator for model-mediated control. Finally, we demonstrated the advantage of using a model-mediated approach in teleoperation. Analysis shows that activity of the passivity controllers in TDPA results in small oscillations while the box is grasped. This can be explained by the fact that the pressure exerted by the arms on each other acts as an active disturbance, which in turn leads to dissipative action from the passivity controllers. While the arms remain stable, this small oscillation is enough to release the grasp of the box, leaving the operator unable to complete the task. In comparison, the grasp of the PFC architecture without active TDPA is more stable. However, the teleoperator quickly becomes unstable due to external disturbances such as contacts with the shelve. The MMT approach proved to be most successful for this task. Not only was the operator able to complete the task, but the system remained stable even during the challenging action of sliding the box into the shelve, which involves multiple contacts similar to the classical peg-in-hole problem of robotics.

5 Conclusion

We introduced the PARTI system, a multimodal teleoperation station, intended for teleoperation tasks under a hardware-in-the-loop framework for improved performance even in the presence of uncertainties and time delay. For the physics engine deployed in the HIL framework, experiments in virtual envrionment reveal that MuJoCo is advantageous for simulating kinematic chains during physical interaction as it enforces constraints in generalized coordinates. This in turn can be directly leveraged for self collision avoidance by designing collision meshes in simulation that enforce a certain safety margin. Furthermore, in a bimanual teleoperated manipulation task, we see that TDPA led to unstable system behaviour, emphasizing the model mediated teleoperation approach for future research. Since MuJoCo offers physical realism, we see our simulation environment as an ideal testbench for future research, including model-mediated teleoperation, system identification, and model-based/predictive control. The simulation can also serve as digital twin, providing an operator with additional information on the teleoperated system status in a human-understandable (visual) representation. For increasing the autonomy of the humanoid robot GARMI, the simulation might be potentially useful for kinesthetic teaching purposes. We also plan to incorporate VR in the near future, which will provide additional interaction options during teleoperation in tradeoff for situational awareness in the operators environment. We interpret a VR set as a minor restriction, since the operator is losing environmental awareness when wearing the head mounted display.

Data Availability Statement

The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.

Ethics Statement

Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author Contributions

JE: Implemented the HIL and haptic interfaces, digital twins in MuJoCo, two-arm teleoperation and wrote the corresponding sections. GR: Implemented and ran the experiments for Gazebo. He wrote Section 5, Section 3.2.1, and Section 2.1. LF, AN, and UW: Supervised the study, provided feedback and wrote/edited the manuscript. SH: Contributed the idea and concept of PARTI and collected the funding to perform the study.

Conflict of Interest

SH has a potential conflict of interest as shareholder of F.E. GmbH.

The handling editor is currently co-organizing a Research Topic with one of the authors LF, and confirms the absence of any other collaboration.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We would like to thank Nicolas Zunhammer for his valuable contributions to earlier prototypes of the PARTI system and Franka Emika for their support in developing the hardware. We especially thank Xiao Xu, who provided us with valuable feedback for the manuscript. We gratefully acknowledge the funding of the Lighthouse Initiative Geriatronics by StMWi Bayern (Project X, grant no. IUK-1807-0007//IUK582/001).

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frvir.2022.925794/full#supplementary-material

Footnotes

¹Notice the Panda system, as well as, any other collaborative robot with joint-torque sensors have only be made available around 2017, the study of digital twins as dynamic proxies in model-mediated control applications is a recent development (cf. Mitra and Niemeyer (2007)), and MuJoCo was firstly made available in Todorov et al. (2012).

²www.mujoco.org

³MuJoCo uses the Minkowski Portal Refinement algorithm to detect collisions and as such requires convex meshes. However, non-convex geometries may be modelled by decomposing them into a union of convex geometries (Todorov et al., 2012)

⁴The initial joint positions are chosen such that they have the largest distance to the joint limits

⁵We mapped the net displacement along the robot’s saggital axis (the x axis in the robot base frame) of the end effectors relative to the initial pose to the mobile base velocity in x direction

References

Balachandran, R., Kozlova, N., Ott, C., and Albu-Schäeffer, A. (2018). Non-Linear Local Force Feedback Control for Haptic Interfaces. IFAC-PapersOnLine 51, 486–492. doi:10.1016/j.ifacol.2018.11.587

CrossRef Full Text | Google Scholar

Erez, T., Tassa, Y., and Todorov, E. (2015). “Simulation Tools for Model-Based Robotics: Comparison of Bullet, Havok, Mujoco, Ode and Physx,” in 2015 IEEE international conference on robotics and automation (ICRA), Seattle, WA, USA, May 26–30, 2015 (IEEE), 4397–4404. doi:10.1109/icra.2015.7139807

CrossRef Full Text | Google Scholar

Gaz, C., Cognetti, M., Oliva, A., Robuffo Giordano, P., and De Luca, A. (2019). Dynamic Identification of the Franka Emika Panda Robot with Retrieval of Feasible Parameters Using Penalty-Based Optimization. IEEE Robot. Autom. Lett. 4, 4147–4154. doi:10.1109/LRA.2019.2931248

CrossRef Full Text | Google Scholar

Haddadin, S., De Luca, A., and Albu-Schäffer, A. (2017). Robot Collisions: A Survey on Detection, Isolation, and Identification. IEEE Trans. Robot. 33, 1292–1312. doi:10.1109/TRO.2017.2723903

CrossRef Full Text | Google Scholar

Hulin, T., Hertkorn, K., Kremer, P., Schätzle, S., Artigas, J., Sagardia, M., et al. (2011). “The Dlr Bimanual Haptic Device with Optimized Workspace,” in 2011 IEEE International Conference on Robotics and Automation, Shanghai, China, May 9–13, 2011, 3441–3442. doi:10.1109/ICRA.2011.5980066

CrossRef Full Text | Google Scholar

Kirschner, R. J., Kurdas, A., Karacan, K., Junge, P., Baradaran Birjandi, S. A., Mansfeld, N., et al. (2021). “Towards a Reference Framework for Tactile Robot Performance and Safety Benchmarking,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic, September 27–October 1, 2021, 4290–4297. doi:10.1109/IROS51168.2021.9636329

CrossRef Full Text | Google Scholar

Li, X., He, B., Wang, Z., Zhou, Y., Li, G., and Jiang, R. (2021). Semantic-Enhanced Digital Twin System for Robot-Environment Interaction Monitoring. IEEE Trans. Instrum. Meas. 70, 1–13. doi:10.1109/TIM.2021.3066542

PubMed Abstract | CrossRef Full Text | Google Scholar

Mitra, P., and Niemeyer, G. (2007). Haptic Simulation of Manipulator Collisions Using Dynamic Proxies. Presence Teleoperators Virtual Environ. 16, 367–384. doi:10.1162/pres.16.4.367

CrossRef Full Text | Google Scholar

Ni, D., Song, A., Xu, X., Li, H., Zhu, C., and Zeng, H. (2017). 3d-point-cloud Registration and Real-World Dynamic Modelling-Based Virtual Environment Building Method for Teleoperation. Robotica 35, 1958–1974. doi:10.1017/S0263574716000631

CrossRef Full Text | Google Scholar

Petrea, R. A. B., Bertoni, M., and Oboe, R. (2021). “On the Interaction Force Sensing Accuracy of Franka Emika Panda Robot,” in IECON 2021 – 47th Annual Conference of the IEEE Industrial Electronics Society, Toronto, ON, Canada, October 13–16, 2021, 1–6. doi:10.1109/IECON48115.2021.9589424

CrossRef Full Text | Google Scholar

Ryu, J.-H., and Preusche, C. (2007). “Stable Bilateral Control of Teleoperators under Time-Varying Communication Delay: Time Domain Passivity Approach,” in Proceedings 2007 IEEE international conference on robotics and automation, Rome, Italy, April 10–14, 2007 (IEEE), 3508–3513. doi:10.1109/robot.2007.364015

CrossRef Full Text | Google Scholar

Sagardia, M., Hulin, T., Hertkorn, K., Kremer, P., and Schätzle, S. (2016). “A Platform for Bimanual Virtual Assembly Training with Haptic Feedback in Large Multi-Object Environments,” in Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology, Munich, November 2–4, 153–162. doi:10.1145/2993369.2993386

CrossRef Full Text | Google Scholar

Schwarz, M., Lenz, C., Rochow, A., Schreiber, M., and Behnke, S. (2021). “Nimbro Avatar: Interactive Immersive Telepresence with Force-Feedback Telemanipulation,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic, September 27–October 1, 2021, 5312–5319. doi:10.1109/IROS51168.2021.9636191

CrossRef Full Text | Google Scholar

Todorov, E., Erez, T., and Tassa, Y. (2012). “Mujoco: A Physics Engine for Model-Based Control,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura-Algarve, Portugal, October 7–12, 2012 (IEEE), 5026–5033. doi:10.1109/iros.2012.6386109

CrossRef Full Text | Google Scholar

Todorov, E. (2014). “Convex and Analytically-Invertible Dynamics with Contacts and Constraints: Theory and Implementation in Mujoco,” in 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China, May 31–June 7, 2014 (IEEE), 6054–6061. doi:10.1109/icra.2014.6907751

CrossRef Full Text | Google Scholar

Tröbinger, M., Jähne, C., Qu, Z., Elsner, J., Reindl, A., Getz, S., et al. (2021). Introducing GARMI - A Service Robotics Platform to Support the Elderly at Home: Design Philosophy, System Overview and First Results. IEEE Robot. Autom. Lett. 6, 5857–5864. doi:10.1109/lra.2021.3082012

CrossRef Full Text | Google Scholar

Valenzuela-Urrutia, D., Muñoz-Riffo, R., and Ruiz-del-Solar, J. (2019). Virtual Reality-Based Time-Delayed Haptic Teleoperation Using Point Cloud Data. J. Intell. Robot. Syst. 96, 387–400. doi:10.1007/s10846-019-00988-1

CrossRef Full Text | Google Scholar

Xu, X., Cizmeci, B., Schuwerk, C., and Steinbach, E. (2016). Model-mediated Teleoperation: Toward Stable and Transparent Teleoperation Systems. IEEE Access 4, 425–449. doi:10.1109/access.2016.2517926

CrossRef Full Text | Google Scholar