A Digital Twin Approach for Contextual Assistance for Surgeons During Surgical Robotics Training

1 Introduction

In the last 30 years, Minimally Invasive Robotic Surgery (MIRS) has become an important technology in modern medicine. It leverages the advantages of minimal invasive surgery, such as shorter recovery times for patients, and avoids several of its disadvantages for the surgeons, such as the interrupted hand-eye coordination and the loss of Degrees of Freedom (DoF) inside the patient’s body (Klodmann et al., 2020). Consequently, MIRS decreases the physical and cognitive load once it has been mastered by the surgeon. However, providing individualized and contextual assistance to enhance surgical performance even further remains an open research topic.

Exploitation of commonly used haptic input devices, for instance by means of Virtual Fixtures (VFs), is one possibility in which the surgeon’s performance can be further augmented. However, VFs have to be parameterized according to the context of the task at hand in order to provide reasonable assistance. Due to the unstructured dynamic environment inside the patient’s body, automatic perception of the procedural context in a real surgery remains challenging (van Amsterdam et al., 2021). Surgical training, in turn, offers a well-defined environment. Herein, abstracted training tasks as visualized in Figure 1, are utilized to train key surgical skills.

FIGURE 1

FIGURE 1. The Nylon Twist training pad from Lübecker Toolbox (LLC, 2021) is performed with the DLR MiroSurge system to train the hand-eye-coordination, bi-manual work, and working over cross.

Our approach introduces a Digital Twin (DT) to monitor procedural context information during training. Based on this, we propose a novel concept for the on-line parametrization of haptically rendered assistance functions using VFs. This preserves the dexterity of the surgeon and adds an additional feedback channel to the visual one. We hypothesize that training with our approach enables a novice surgeon to realize a task faster, more accurately and with less cognitive load. The main contributions of this article are 1) an approach to estimate geometric and semantic task states using a DT that mirrors the real robotic system, 2) a novel way to flexibly parameterize assistance functions online based on the estimation of the current semantic task states, 3) the combination of both methods to assist in training for MIRS using multi-modal support, and 4) an evaluation in a proof-of-concept experiment in surgical training.

This work is divided into the following sections. Section 2 further motivates our work based on the current state of the art. The methods used in our approach are explained in Section 3 followed by the description of the pilot study and its results in Section 4. Section 5 presents the conclusions and an outlook.

2 State of The Art

Surgical training tasks such as the Lübecker Toolbox (LLC, 2021) offer various training scenarios that are relevant for real operations, as shown in Figure 1. They are designed to train preparation, resection and reconstruction skills. This sequence maps in parts to a cholecystectomy (removal of the gall bladder) as it requires the surgeons to expose critical structures consisting of the cystic duct and the cystic artery by removing surrounding tissue, followed by inserting multiple clips at the correct locations to stop the fluid flow in these structures so they can be cut to free the gall bladder. Even though modern training curricula for MIRS (Thornblade and Fong, 2021) extend traditional surgical training (Sealy, 1999), it is still cumbersome to acquire the required motor skill level to control a surgical robot for the use in MIRS.

Increased autonomy of surgical robots could decrease the required skill level. Haidegger (2019) introduces a scale to classify the level of autonomy in robotic surgery ranging from no autonomy to full autonomy. Commercially available surgical robotic systems assisting in laprascopy fulfill the criteria for level 1 named robot assistance, whereas prototypes of surgical robots reach task-level autonomy (level 2). Robotic systems in other surgical domains that tackle tasks like autonomously taking blood samples reach up to level 4, high-level autonomy. A great challenge of laparascopic surgery consist of the development of perception algorithms, that extract information from the unpredictable environment containing soft tissue. Therefore, autonomous navigation in MIRS is still challenging (Nagy and Haidegger, 2019).

The introduction of assistance functions by means of shared control (Endsley, 1999) offers a possible solution to support surgeon training for MIRS. As a subsequent step the assistance could transfer to real OR situations. Therefore, shared control could be seen as an intermediate step towards the increase to task-level autonomy. Assistance functions may range from displaying relevant data using mixed reality (Qian et al., 2019), over tremor filtering and scaling of input movements (Weber et al., 2013) to the automation of certain steps of a task under the supervision of the surgeon (Nagy and Haidegger, 2019). VFs as described in Rosenberg (1993), Abbott et al. (2007), and Bowyer et al. (2014) represent a different type of shared control which enables haptic augmentation. They comprise of attractive VFs that haptically guide the user towards a goal structure as in Selvaggio et al. (2018), repulsive VFs protecting him or her from entering in prohibited zones as in Li et al. (2020), and automated scaling of the user input compared to the instrument tip’s movement as in Zhang et al. (2018). Ovur et al. (2019) show that haptic guidance during training in simulation for simple and complex path-following tasks improves performance. Enayati et al. (2018) assess the specific competences and the overall skill level of novice and expert users during training of a path-following task and adapt the strength of assistance functions according to the competence level.

It is essential that assistance functions are represented such that they can be rapidly adjusted to the ongoing task and the dynamically changing environment. In cholecystectomy for instance, critical structures are first exposed, and it is crucial not to harm them during this step. But in later phases, they are manipulated and even cut. Task-dependent shared control functions are required to account for such drastic change of the task context. The problem of adapting shared-control functions online to a changing environment is stated in Selvaggio et al. (2018) and tackled by an human-in-the-loop approach using VFs. The surgeon can generate attractive VFs and adapt their position and geometry by recording interaction points. Fontanelli et al. (2018) design a vision-free pipeline to support surgeons during suturing. A state machine defines the task. The transitions between states are based on predefined gestures and are, therefore, controlled by the surgeons. Different controllers with varying degree of autonomy and VFs are applied. Maier et al. (2019) develop a training simulator for orthopedic hand surgery in virtual reality generating haptic impressions similar to bone cutting for the surgeon using VFs. A state machine is used to switch between different implementations of VFs according to the task state. All environment states are pre-known due to the virtual reality approach.

Human-in-the-loop approaches for the positioning and parametrization of VF add to the surgeon’s workload. Ideally, the parametrization of VFs should be done automatically by robotic system based on semantic knowledge about the task phases and context, i.e., an abstract knowledge representation, about the environment and the state of the system. A DT approach offers the possibility to derive semantic knowledge from fused data obtained by different sources. A robotic system provides diverse data about its poses and control states. Laaki et al. (2019) design a DT of a simplified surgical environment containing one robotic arm that is controlled in a virtual reality environment by a HTC vive system. The hand movements of the user control the DT in virtual reality and are additionally projected to a robot arm in real-time. Modern machine learning frameworks may eventually be able to segment semantic information of endoscopic images as stated by Scheikl et al. (2020); however, Ahmed and Devoto (2020) consider the modeling of tissue with deformations and movement as the main limitation of DT technologies in surgical robotics. Nevertheless DT-based assistance functions may very well be able to assist in training surgeons where the environment is well-defined by abstracted training pads. In other domains we have shown that physics simulations are able to yield semantic information about the environment as seen in Bauer et al. (2018).

3 Methods

The approach described in this work consists of three modules as visualized in Figure 2. That is, a parameterizable concept for Virtual Fixtures (VFs), the Shared Control Parametrization Engine (SCOPE), and a Digital Twin (DT) that provides the required semantic state information. The three modules are introduced in the following and explained on the basis of the benchmark scenarios for in the pilot study. The benchmark scenarios consist of two variations of a pick and place task. They are designed to act on the Nylon Twist training pad as introduced in Figure 1 and are conducted using the DLR MiroSurge surgical robotic platform (Schlenk et al., 2018; Seibold et al., 2018).

FIGURE 2

FIGURE 2. Overview of the three core components and their interaction within the proposed concept. Virtual fixtures (A) are introduced in Section 3.1. The parameters are determined by SCOPE (B), which is described in Section 3.2. SCOPE requires the task context, which is provided by the digital twin DT, (C), as described in Section 3.3.

For this work one 7-DoF light weight robotic arm called MIRO is mounted on the left standard side rail of the operating room table. The MIRO carries a wristed surgical instrument i whose additional 2-DoFs are actuated by a MICA drive unit. The pose of the instrument’s Tool-Center Point (TCP)Σ_TCP is telemanipulated by a Force Dimension’s sigma.7, a 6-DoF input device with an additional gripping DoF, offering the possibility to haptically render forces (Dimension, 2021). A second MIRO is mounted on the right side of the operating room table holding a stereo endoscope. In telemanipulation mode, a software layer integrates multiple avatar robots and maps the motion of up to two input devices to ${\mathit{\Sigma }}_{TC{P}_i}$ and the endoscope ${\mathit{\Sigma }}_{TC{P}_e}$ of the position-controlled robotic system. The mapping is based on an inverse kinematics, which preserves the motion constraints at the incision point, called trocar point Σ_trocar.¹

3.1 Virtual Fixtures

In the following paragraphs, some general design principles of VFs for a surgical robotic platform are explained followed by the description of the implemented VFs, which are divided into system related and parameterizable, task related VF.

VF generate a virtual wrench w_POI = [f_POI; m_POI], comprising the forces f_POI and torques m_POI, which act on the Pose of Interest (POI) Σ_POI. A virtual proxy Σ_proxy mirrors Σ_POI. The motion of Σ_proxy is restricted by the virtual object that describes the geometries of the VF, i.e. Σ_proxy stays on the surface of the virtual object while Σ_POI penetrates it. Σ_proxy is linked to Σ_POI by a six DoF spring damper system as explained by Caccavale et al. (1999). The displacement between Σ_POI and Σ_proxy generates w_POI depending on the stiffness K and the damping D. D is scaled by a function taking into account the distance between Σ_POI and Σ_proxy to avoid an abrupt increase of the damping wrench when a displacement occurs.

The resulting w_POI is rendered to the input device. If ${\mathit{\Sigma }}_{POI}={\mathit{\Sigma }}_{TC{P}_i}$, the wrench w_POI is scaled and transformed taking into account the hand-eye-coordination matrix (Tobergte, 2010; Steidle et al., 2014). However, VFs applied to the tool body (e.g., the wrist or shaft) or the robot’s joints require transforming the resulting wrenches to ${\mathit{\Sigma }}_{TC{P}_i}$ before rendering them to the input device. In a minimally invasive surgical setup, these transformations have to preserve the constraints at Σ_trocar. Therefore, an augmented task vector v_a and the respective augmented Jacobian J_a, combining the 6-DoF task at ${\mathit{\Sigma }}_{TC{P}_i}$ with the 2-DoF constraint at Σ_trocar are set up according to Siciliano and Khatib (2008).

Considering an 8-DoF robot (6-DoF robotic arm and a 2-DoF instrument) J_a is invertible, so an unique mapping between the joint torques τ and the Cartesian wrench ${\mathit{w}}_{TC{P}_i}$ exists. Thus, a unique mapping between q_robot and v_a, as well as between τ and the Cartesian wrench w_a exist. w_a comprises ${\mathit{w}}_{TC{P}_i}$ and the wrench w_trocar at Σ_trocar counteracting the trocar constraints. To map w_POI to ${\mathit{w}}_{TC{P}_i}$, it is first transformed to joint space by J_POI and finally back to the ${\mathit{\Sigma }}_{TC{P}_i}$ by

$\left(\begin{array}{c}\hfill {\mathit{w}}_{TC{P}_i}\hfill \\ \hfill {\mathit{w}}_{trocar}\hfill \end{array}\right)={\mathit{J}}_a^{-T}\mathit{\tau }={\mathit{J}}_a^{-T}{\mathit{J}}_{POI}^T{\mathit{w}}_{POI}.(1)$

With this general implementation several VFs can be realized and haptically rendered to the input device.

3.1.1 System Related VF

System related VFs are defined and parametrized by the restrictions that come with the robotic system e.g. the length of the instrument’s shaft or workspace limits of the hardware (in Cartesian and Joint space). For the pilot experiments a Cartesian workspace limit on the x-axis is set to the bottom of the trainings pad as visualized in Figure 3. According to Nitsch et al. (2012) the intuitiveness is enhanced when haptically guiding the user to avoid such limits by rendering the restrictions to the input devices.

FIGURE 3

FIGURE 3. The training pad used for the benchmark scenarios is shown. Σ_pad,start is the initial pose of the ring and Σ_pad,goal its goal pose. The blue cylinders depict the AFs. The yellow box depicts the RF and the red area visualizes the Cartesian workspace limit in z-axis. Together they help avoid collisions between the instrument and the trainings pad. (A) depicts the first benchmark scenario consisting of an unconstrained pick and place task. (B) displays the second benchmark scenario. The constrained pick and place task is realized by an sinus-shaped AF for the follow_path SCT. It is shown by the green curve.

3.1.2 Task Related VF

Repulsive VFs (RF) inside the workspace prevent the user from hitting a forbidden region. RFs are implemented using geometrical primitives e.g., virtual boxes as depicted on the right lower side in Figure 2. Their corresponding parameters P_rf are displayed in Table 1. When Σ_POI penetrates the virtual box, the Σ_proxy stays on the surface. Depending on the instrument and the obstacle geometry multiple Σ_POI can be defined along the instrument structure. For the pick skill RFs in the form of virtual boxes are implemented with ${\mathit{\Sigma }}_{POI}={\mathit{\Sigma }}_{TC{P}_i}$ to avoid crashes between ${\mathit{\Sigma }}_{TC{P}_i}$ and the training pad.

TABLE 1

TABLE 1. This table displays the corresponding parameters of RF and AF.

Attractive VFs (AF) guide the user towards or along a certain predefined path or target as visualized on the right upper side in Figure 2. AFs are implemented by a cylindrical geometry around the attractive pose Σ_af, where Σ_proxy is free to move along the z-axis. Along the remaining axes Σ_proxy is limited by a given radius, which is encoded in the translational dead zone td_af. The rotation of Σ_proxy is fixed to guide the orientation of Σ_POI with a parameterizable rotational dead zone rd_af. Table 1 presents an overview of the parameters of AFs P_af. The pick skill sets Σ_af to the middle point of a ring with td_af equal to the ring radius to align ${\mathit{\Sigma }}_{TC{P}_i}$ with the designated grasp pose. s_f and s_m, scaling the stiffness, are implemented as function of the distance d between Σ_af and Σ_POI (see Figure 4). This stiffness function results in an increasing wrench with decreasing d. This ensures gentle guidance that the user can easily overcome for large d and a strong guidance for small d. It was chosen empirically to provide intuitive and subtle guidance to the user.

FIGURE 4

FIGURE 4. The profile of the cylindrical AF at Σ_AF is displayed. Σ_TCP entered the cylinder, Σ_proxy stayed at its surface and a virtual spring damper system is spanned in between. The blue lines denote the shape of the stiffness K depending on the distance d along axial and radial axes of the cylinder. The gray areas depict the blending of the scaling functions.

The second benchmark scenario implements the follow path skill. It restricts the translation from the left side of the trainings pad to the right side by the shape of a polynomial function. The path following AF is implemented by updating Σ_af in every time step to slide along the predefined path. Σ_af is defined as the projection of Σ_POI onto the path with the smallest distance. s_m = 0 is set to let the user control the rotation of Σ_POI.

A virtual wall introduced at the gripping DoF of the input device assists the grasping of objects. The user feels a small opposing force at the point where the gripper of the instrument closes forcelessly. The user overcomes it by pressing slightly against it and the gripper closes firmly applying force on the grasped tissue. The grasp is loosened applying the same principle.

Different phases of a task may require distinct VF. Parameterizable VFs offer an interface to activate them and to adapt their parameters at run time. In the presented work the switching is realized by gradually fading down the initial wrench difference the involved VFs have at the time of switching. Switching between VFs may cause instabilities. While in practice instabilities aren’t observed, a time domain passivity controller as introduced in Hannaford and Ryu (2002) and further developed in Hertkorn et al. (2010), Ryu et al. (2005), and Hulin et al. (2021) is to be integrated into the presented approach in the future, to guarantee a safe operation for arbitrary tasks in all circumstances. Switching between VFs and their parametrization allows the flexible application of various VFs depending on the required task.

3.2 Shared Control Parametrization Engine

SCOPE provides a framework to parametrize VFs depending on the current task state. The SCOPE concept builds on our former work on Shared Control Templates (SCTs) (Quere et al., 2020). SCTs represent robot skills as a sequence of states leading to a certain goal state as depicted in the center of Figure 2. One SCT may encode different phases in order to guide the user through different phases of a task. In each phase different parameters can be set and different transition conditions apply (e.g., the distance between the end-effector and the target object).

In Quere et al. (2020), SCTs have been implemented to assist in tasks like pick and place, pouring, drinking, or opening doors and drawers. In this work, the portfolio is extended for MIRS tasks required to solve the Nylon Twist training pad from Lübeck Toolbox (see Figure 1), i.e., pick, follow_path, and place. Each of these SCTs describes the interaction between a reference object or a robot component O_ref and a target object O_tar as it is listed in Table 2.

TABLE 2

TABLE 2. Mapping of O_ref and O_tar for the individual SCTs implemented for MIRS assistance.

SCOPE progresses through the SCTs to parameterize haptically rendered assistance functionality as described in Algorithm 1. Until the goal state is reached SCOPE queries the actual state of the environment and the robot. It receives the current task context $\mathcal{C}$ to check if it is possible to progress to the next state. If a state transition is initialized, the new parameters P_rf, P_af, P_gripper encoded in this state are used to parametrize and activate VFs.

Algorithm 1. Pseudocode of the SCOPE main loop to parse SCTs and parameterize VFs.

Input: The SCT encoding the current task.

Output: The algorithm terminates nominal as all states of the SCT are processed.

1 state_id ← 0

2 state_goal ← getGoal (SCT)

3 O_ref, O_tar ← getReferenceAndTarget (SCT)

4 whilestate_id ≠ state_goaldo

5  $\mathcal{C}\leftarrow$getContextInformation (O_ref, O_tar)

6  state_curr ← getCurrentState ($\mathcal{C},SCT,stat{e}_{id}$)

7  ifstate_id ≠ state_currthen

8   P_rf, P_af, P_gripper ← getParameter (state_curr)

9   parametrizeVF(P_rf, P_af, P_gripper)

10  state_id ← state_curr

11 return True

The SCOPE approach relies on current updates of Σ_ref, Σ_tar, w_ref,tar, and q_gripper which are included in the task context $\mathcal{C}$ provided by the function getContextInformation. A physics-based state inference utilizing the proposed DT model is used to generate the information provided by getContextInformation. This extends the SCT approach which infers the required information through visual perception only.

3.3 Digital Twin

The DT approach offers the possibility to generate the required information about the world and the current task and to fuse it with data about the robot obtained by different sources. The DT is divided into a Robot-centric DT (RDT) and an Object-centric DT (ODT) which constantly exchange information as shown on the left in Figure 2.

The RDT mirrors the DLR Mirosurge system by interfacing its telemanipulation layer. It provides information about the interaction of the different robots. Each MIRO robot generates data at different granularity levels like sensor readings, joint angles and control states. Additionally, information about the relations of the different robots, their workspaces and the poses of their bases are gathered. The RDT offers an interface to select information needed for a task and to provide it for further processing. A middleware provides the infrastructure to query all this information in real time. A viewer displays all relevant information and, therefore, serves as a visualization of the RDT. In Figure 2 the RDT of the DLR MiroSurge system is displayed on the top left. For the pilot study it gathers information about ${\mathit{\Sigma }}_{TC{P}_i}$ and can acquire further information about the robot.

The ODT provides information about the world state. It makes use of the world state representation as presented in Leidner (2019), which provides an interface to instantiate the robot’s believe about the current state of the world and query this information. It has knowledge about the present objects and their poses. The objects’ poses Σ_o are dynamically updated. The information about Σ_o can be provided by an imaging system. However, reasoning about the current state of a task not only requires knowledge about the involved objects and their poses, but also about their interactions. Interactions between objects are encoded in semantic states. E.g. “ring is grasped by gripper” requires information about interaction wrenches between the ring and the gripper. In this work we use a physics simulation to keep track of Σ_o of the objects of interest as it gives us the possibility to generate interaction wrenches between ${\mathit{\Sigma }}_{TC{P}_i}$ and Σ_o and derive semantic states.

Pre-known information about a variety of objects is provided by the object database, presented in Leidner (2019), offering a hierarchical structure for objects with respect to their functionality. Physical objects are derived from high level abstract object definitions which provide not only information about the object but also about actions. These can be refined for certain types of the abstract objects. The physics simulation queries the meshes of present objects from the object database and receives their poses from the world state representation. Then it instantiates the objects that are currently present in the robot’s environment. Furthermore, it loads the instrument’s gripper and places it at ${\mathit{\Sigma }}_{TC{P}_i}$, which is acquired from the RDT and updated in real-time (1 kHz). Σ_o of all objects can be simulated and is provided to the world state representation. Additionally, this allows the computation of interaction wrenches w_ref,tar between O_ref and O_tar, e.g., between the instrument and different objects, and the deduction of semantic states. Figure 2 shows the visualization of the ODT on the bottom left.

The proof-of-concept implementation uses AMBF physics simulation engine (Munawar et al., 2019) which integrates the Bullet Dynamics Engine (Coumans, 2015) as a dynamic solver and interfaces of CHAI-3D (Conti, 2003). This simulation framework was developed to meet the requests of closed-loop kinematic chains and redundant mechanisms as often used in surgical robotics.

This DT approach enables the implementation of the getContextInformation function as explained in Algorithm 2. It decides if O_ref is an object or a robot component and returns the accumulated context information $\mathcal{C}$ accordingly.

Algorithm 2. Pseudocode of the getContextInformation function

Input: The reference object/robot O_ref and the target object O_tar.

Output: The algorithm returns context information $\mathcal{C}$ from the combined DT state.

1 ifO_ref is robot then

2  q_gripper, Σ_ref ← getUpdateRDT (O_ref)

3  Σ_tar, w_ref,tar ← getUpdateODT (O_tar)

4  $\mathcal{C}\leftarrow {\mathit{q}}_{gripper},{\mathit{\Sigma }}_{ref},{\mathit{\Sigma }}_{tar},{\mathit{w}}_{ref,tar}$

5 else

6  Σ_ref, Σ_tar, w_ref,tar ← getUpdateODT (O_ref, O_tar)

7  $\mathcal{C}\leftarrow {\mathit{\Sigma }}_{ref},{\mathit{\Sigma }}_{tar},{\mathit{w}}_{ref,tar}$

8 return$\mathcal{C}$

4 Evaluation

The presented approach is evaluated in a pilot study. The experimental setup is described in subsection 4.1. The results are presented in subsection 4.2.

4.1 Experiment Setup

The evaluation is conducted by means of the Nylon Twist training introduced in Figure 1. Two benchmark scenarios are derived as visualized in Figure 4. First, an unconstrained pick and place task ([a]) and second, a constrained pick and place task ([b]). A ring is initially placed on the training pad at the start pose Σ_pad,start. The subjects are tasked to pick up this ring and place it on the goal pin located at Σ_pad,goal. In benchmark one, the transition between Σ_pad,start and Σ_pad,goal is not constrained as shown in 4 [a]. The benchmark scenario two restricts the transition from Σ_pad,start to Σ_pad,goal to a sine-shaped path as shown in 4 [b]. The predefined path is marked on the real training pad for visual guidance.

The gripper of the instrument is simulated in the ODT at ${\mathit{\Sigma }}_{TC{P}_i}$. The RDT updates ${\mathit{\Sigma }}_{TC{P}_i}$ with a rate of 1 kHz. The training pad is located at Σ_pad. In the beginning of each trial, the ring is located at Σ_ring = Σ_pad,start in the real world and, therefore, also in the ODT. The physical state of the Σ_ring is simulated by the ODT based on the interaction wrenches between the ring, the environment (i.e., the training pad), and the gripper. Altogether, the DT provides ${\mathit{\Sigma }}_{TC{P}_i}$ and Σ_ring, the start and goal position of the ring Σ_pad,start and Σ_pad,goal, and the gripper angle q_gripper. Furthermore, the interaction wrenches ${\mathit{w}}_{ref,tar}={\mathit{w}}_{TC{P}_i,ring}$ between ${\mathit{\Sigma }}_{TC{P}_i}$ and Σ_ring are generated by the physic simulation.

The proof-of-concept implementation of SCOPE provides three SCTs, namely pick, place, and follow_path as shown in Figure 5. References and targets are visualized in Table 2. The implemented AFs support the successful completion of the task. The Cartesian workspace limit and the RF prevent collisions between the instrument and the training pad. The pick SCT is visualized in Figure 5A with the transition requirements and the corresponding parameters. The state init is the entry point of the pick SCT. If there is no wrench ${\mathit{w}}_{TC{P}_i}$, the gripper is empty and the transition to state start is performed. The transition to state close is constrained by the distance d, defined between ${\mathit{\Sigma }}_{TC{P}_i}$ and ${\mathit{\Sigma }}_{TC{P}_{ring}}$, and q_gripper. The state grasped is reached if d_z, the distance along the z-axis, is sufficiently small, the gripper is closed and ${\mathit{w}}_{TC{P}_i,ring}$ exceed a threshold. The place SCT is implemented very similar to the pick SCT and makes use of the same VFs. It is visualized in Figure 5B, but not described in more detail here. After successfully picking a ring the follow_path SCT is activated as depicted in Figure 5C. The first state of the SCT is grasped. If d fulfills all criteria the transfer state is reached and a path following AF (PF) is set. If d_x indicates that ${\mathit{\Sigma }}_{TC{P}_i}$ is at the right side of the pad the at_target state is reached and the follow_path SCT finishes.

FIGURE 5

FIGURE 5. States, transitions and the respective parameters describing the pick SCT (A), the place SCT (B), and the follow_path SCT (C). The blue states indicate the starting states of the respective SCT. The yellow information are provided by the RDT, the green by the ODT and the black are computed as a combination of ODT and RDT.

Each of the two benchmark scenarios starts with the pick SCT as the gripper is initially empty. The first benchmark progresses with the place SCT, while the second benchmark transitions to the follow_path SCT after grasping the ring successfully. As the distance between Σ_ring and Σ_pin becomes sufficiently small, the place SCT is activated as last task of both benchmark scenarios.

Six subjects performed both benchmark scenarios. They are split in two groups namely a test group and a control group. They have little to no experience with telemanipulation of the DLR MiroSurge System. Before starting with the benchmark scenarios a short introduction to the system is provided with a few introduction movements. The test group performed trials one and five of each benchmark scenario without assistance and the remaining trials with assistance. The control group performed each benchmark five times without assistance to investigate the hypothesis, that the proposed approach accelerates training. After each trial the participant is asked to rate the current workload based on the Overall Workload Scale (Vidulich and Tsang, 1987).

4.2 Results

To investigate the hypothesis that training with our approach enables a novice surgeon to realize a task faster, more accurately and with less cognitive load, the following metrics are presented: duration of a task Δt, translational and rotational deviations from the predefined pose Δp, consisting of Δp_trans and Δp_rot, and the assessed workload. This section provides the pure results, a detailed interpretation of the results is provided in Section 4.3.

4.2.1 Pose Deviation

The activated AFs attract ${\mathit{\Sigma }}_{TC{P}_i}$ towards the predefined pose by presenting a wrench w in each time step. We investigate Δp analyzing the recorded wrenches. For trials without assistance presented to the subjects, the VFs are calculated and used as evaluation metric but not haptically rendered to the input device and not affecting the robot’s movements.

The recorded wrenches are analyzed separately for each skill of the benchmark scenarios. Furthermore, the analysis splits wrenches into forces f and torques m. The absolute force f_t measured in one time step is calculated from the forces along the x, y, z-axis by the euclidean norm. For each subject all f_t measured during one skill of one trial are accumulated to f_a. f_a is normalized over all subjects and trials and over both benchmarks within one skill. Therefore, the results within one skill are comparable over both user groups and both benchmarks. Furthermore, the pick and place skills are normalized by the same value to make them comparable as well. The mean and standard deviation over all subjects’ normalized deviations is acquired separately for each of the trials. The recorded torques are processed equally. Figure 6 shows the processed deviations for both benchmark scenarios.

FIGURE 6

FIGURE 6. The translational and rotational deviations for benchmarks one (A) and two (B) are shown. The left column depicts results of the test group, the right column of the control group. Blue bars display trials with assistance, green ones trials without assistance. The standard deviation is depicted by black arrows.

4.2.2 Duration

The duration of completing a task is analyzed for each participant. Figure 7 shows the mean duration and standard deviation over all subjects for each trial for both benchmarks and both groups.

FIGURE 7

FIGURE 7. The duration of each tasks in benchmark scenario one (A) and benchmark scenario two (B) are depicted. The test group is displayed in the left column, the control group in the right column. Trials with assistance are shown in blue, trials without assistance in green. The standard deviation is depicted by black arrows.

4.2.3 Workload

The workload is assessed after each trail with the Overall Workload Scale. The subjects are asked to rate their workload on a scale from 0 (very low) to 20 (very high) after each trial. The evaluation of the different tasks of a benchmark scenario is combined into one value. The mean and standard deviation is acquired by averaging over all participants but separating the different trials of a benchmark scenario. Figure 8 shows the mean and standard deviation over all subjects for each trial of both benchmark scenarios and both groups.

FIGURE 8

FIGURE 8. The workload for both benchmarks is depicted, comparing the test group (A) and the control group (B). Blue bars depict trials with assistance, green bars trials without assistance. The standard deviation is shown by black arrows.

4.3 Performance Analysis

The small number of subjects included in this pilot study allows us to identify trends in how the proposed approach influences pose deviation, duration until benchmark scenario completion, and subjective workload. As each of the involved groups include only three participants the trends are presented in the form of plots rather then with a statistical analysis of the significance levels. The pilot study therefore does not aim to draw statistically significant conclusions, but instead to provide an initial assessment of feasibility.

4.3.1 Pose Deviation

The analysis of the pose deviation aims to find trends investigating if our approach helps a novice surgeon to realize a task more accurately. The expected result is that after training with our approach the measured deviations of the predefined path are smaller than without such training represented by the control group. Figure 6 shows the processed deviations for both benchmark scenarios. The test group is depicted in the left column, the control group in the right.

In general, three main observations can be made. First, it is visible that the test group performs more accurate for some task if the assistance functions are activated (see trials with blue bars in Figure 6).

Second, the test group performs more accurate with respect to rotational deviation by means of absolute numbers, whereas the translational deviation is mostly comparable over all tasks. Third, comparing the performance (both translational and rotational) over all trials and all tasks of both benchmarks shows a clear trend: That is, for each task, the gradient on the mean deviation over the five trials is decreasing or constant for the test group, whereas it is increasing or constant for the control group.

Investigating the tasks individually, it is visible that especially the path_following SCT is providing assistance. For the trials in which the test group is assisted by means of haptic guidance, the accuracy is increased by 60%.

In addition, a learning effect can be observed for the test group, while the control group does not show a training effect. This training effect is further analyzed statistically by performing a two-sided related t-test comparing the first with the last trial for each task within each group. A two sided p-value p < 0.05 will be considered statistically significant. The resulting p reveal that the rotational deviation for the pick skill causes a statistically significant learning effect (p = 0.009). The remaining p are below the considered significance level. Considering that the statistical power of t-tests increases with the size of a data set, the result is deemed very promising and will be further investigated. The pick and place skills do not show a clear trend in the mean pose deviation. However, it is visible that the standard deviation is positively affected by the assistance functions for both tasks concerning translations and rotations.

In summary, the analysis of pose deviation shows that assistance functions have a certain impact on training for some tasks. Furthermore, the steady increase in pose deviation for some skills across both benchmarks for the control group shows that accuracy is further compromised as more repetitions are performed. This may already be an indication of increased workload, which will be discussed further at the end of this section.

4.3.2 Duration

The duration of task execution is evaluated to investigate if SCOPE supports a novice surgeon to complete a required task faster. In comparison to the pose deviation, no general observation can be made with respect to this hypothesis.

Nevertheless, Figure 7 reveals that the test group improved the duration to complete the pick task, both with respect to mean values but as well with respect to the standard deviation. The place and path_following skills do not show clear effects in duration for both groups.

4.3.3 Workload

The workload is assessed to investigate if SCOPE decreases the workload of a novice surgeon during task execution. The results shown in Figure 8 supports this hypothesis. In general, the analysis reveals that the perceived workload of the participants decreases when offering task dependent assistance functions. After training with our approach the execution of the required skills without assistance requires less workload then without training, which is also visible in detail when analyzing the two benchmark scenarios in detail.

Benchmark one shows that the workload decreases in the test group when repeating the same task multiple times. After some trials with assistance the workload for the trial without assistance is decreased compared to the first trial without assistance. The control group rates the workload for all trials in benchmark with a comparable value.

Benchmark two also shows that the trials with assistance for the test group are rated with less workload then the trials without assistance. The training effect is smaller comparing trial 1 and 5 without assistance. The control group rates do not show a learning effect in the workload for benchmark two.

5 Conclusion

This paper presents a novel approach to parameterize haptic assistance functions at run-time, entitled SCOPE. The procedural context to do so, is derived utilizing a DT approach. The concept is integrated into the DLR MiroSurge system and evaluated in a pilot study for surgical training. The experiments show a promising trend, that novel users profit from haptic augmentation during training of certain tasks, such that they perform a task faster, more accurately and with less cognitive load.

5.1 Scalability and Generalizability

Even though the pilot study is conducted with only a small amount of participants, it reveals promising trends with respect to the learning effect. In a next step, a follow-up user study will be conducted with partnering novice surgeons. This will allow the recording of a sufficiently large data set to analyze the statistical significance of the trends presented in this work.

As the portfolio of available skills in terms of SCTs is currently predefined by a technical expert, the amount of available skills is limited. The skill set will be extended by aligning the set of training tasks to established curricula (Ritter and Scott, 2007; Laubert et al., 2018; Chen et al., 2020), e.g., by introducing peg-transfer, cutting along resection lines, needle guidance and suturing. In addition, the available assistance functionality will be extended by parameterizable assistance functions such as motion scaling between input device and ${\mathit{\Sigma }}_{TC{P}_i}$.

The stiffness function parametrizing AF visualized in Figure 3 influences the surgeon’s haptic perception of the guidance. The preferred guidance mode might differ between surgeons and most likely influences training outcome. Therefore, we plan to automatically generate the stiffness function as extension to the empirically chosen function currently implemented for both benchmarks. Two approaches will be compared in the future improving different aspects of the haptic perception of VFs. First, the generation of the stiffness function will be based on trials performed by expert surgeons which describe the optimal stiffness. The second approach adapts the stiffness function on the surgeon’s preferences learned from previous trials.

The presented approach focuses on the task state estimation of a known task series. This approach could be extended by surgical gesture recognition inferring which task the surgeon aims to complete (van Amsterdam et al., 2021). This would allow for more flexible task series.

Furthermore it is crucial to note that the initial pose of all relevant objects in the scene is currently predefined. After initialization the objects’ poses are tracked by the physics simulation. The presented tasks consist of short sequences to prevent the divergence of the simulated scene from the real scene. Longer action sequences will require the integration of a vision pipeline to further prevent divergence of the simulated world and the real world. This includes object detection and tracking to allow for a more generic scene initialization and the update of the parameters, i.e., pose of the objects, used by the physics simulation. Additionally, the integration of a vision pipeline into our approach will enhance the flexibility of our setup.

5.2 Outlook

Based on the insights of this work, we argue that adaptable haptically augmented VFs explain a task more accurately than oral or visual explanations. In the future, the tasks to be trained could be learned from expert demonstrations (Steinmetz et al., 2019). To achieve this, an expert surgeon could perform a task first, allowing the abstraction of skills and corresponding support functions. A novice surgeon could then benefit from a haptically augmented training process that encodes the expert knowledge inherently.

As SCOPE allows the adaptable parametrization of VFs, it enables the personalization of VFs to the surgeons’ needs during the different phases of training. With increasing proficiency, the strength of the haptic guidance could be reduced to further train the novice surgeons’ ability follow the predefined path by their own (Enayati et al., 2018). According to Ovur et al. (2019) decreasing the strength of guidance and finally removing it doesn’t affect the achieved skill level. Furthermore, VFs could be parametrized according to the surgeon’s preferences observed by the robotic system.

At last, we envisage the possibility to train critical steps before surgery in virtual reality. Assistance functions could be predefined in virtual reality and applied during real surgery. However, the automatic perception of procedural context information in real surgery is still an open research question. Perceiving the procedural context would allow to integrate SCOPE into real surgical procedures.

This outlook leads to the final goal of supporting surgeons during MIRS with individualized contextual assistance and therefore enhancing surgical performance.

Data Availability Statement

The datasets presented in this study can be found using the following link: https://github.com/DLR-RM/Contextual-MIRS-Assistance.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author Contributions

KH, AH, RR, JK, FS, and DL contributed to conception and design of the presented approach. KH, AH, and RR implemented the approach. KH, AH, JK, FS, and DL contributed to conception and design of the presented pilot study. KH evaluated the results. KH wrote the first draft of the article. KH, AH, JK, and DL wrote sections of the article. All authors contributed to article revision, read, and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The Authors would like to gratefully acknowledge the financial support and endorsement from the DLR Management Board Young Research Group Leader Program and the Executive Board Member for Space Research and Technology.

Footnotes

¹For more details on the control of the system, please refer to Le Tien et al. (2007) and Le Tien et al. (2008) for the MIRO, to Thielmann et al. (2010) for the MICA drive unit and to Tobergte et al. (2009) and Seibold et al. (2018) for the overall system control.

References

Abbott, J. J., Marayong, P., and Okamura, A. M. (2007). “Haptic Virtual Fixtures for Robot-Assisted Manipulation,” in Robotics Research (Springer), 49–64.

Google Scholar

Ahmed, H., and Devoto, L. (2020). The Potential of a Digital Twin in Surgery. Surg. Innov. 28, 509–510. doi:10.1177/1553350620975896

PubMed Abstract | CrossRef Full Text | Google Scholar

Bauer, A. S., Schmaus, P., Albu-Schäffer, A., and Leidner, D. (2018). “Inferring Semantic State Transitions during Telerobotic Manipulation,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 1–9. doi:10.1109/iros.2018.8594458

CrossRef Full Text | Google Scholar

Bowyer, S. A., Davies, B. L., and Rodriguez y Baena, F. (2014). Active Constraints/Virtual Fixtures: A Survey. IEEE Trans. Robot. 30, 138–157. doi:10.1109/TRO.2013.2283410

CrossRef Full Text | Google Scholar

Caccavale, F., Natale, C., Siciliano, B., and Villani, L. (1999). Six-dof Impedance Control Based on Angle/axis Representations. IEEE Trans. Robot. Automat. 15, 289–300. doi:10.1109/70.760350

CrossRef Full Text | Google Scholar

Chen, R., Armijo, P. R., Rodrigues Armijo, P., Krause, C., Siu, K.-C., and Oleynikov, D. (2020). A Comprehensive Review of Robotic Surgery Curriculum and Training for Residents, Fellows, and Postgraduate Surgical Education. Surg. Endosc. 34, 361–367. doi:10.1007/s00464-019-06775-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Conti, F. (2003). The CHAI Libraries. Tech. Rep.

Google Scholar

Coumans, E. (2015). Bullet Physics Simulation. ACM SIGGRAPH 2015 Courses 1. doi:10.1145/2776880.2792704

CrossRef Full Text | Google Scholar

Dimension, F. (2021). Force Dimension sigma.7. Available at: https://www.forcedimension.com/products/sigma (Accessed 06 27, 2021).

Google Scholar

Enayati, N., Ferrigno, G., and De Momi, E. (2018). Skill-based Human-Robot Cooperation in Tele-Operated Path Tracking. Auton. Robot 42, 997–1009. doi:10.1007/s10514-017-9675-4

CrossRef Full Text | Google Scholar

Endsley, M. R., and Kaber, D. B. (1999). Level of Automation Effects on Performance, Situation Awareness and Workload in a Dynamic Control Task. Ergonomics 42, 462–492. doi:10.1080/001401399185595

PubMed Abstract | CrossRef Full Text | Google Scholar

Fontanelli, G. A., Yang, G.-Z., and Siciliano, B. (2018). “A Comparison of Assistive Methods for Suturing in MIRS,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (Madrid: IEEE), 4389–4395. doi:10.1109/iros.2018.8593607

CrossRef Full Text | Google Scholar

Haidegger, T. (2019). Autonomy for Surgical Robots: Concepts and Paradigms. IEEE Trans. Med. Robot. Bionics 1, 65–76. doi:10.1109/tmrb.2019.2913282

CrossRef Full Text | Google Scholar

Hannaford, B., and Jee-Hwan Ryu, J.-H. (2002). Time-domain Passivity Control of Haptic Interfaces. IEEE Trans. Robot. Automat. 18, 1–10. doi:10.1109/70.988969

CrossRef Full Text | Google Scholar

Hertkorn, K., Hulin, T., Kremer, P., Preusche, C., and Hirzinger, G. (2010). “Time Domain Passivity Control for Multi-Degree of freedom Haptic Devices with Time Delay,” in 2010 IEEE International Conference on Robotics and Automation (IEEE), 1313–1319. doi:10.1109/robot.2010.5509148

CrossRef Full Text | Google Scholar

Hulin, T., Panzirsch, M., Singh, H., Balachandran, R., Coelho, A., Pereira, A., et al. (2021). Model-augmented Haptic Telemanipulation: Concept, Retrospective Overview and Current Use-Cases. Front. Robotics AI 8, 76. doi:10.3389/frobt.2021.611251

CrossRef Full Text | Google Scholar

Jee-Hwan Ryu, J.-H., Preusche, C., Hannaford, B., and Hirzinger, G. (2005). Time Domain Passivity Control with Reference Energy Following. IEEE Trans. Contr. Syst. Technol. 13, 737–742. doi:10.1109/tcst.2005.847336

CrossRef Full Text | Google Scholar

Klodmann, J., Schlenk, C., Borsdorf, S., Unterhinninghofen, R., Albu-Schäffer, A., and Hirzinger, G. (2020). Robotische Assistenzsysteme für die Chirurgie. Chirurg 91, 533–543. doi:10.1007/s00104-020-01205-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Laaki, H., Miche, Y., and Tammi, K. (2019). Prototyping a Digital Twin for Real Time Remote Control over Mobile Networks: Application of Remote Surgery. IEEE Access 7, 12. doi:10.1109/access.2019.2897018

CrossRef Full Text | Google Scholar

Laubert, T., Esnaashari, H., Auerswald, P., Höfer, A., Thomaschewski, M., Bruch, H. P., et al. (2018). Conception of the Lübeck Toolbox Curriculum for Basic Minimally Invasive Surgery Skills. Langenbecks Arch. Surg. 403, 271–278. doi:10.1007/s00423-017-1642-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Le Tien, L., Albu-Schaffer, A., De Luca, A., and Hirzinger, G. (2008). “Friction Observer and Compensation for Control of Robots with Joint Torque Measurement,” in 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE), 3789–3795.

CrossRef Full Text | Google Scholar

Le Tien, L., Schaffer, A. A., and Hirzinger, G. (2007). “Mimo State Feedback Controller for a Flexible Joint Robot with strong Joint Coupling,” in Proceedings 2007 IEEE International Conference on Robotics and Automation (IEEE), 3824–3830. doi:10.1109/robot.2007.364065

CrossRef Full Text | Google Scholar

Leidner, D. S. (2019). Cognitive Reasoning for Compliant Robot Manipulation. Springer.

Li, Z., Gordon, A., Looi, T., Drake, J., Forrest, C., and Taylor, R. H. (2020). Anatomical Mesh-Based Virtual Fixtures for Surgical Robots. arXiv:2006.02415 [cs, eess].

Llc, M. M. (2021). Trainingsmodule Lübbecker Toolbox. Available at: http://www.luebeck-toolbox.com/training.html (Accessed 06 27, 2021).

Google Scholar

Maier, J., Perret, J., Huber, M., Simon, M., Schmitt-Rüth, S., Wittenberg, T., et al. (2019). Force-feedback Assisted and Virtual Fixtures Based K-Wire Drilling Simulation. Comput. Biol. Med. 114, 103473. doi:10.1016/j.compbiomed.2019.103473

PubMed Abstract | CrossRef Full Text | Google Scholar

Munawar, A., Wang, Y., Gondokaryono, R., and Fischer, G. S. (2019). “A Real-Time Dynamic Simulator and an Associated Front-End Representation Format for Simulating Complex Robots and Environments,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 1875–1882. doi:10.1109/iros40897.2019.8968568

CrossRef Full Text | Google Scholar

Nagy, T. D., and Haidegger, T. (2019). A DVRK-Based Framework for Surgical Subtask Automation. Acta Polytechnica Hungarica 16, 18. doi:10.12700/aph.16.8.2019.8.5

CrossRef Full Text | Google Scholar

Nitsch, V., Färber, B., Hellings, A., Jörg, S., Tobergte, A., and Konietschke, R. (2012). “Bi-modal Assistance Functions and Their Effect on User Perception and Movement Coordination with Telesurgery Systems,” in Proceedings of 2012 IEEE International Workshop on Haptic Audio Visual Environments and Games (HAVE 2012) (IEEE), 32–37. doi:10.1109/have.2012.6374427

CrossRef Full Text | Google Scholar

Ovur, S. E., Cobanaj, M., Vantadori, L., De Momi, E., and Ferrigno, G. (2019). “Surgeon Training with Haptic Devices for Computer and Robot Assisted Surgery: An Experimental Study,” in Mediterranean Conference on Medical and Biological Engineering and Computing (Springer), 1526–1535. doi:10.1007/978-3-030-31635-8_189

CrossRef Full Text | Google Scholar

Qian, L., Wu, J. Y., DiMaio, S. P., Navab, N., and Kazanzides, P. (2019). A Review of Augmented Reality in Robotic-Assisted Surgery. IEEE Trans. Med. Robotics Bionics 2, 1–16.

Google Scholar

Quere, G., Hagengruber, A., Iskandar, M., Bustamante, S., Leidner, D., Stulp, F., et al. (2020). “Shared Control Templates for Assistive Robotics,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), 1956–1962. doi:10.1109/icra40945.2020.9197041

CrossRef Full Text | Google Scholar

Ritter, E. M., and Scott, D. J. (2007). Design of a Proficiency-Based Skills Training Curriculum for the Fundamentals of Laparoscopic Surgery. Surg. Innov. 14, 107–112. doi:10.1177/1553350607302329

PubMed Abstract | CrossRef Full Text | Google Scholar

Rosenberg, L. B. (1993). “Virtual Fixtures: Perceptual Tools for Telerobotic Manipulation,” in Proceedings of IEEE virtual reality annual international symposium (IEEE), 76–82. doi:10.1109/vrais.1993.380795

CrossRef Full Text | Google Scholar

Scheikl, P. M., Laschewski, S., Kisilenko, A., Davitashvili, T., Müller, B., Capek, M., et al. (2020). Deep Learning for Semantic Segmentation of Organs and Tissues in Laparoscopic Surgery. Curr. Dir. Biomed. Eng. 6. doi:10.1515/cdbme-2020-0016

CrossRef Full Text | Google Scholar

Schlenk, C., Bahls, T., Tarassenko, S., Klodmann, J., Bihler, M., and Wuesthoff, T. (2018). Robot Integrated User Interface for Physical Interaction with the DLR MIRO in Versatile Medical Procedures. J. Med. Robot. Res. 03, 1840006. doi:10.1142/S2424905X18400068

CrossRef Full Text | Google Scholar

Sealy, W. C. (1999). Halsted Is Dead: Time for Change in Graduate Surgical Education. Curr. Surg. 56, 34–39. doi:10.1016/s0149-7944(99)00005-7

CrossRef Full Text | Google Scholar

Seibold, U., Kübler, B., Bahls, T., Haslinger, R., and Steidle, F. (2018). “THE DLR MIROSURGE SURGICAL ROBOTIC DEMONSTRATOR,” in The Encyclopedia of Medical Robotics. Editor R. Patel, 111–142. doi:10.1142/9789813232266_0005

CrossRef Full Text | Google Scholar

Selvaggio, M., Fontanelli, G. A., Ficuciello, F., Villani, L., and Siciliano, B. (2018). Passive Virtual Fixtures Adaptation in Minimally Invasive Robotic Surgery. IEEE Robot. Autom. Lett. 3, 3129–3136. doi:10.1109/LRA.2018.2849876

CrossRef Full Text | Google Scholar

Siciliano, B., and Khatib, O. (2008). Springer Handbook of Robotics. Springer Science & Business Media.

Steidle, F., Hellings, A., Klodmann, J., Schlenk, C., Tarassenko, S., and Kübler, B. (2014). “Adaption of the DLR MiroSurge System’s Kinematics to Surgical Table Based Patient Repositioning in Minimally Invasive Surgery,” in Tagungsband der 13. Jahrestagung der Deutschen Gesellschaft für Computer-und Roboterassistierte Chirurgie (CURAC), 4.

Google Scholar

Steinmetz, F., Nitsch, V., and Stulp, F. (2019). Intuitive Task-Level Programming by Demonstration through Semantic Skill Recognition. IEEE Robot. Autom. Lett. 4, 3742–3749. doi:10.1109/lra.2019.2928782

CrossRef Full Text | Google Scholar

Thielmann, S., Seibold, U., Haslinger, R., Passig, G., Bahls, T., Jörg, S., et al. (2010). “Mica-a New Generation of Versatile Instruments in Robotic Surgery,” in 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE), 871–878. doi:10.1109/iros.2010.5649984

CrossRef Full Text | Google Scholar

Thornblade, L. W., and Fong, Y. (2021). Simulation-based Training in Robotic Surgery: Contemporary and Future Methods. J. Laparoendoscopic Adv. Surg. Tech. 31, 556–560. doi:10.1089/lap.2021.0082

CrossRef Full Text | Google Scholar

Tobergte, A., Konietschke, R., and Hirzinger, G. (2009). “Planning and Control of a Teleoperation System for Research in Minimally Invasive Robotic Surgery,” in 2009 IEEE International Conference on Robotics and Automation (IEEE), 4225–4232. doi:10.1109/robot.2009.5152512

CrossRef Full Text | Google Scholar

Tobergte, A. (2010). MiroSurge-Advanced User Interaction Modalities in Minimally Invasive Robotic Surgery. Presence: Teleoperators and Virtual Environments 19, 400–414. doi:10.1162/pres_a_00022

CrossRef Full Text | Google Scholar

van Amsterdam, B., Clarkson, M., and Stoyanov, D. (2021). Gesture Recognition in Robotic Surgery: a Review. IEEE Trans. Biomed. Eng. 99, 1. doi:10.1109/tbme.2021.3054828

CrossRef Full Text | Google Scholar

Vidulich, M. A., and Tsang, P. S. (1987). Absolute Magnitude Estimation and Relative Judgement Approaches to Subjective Workload Assessment. Proc. Hum. Factors Soc. Annu. Meet. 31, 1057–1061. doi:10.1177/154193128703100930

CrossRef Full Text | Google Scholar

Weber, B., Hellings, A., Tobergte, A., and Lohmann, M. (2013). “Human Performance and Workload Evaluation of Input Modalities for Telesurgery,” in Proceedings of the German Society of Ergonomics (GfA) Spring Congress (Dortmund: GfA Press).

Google Scholar

Zhang, D., Xiao, B., Huang, B., Zhang, L., Liu, J., and Yang, G.-Z. (2018). A Self-Adaptive Motion Scaling Framework for Surgical Robot Remote Control. IEEE Robotics Automation Lett. 4, 359–366.

Google Scholar