A Human Gesture Mapping Method to Control a Multi‐Functional Hand for Robot‐Assisted Laparoscopic Surgery: The MUSHA Case

1 Introduction

In minimally invasive surgery (MIS), surgical instruments are introduced into the patient’s body through small incisions, this practice allows to reduce trauma. As scientific research has amply demonstrated, MIS offers several benefits such as reduced pain, limited post-operative course, and short patient recovery times (Perigli et al., 2008; Sood et al., 2017; Hanley et al., 2019; Chadi et al., 2020). On the other hand, 1) the surgeon loses dexterity due to the limited range of movement of the instruments and their limited number of degrees of freedom (DoFs), 2) there is a reduction of tactile sensations with a consequent risk of tissue damage, and 3) the hand-eye coordination decrease. Nowadays, according to Tonutti et al. (2017), such of these issues are addressed by the introduction of technology in surgical rooms. Indeed, robotics has proven to be a viable solution to mitigate the limitations associated with the MIS. Rapid advances in image systems and computer technologies have led to significant advances in robotics applied to medicine. Robot-assisted surgery is now a common practice for several surgical operations and will probably become the most exploited surgical procedure of the future (Song et al., 2009; Parisi et al., 2017; Kulaylat et al., 2018; Sheetz et al., 2020). In this context, the da Vinci robot (Intuitive Surgical Inc.) demonstrated effective recovery of hand-eye coordination and motion dexterity. Research done in the last decade has been focused on overcoming existing deficiencies of robotic surgery systems, e.g., lack of haptic feedback (Okamura, 2009; Manganelli et al., 2010; Meli et al., 2014; Mohareri et al., 2014), enhancement of system integration (Handini et al., 2004; Leven et al., 2005; Takács et al., 2015), and combination of augmented reality navigation functionalities (Sugimoto et al., 2010; Yamamoto et al., 2012; Tabrizi and Mahvash, 2015). Other research and development activities aim to realize outstanding training systems, including the next generation of virtual reality simulators (Van der Meijden et al., 2010; Fontanelli et al., 2018).

More than one thousand publications are available in literature discussing medical robots control approaches and surgeon side console (Beasley, 2012). A bilateral control, with position-position architecture, is the standard approach to control short distance tele-robotic systems for surgery, according to Niemeyer and Slotine (1991). For instance, in setups equipped with the da Vinci robot the position of the surgeon’s hand is mapped into the position of the surgery tool and the foot-pedal enables a temporary disconnection between the two sides to reconfigure the control manipulator into a more comfortable and ergonomic pose. For some applications, impedance control and switched-impedance control are used as a valid alternative (Cheng and Tavakoli, 2018). For instance, a strategy exploiting force based model predictive control for beating heart surgery is presented by Bebek and Cavusoglu (2006). These approaches are usually reinforced by identification techniques of robotic arms as described in Fontanelli et al. (2017), and by techniques of shared autonomy as motion compensation, collision avoidance (Moccia et al., 2020), active constrains, and shared task space (Bowyer et al., 2013).

An innovative surgery control console is presented in Sukhoon et al. (2009), the developed robot controller fully replicate the reference movements. Similarly, in Kim et al. (2013) the authors propose a control device mechanically similar to the robot end effector, coupled with a system for regulating the joints stiffness in accordance with the exerted force. On this line, new consoles are designed to recover tactile perception integrating haptic technologies. For instance, (Yin et al., 2015) used magnetorheological fluids to return haptic sensations. Grounded commercial haptic devices as Omega (Force Dimension, CH) or Falcon (Novint Technologies Inc., United States) replace the control side in some experiments to project a more portable and sensorily complete experience of control as in the case reported in Tang et al. (2016). In Madhani et al. (1998) a solution built with two PHANToM haptic devices is presented, whereas (Cavusoglu et al., 1999) shows a six DoFs laparoscopic telesurgery workstation composed of an Immersion System Impulse Engine 3000 and two additional motors. the authors in Hu et al. (2016) developed a tendon driven exoskeleton to control a surgical robot and provide force stimuli, a camera system tracks the positions of fingers of the surgeon, while bilateral Bowden cables return kinesthetic information.

However, an open issue is the limited number of DoFs of robotic surgical end-effectors. In fact, the surgeon side manipulators of the da Vinci robot provides only eight DoFs to control the (Patient Side Manipulator) PSM position and the orientation and enclosure state of the common EndoWrist (Intuitive Surgical Inc., United States), bipolar and monopolar tools used to grasp, stamp or cut the organic tissues. Introducing extra degrees of freedom requires the development of suitable control (Gioioso et al., 2013; Meeker et al., 2018), or the development of alternative control consoles to move them independently. For instance, the MUSHA hand (depicted in Figure 1) is a three-soft-finger robotic gripper for surgical application (Liu et al., 2020). It is developed to be connected with the da Vinci robot, but it has more degrees of actuation (DoA) than DoFs of surgeon console interface. For this reason, the DoAs of the MUSHA hand are controlled synergistically, however, this reduces the potential of the instrument by limiting the number of configurations that can be reached by the fingers.

FIGURE 1

FIGURE 1. The MUSHA hand.

The contribution of this paper is a human to MUSHA hand mapping algorithm to perform full teleoperation of surgical hand DoAs, without increasing the mechanical complexity of the surgeon console. For this purpose, we use a camera system to track the human hand motion. The algorithm can easily be adapted to any robotic tool, the resulted algorithm is corroborated by a theoretical analysis of stability and it is followed by a preliminary study about forces exerted by fingertips.

2 Materials and Methods

The teleoperated MUSHA hand system is composed of surgical gripper and the da Vinci robot arm in patient side, while the surgeon side is comprehensive of a human hand tracker system. Communication between the surgeon and patient side is provided by Robot Operating System (ROS).

2.1 MUSHA Hand Modelling

The MUSHA hand is the main and innovative part of the teleoperation system, while the PSM of the da Vinci robot is already a known system. Then, in what follows we describe the multi-functionality of the MUSHA hand and its kinematic model which is crucial in the development of the entire control approach. According to Liu et al. (2019), and as anticipated in Section 1, the tool is a three-fingered underactuated miniature hand, each finger is composed of 12 segments and connected with a one DoF wrist, the kinematic model is composed of 37 joints connected by 36 links. An optoelectronic force sensor is designed, calibrated and presented in Liu et al. (2020), and located at the tip of each finger. In the range of the sensors, equal to 4 N, the maximal error on x-, y-, and z-direction is 0.28, 0.24, and 0.75 N, respectively. The MUSHA is actuated by the four motors of da Vinci PSM plus two additional motors. As a result, six joints are independent and directly connected to actuators by tendons, while the other ones are passive. For these reasons, it is possible to define a reduced kinematic model composed of actuated joints and equivalent links (Figure 2). The kinematic structure and the actuation system are designed to be multi-functional allowing the MUSHA hand performing three main methods of grasp, depicted in Figure 3: 1) power grasp (Po), 2) precision grasp (Pr) and 3) retractor (Rt). During the power grasp method, all fingertips and phalanges are involved to hold firmly tissue, in the precision grasp method only the fingertips are used to pinch tissue, and the retractor method allows the retraction of the tissues that come in contact with phalanges of index and middle fingertips. The relationship between joints and the tips position for a single robotic finger results as:

$\begin{array}{cc}\hfill \hfill & \hfill x_{r,\rho }=\cos \left(q_{1,\rho }\right)\left(a_2\cos \left(q_{2,\rho }\right)+a_3\cos \left(q_{2,\rho }+q_{3,\rho }\right)\right)\hfill \\ \hfill \hfill & \hfill y_{r,\rho }=\sin \left(q_{1,\rho }\right)\left(a_2\cos \left(q_{2,\rho }\right)+a_3\cos \left(q_{2,\rho }+q_{3,\rho }\right)\right)\hfill \\ \hfill \hfill & \hfill z_{r,\rho }=a_2\sin \left(q_{2,\rho }\right)+a_3\sin \left(q_{2,\rho }+q_{3,\rho }\right)\hfill \\ \hfill \hfill & \hfill \forall \rho \in \left\{T,I,M\right\}\hfill \end{array}(1)$

where q_i,ρ is the value of the i-th joint of the robotic hand r ρ-th finger and a_i is the length of the i-th link. We refer with T, I, M the thumb, index, and middle finger, respectively. Thanks to the mechanical properties of the robotic tool tendon-driven transmission, q_3,ρ and q_2,ρ are decoupled and thus they can be independently controlled, it follows that the model can be rewritten as

$\begin{array}{cc}\hfill & x_{r,\rho }=\cos \left(q_{1,\rho }\right)\left(a_2\cos \left(q_{2,\rho }\right)+a_3\cos \left(q_{3,\rho }\right)\right)\hfill \\ \hfill & y_{r,\rho }=\sin \left(q_{1,\rho }\right)\left(a_2\cos \left(q_{2,\rho }\right)+a_3\cos \left(q_{3,\rho }\right)\right)\hfill \\ \hfill & z_{r,\rho }=a_2\sin \left(q_{2,\rho }\right)+a_3\sin \left(q_{3,\rho }\right)\hfill \\ \hfill & \forall \rho \in \left\{T,I,M\right\}.\hfill \end{array}(2)$

FIGURE 2

FIGURE 2. The full kinematic model of the MUSHA hand and the reduced model into joint actuated space.

FIGURE 3

FIGURE 3. Main grasping method of MUSHA, from left to right are respectively depicted the approach poses in case of precision grasp (Pr) in A, power grasp (Po) in B and retractor (Rt) in C.

The complete kinematic model of the MUSHA hand, p_r = K(⋅), is obtained exploiting (2) for each finger, thus p_r = K(q_1,T, q_2,T, q_3,T, q_1,I, q_2,I, q_3,I, q_1,M, q_2,M, q_3,M). According to the sub-actuation constrains we define the wrist angle position, q_W, and the index and medium proximal one, qPIP,:

$\left\{\begin{array}{c}{q}_W=q_{1,I}\hfill \\ q_W=q_{1,T}+\frac{2\pi }{3}\hfill \\ q_W=q_{1,M}-\frac{2\pi }{3}\hfill \end{array}\right.\text{\hspace{0.17em}and\hspace{0.17em}}\left\{\begin{array}{c}{q}_{PIP}=q_{2,M}\hfill \\ q_{PIP}=q_{2,I}\hfill \end{array}\right.(3)$

we can simplify the kinematic relationship, obtaining:

$p_r=K\left(q_W,q_{2,T},q_{3,T},q_{PIP},q_{3,I},q_{3,M}\right)(4)$

where p_r is a vector containing the fingertips position expressed in Cartesian coordinate. It is worth noticing that in the exploited model, the length of the links a₂ and a₃ depends on the joint values. Indeed, the coupling motion of the phalanx segments (generated by the tendons) causes links shrinkage. In accordance with the chord theorem (see Figure 4), the length of the links can be computed as

$\begin{array}{c}\hfill a_{i,\rho }=2r_{i,\rho }\sin \left(\frac{{\beta }_i}{2}\right).\hfill \end{array}(5)$

FIGURE 4

FIGURE 4. Details of a bended MUSHA finger.

Let $q_{i_j,\rho }$ be the joint j in the phalanx i of the finger ρ, ∀i ∈ {2, 3} and ∀ρ ∈ {T, I, M}. Consequently, it is possible to write the length of the chord in kinematic model and the length of the link in the actuated model as follows:

$a_{i,\rho }=\left\{\begin{array}{c}\left(\frac{2L_{i,\rho }}{{\sum }_{j=2}^6{q}_{i_j,\rho }+{\sum }_{j=1}^5{q}_{i_j,\rho }}\right)\sin \left(\frac{{\sum }_{j=2}^6{q}_{i_j,\rho }+{\sum }_{j=1}^5{q}_{i_j,\rho }}{2}\right)\hfill \\ L_{i,\rho }\text{\hspace{0.17em}if\hspace{0.17em}}{q}_{i,\rho }=0\hfill \end{array}\right.(6)$

where L_i, ρ is the length of the arc of the i-th phalanx of the finger ρ. Assuming that the action of the tendons is equal on all the segments, the angular displacements are equal to a fraction of the equivalent joint in the reduced model:

$q_{i_j,\rho }=\frac{q_{i,\rho }}{6}\forall j\in \left[\mathrm{1,6}\right]\text{,\hspace{0.17em}}\forall i\in \left\{\mathrm{2,3}\right\}\text{\hspace{0.17em}and\hspace{0.17em}}\forall \rho \in \left\{T,I,M\right\}.(7)$

As a consequence, ∀ρ ∈ {T, I, M} the single finger model becomes:

$x_{r,\rho }=\cos \left(q_{1,\rho }\right)\left(a_{2,\rho }\cos \left(q_{2,\rho }\right)+a_{3,\rho }\cos \left(q_{3,\rho }\right)\right)(8)$

$y_{r,\rho }=\sin \left(q_{1,\rho }\right)\left(a_{2,\rho }\cos \left(q_{2,\rho }\right)+a_{3,\rho }\cos \left(q_{3,\rho }\right)\right)$

$z_{r,\rho }=a_{2,\rho }\sin \left(q_{2,\rho }\right)+a_{3,\rho }\sin \left(q_{3,\rho }\right)$

$a_{2,\rho }=\left\{\begin{array}{c}\left(\frac{6L_{2,\rho }}{5q_{2,\rho }}\right)\sin \left(\frac{5q_{2,\rho }}{6}\right)\hfill \\ L_{2,\rho }\text{\hspace{0.17em}if\hspace{0.17em}}{q}_{2,\rho }=0\hfill \end{array}\right.$

$a_{3,\rho }=\left\{\begin{array}{c}\left(\frac{6L_{3,\rho }}{5q_{3,\rho }}\right)\sin \left(\frac{5q_{3,\rho }}{6}\right)\hfill \\ L_{3,\rho }\text{\hspace{0.17em}if\hspace{0.17em}}{q}_{3,\rho }=0\hfill \end{array}\right.$

Therefore, the explicit full hand model (reported in the Supplementary Material) has 14 equations nine of which linearly independent, in six variables. Finally, we can express ${\dot{p}}_r=J_r^w{\dot{q}}_r$, where $J_r^w$ is the Jacobian matrix of the robotic fingers of MUSHA expressed in reference frame of the wrist. The differential model is fully reported in the Supplementary (see Supplementary Section S5.2).

2.2 Human Hand Modelling

In this section, we report the simplified kinematic model of the hand, which will be instrumental in the design of the control technique. Even if a complete human hand model has about 30 DoF (Lee and Kunii, 1995), for the sake of simplicity and without loss of generality we use a simplified kinematic hand structure. We modeled each finger as a planar kinematic chain, with three 1-D hinges. As in Lisini et al. (2017), we assume that each finger has the metacarpal (MC) bone fixed with respect to the hand frame, and it is characterized by three DoFs. In a more complete model of the human hand, the thumb has at least five DoFs: two in the trapeziometacarpal (TM) joint, two in the metacarpophalangeal (MCP) joint, and one in the interphalangeal (IP) joint. However, the thumb abduction/adduction motion range of the MCP joint usually can be neglected so the thumb can be modeled with two DoFs. Index, middle, ring, and pinky fingers have one DoFs in the MCP revolute joint for flexion/extension, one in the proximal IP (PIP), and one in the distal IP (DIP). Figure 5 shows the model of the hand used in this work. Therefore, the exploited hand model has 20 DoFs. i.e., 14 DoFs describing the fingers and six DoFs (position and orientation) for the palm. Finally, we introduce the relationship between fingertips and joints velocities, which can be described by the differential model:

${\dot{\widehat{p}}}_h=J_h\left(q_h\right){\dot{q}}_h(9)$

where the matrix $J_h\in {\mathbb{R}}^{15\times 20}$ is the hand Jacobian, that maps the joint velocities (${\dot{q}}_h\in {\mathbb{R}}^{20}$) to the Cartesian fingertip velocities (${\dot{\widehat{p}}}_h\in {\mathbb{R}}^{15}$).

FIGURE 5

FIGURE 5. The simplified kinematic model of the human hand, structure with links, joints, base, and rotation axes are in evidence.

2.3 Mapping Algorithms to Teleoperate da Vinci Arm and MUSHA Hand

Starting from the definition of the human hand and the MUSHA hand models, the developed method to control the MUSHA hand and the da Vinci robot arm are explained. A standard point-to-point mapping technique is used to control the PSM, while an innovative approach based on the theory of mapping in object domain to teleoperate the MUSHA fingers is designed and implemented. We assume that the human and the robotic hands are constantly in touch with a similar virtual object, all possible contact points are called active points. The positions of MUSHA active points p_r are defined as the positions of fingertips, while the human active points ($p_h\subset {\widehat{p}}_h$) are given by the positions of palm (P), thumb, index, and middle fingertips:

$p_h={\left[p_{h,P},p_{h,T},p_{h,I},p_{h,M}\right]}^T.(10)$

We indicate with p_h,ρ a vector of Cartesian positions of the ρ-th finger of the human hand h expressed in the world reference frame:

$p_{h,\rho }={\left[x_{h,\rho },y_{h,\rho },z_{h,\rho }\right]}^T,\forall \rho \in \left\{P,T,I,M\right\}.(11)$

Depending on the grasping method, we define the vector of interaction points, c_h, equal to a subset of active points of the human hand involved in the desired grasp, and evaluated as:

$c_h={\int }_t{\dot{c}}_h\left(\tau \right)d\tau (12)$

being

${\dot{c}}_h=I\mathrm{\Omega }{\dot{p}}_h\text{\hspace{0.17em}with\hspace{0.17em}}I\in {\mathbb{R}}^{12\times 12}(13)$

and

$\mathrm{\Omega }=\left\{\begin{array}{cc}{\mathbb{1}}_{12\times 1}\hfill & \text{if\hspace{0.17em}method}=\mathrm{Pr}\hfill \\ {\left({\mathbb{0}}_{1\times 3},{\mathbb{1}}_{1\times 9}\right)}^T\hfill & \text{if\hspace{0.17em}method}=Po\hfill \\ {\left({\mathbb{1}}_{1\times 3},{\mathbb{0}}_{1\times 3},{\mathbb{1}}_{1\times 6}\right)}^T\hfill & \text{if\hspace{0.17em}method}=Rt\hfill \end{array}\right.(14)$

It is worth noticing that Ω is a selection vector that is evaluated on-line to allow switch control reference without discontinuity and according to the grasping method required. The palm-thumb distance is used to recognize the current grasping method.

The reference of PSM in cartesian space is calculated as integration over time τ of the velocity components of the user hand centroid ${\dot{\stackrel{̃}{c}}}_h$ until the time instant t. In this way, during a temporary disconnection between the patient side and control side, the relocation of the palm in more ergonomic poses or in a position far from the boundaries of the vision cone of the camera system is allowed. This results in:

$p_{PSM}\left(t\right)=p_{PSM}\left(t-\tau \right)+{\int }_t{\dot{\stackrel{̃}{c}}}_h\left(\tau \right)d\tau (15)$

Where the value of centroid changes according to method of grasping, in fact:

${\tilde{c}}_h=\frac{\sum _{\rho \in \left\{P,T,I,M\right\}}\left(c_{h,\rho }\right)}{N}\text{\hspace{0.17em}with\hspace{0.17em}}N=\left\{\begin{array}{c}\hfill 3\text{\hspace{0.17em}if\hspace{0.17em}method}\ne Po\hfill \\ \hfill 4\text{\hspace{0.17em}if\hspace{0.17em}method}=Po\hfill \end{array}\right.(16)$

Similarly, the orientation of the human hand is exploited to orient the PSM. To this end Euler angles in roll (α), pitch (β), and yaw (γ) convention are computed as:

$\alpha =\frac{1}{2}{\tan }^{-1}\left(\frac{2{\mu }_{\mathrm{0,1,1}}}{{\mu }_{\mathrm{0,2,0}}-{\mu }_{\mathrm{0,0,2}}}\right),(17)$

$\beta =\frac{1}{2}{\tan }^{-1}\left(\frac{2{\mu }_{\mathrm{1,0,1}}}{{\mu }_{\mathrm{0,0,2}}-{\mu }_{\mathrm{2,0,0}}}\right),(18)$

$\gamma =\frac{1}{2}{\tan }^{-1}\left(\frac{2{\mu }_{\mathrm{1,1,0}}}{{\mu }_{\mathrm{2,0,0}}-{\mu }_{\mathrm{0,2,0}}}\right),(19)$

being μ_i,j,k the second order central moments of cloud of contact points

${\mu }_{i,j,k}=\sum _{\rho \in P,I,T,M}{\left(x_{h,\rho }-{\tilde{x}}_h\right)}^j{\left(y_{h,\rho }-{\tilde{y}}_h\right)}^i{\left(z_{h,\rho }-{\tilde{z}}_h\right)}^k(21)$

We exploited a sphere as virtual object to easily project the human hand motions into MUSHA fingers and wrist movements.

The roto-translations and deformations of a sphere can be completely defined by the following seven parameters: 1) $o_h\in {\mathbb{R}}^3$ Cartesian coordinate of the center of sphere; 2) ${\theta }_h\in {\mathbb{R}}^3$ the orientation of the frame attached to the sphere with respect to the world frame, expressed as roll, pitch, and yaw angles; 3) $r_h\in \mathbb{R}$ the sphere radius. Combining (13) and the aforementioned seven parameters, it is possible to obtain a linear relationship between human hand motions and sphere transformation

${\dot{c}}_{h,\rho }={\dot{o}}_h+{\dot{\theta }}_h\times \left(c_{h,\rho }-o_h\right)+{\dot{r}}_h\left(c_{h,\rho }-o_h\right).(22)$

Thus, there exists a Jacobian matrix J_hs such that its pseudo-inverse, $J_{hs}^{†}$, projects the human fingertip velocities into derivative parameters of the sphere:

$\left[\begin{array}{c}\hfill \dot{o_h}\hfill \\ \hfill \dot{{\theta }_h}\hfill \\ \hfill \dot{r_h}\hfill \end{array}\right]=J_{hs}^{†}\left(o_h,c_h\right)I\mathrm{\Omega }\dot{p_h}(23)$

The resulting Jacobian matrix depends on the current position of interaction points of human hand and on the sphere center that is a function of the active points of the human hand. Indeed, the sphere center is the circumcenter of a triangle with three of the current interaction points as vertices.

$o_h=c_{h,i}+\frac{{|\left({\mathrm{\Delta }}_{h,ki}\right)|}^2\left[\left({\mathrm{\Delta }}_{h,ji}\right)\times \left({\mathrm{\Delta }}_{h,ki}\right)\right]\times \left({\mathrm{\Delta }}_{h,ji}\right)}{2{|\left({\mathrm{\Delta }}_{h,ji}\right)\times \left({\mathrm{\Delta }}_{h,ki}\right)|}^2}+\frac{{|\left({\mathrm{\Delta }}_{h,ji}\right)|}^2\left({\mathrm{\Delta }}_{h,ki}\right)\times \left[{\mathrm{\Delta }}_{h,ji}\left.\right)\times \left({\mathrm{\Delta }}_{h,ki}\right)\right]}{2{|\left({\mathrm{\Delta }}_{h,ji}\right)\times \left({\mathrm{\Delta }}_{h,ki}\right)|}^2}(24)$

where Δ_h,ki is the difference between the position vector of the k-th fingertip c_h,k in contact with the sphere and the i-th fingertip c_h,i, then, (23) can be rewritten as:

$\left[\begin{array}{c}\hfill \dot{o_h}\hfill \\ \hfill \dot{{\theta }_h}\hfill \\ \hfill \dot{r_h}\hfill \end{array}\right]=J_{hs}^{†}\left(p_h\right)I\mathrm{\Omega }\dot{p_h}(25)$

As a further step towards the mapping, the sphere manipulated by the human is properly transformed into the one manipulated by the MUSHA through a scale factor K_s, that is equal to the ratio between the radius of human and MUSHA manipulated spheres. The algorithm computes the Jacobian of the sphere manipulated by the robot (i.e., J_rs) and evaluates the velocities of the robotic fingertips,

${\dot{p}}_r=J_{rs}\left(p_r\right)K_s{J}_{hs}^{†}\left(p_h\right)I\mathrm{\Omega }\dot{p_h}.(26)$

Velocities are estimated by a closed loop kinematic inversion

${\dot{q}}_r=J_r^{w†}\left[\left(q_r\right)J_{rs}\left(p_r\right)K_s{J}_{hs}^{†}\left(p_h\right)I\mathrm{\Omega }\dot{p_h}+K_c\left({\tilde{p}}_r-p_r\right)\right](27)$

where $J_r^w$, J_rs and J_hs are the Jacobian matrices of the robot, of the sphere manipulated by the robot, and of the sphere manipulated by the user, respectively. $K_c({\tilde{p}}_r-p_r)$ is the corrective action on the error between the current position of MUSHA interaction points p_r and the desired ones ${\tilde{p}}_r$, weighted by the gain K_c. It is worth pointing out that the pseudo-inversion of J_hs induces an indeterminacy in the estimated positions far from the current pose in case of multiple solutions, for this reason, the right position of the thumb in the retractor method may be not correctly computed. This situation is visually depicted in Figure 6. A projection into the null-space of mapping actions is used to overcome this limitation. To this end, (27) is extended as follows,

${\dot{q}}_r=J_r^{w†}\left(q_r\right)\left[J_{rs}\left(p_r\right)K_s{J}_{hs}^{†}\left(p_h\right)I\mathrm{\Omega }\dot{p_h}+K_c\left({\tilde{p}}_r-p_r\right)+J_r^w\left[I_{6\times 6}-{\left(J_{rs}{K}_s{J}_{hs}I\mathrm{\Omega }\right)}^{†}\left(J_{rs}{K}_s{J}_{hs}I\mathrm{\Omega }\right)\right]u\right](28)$

where

$u=\left\{\begin{array}{cc}\hfill {\mathbb{0}}_{6\times 1}\hfill & \hfill \text{if\hspace{0.17em}method}\ne Rt\hfill \\ \hfill K_{Rt}\left(q_{Rt}-q_r\right)\hfill & \hfill \text{if\hspace{0.17em}method}=Rt\hfill \end{array}\right.(29)$

being K_Rt the gain that multiplies the error between current joint position q_r and q_Rt, the expected joint position in retractor pose. Regarding the initial condition, at the beginning of any task, the MUSHA lies in a rest position with all the fingers closed with joint configuration $q_{r0}={\mathbb{0}}_{6\times 1}$. This particular configuration results in a kinematic singularity, in fact when q(t) = q_r0 the rank of Jacobian of robot decreases, $Rank(J_r^w)<6$. For this reason, an initialization procedure is required. In accordance with

$q_{r_0}=\left\{\begin{array}{c}\hfill \arg \left({\max }_q\left(\sqrt{\det \left(J_r^w\left(q\right)J_r^{wT}\left(q\right)\right)}\right)\right)\hfill \\ \hfill q_{i,\rho }\in \left[q_{i,{\rho }_{\mathit{min}}},q_{i,{\rho }_{\mathit{max}}}\right]\forall i\text{,\hspace{0.17em}}\forall \rho \hfill \end{array}\right.(30)$

we select $q_{r_0}=[0,\frac{\pi }{9},-\frac{\pi }{9},\frac{\pi }{9},-\frac{\pi }{9},-\frac{\pi }{9}]$ as initial position. It corresponds to the joint configuration with the maximum manipulability. In conclusion, combining (28–30), we obtain the entire human-MUSHA hand joints mapping:

$q_r=\left\{\begin{array}{cc}\hfill q_{r_0}\hfill & \hfill \text{\hspace{0.17em}if\hspace{0.17em}}t=0\hfill \\ \hfill {\int }_t{\dot{q}}_r\left(\tau \right)d\tau \hfill & \hfill \text{\hspace{0.17em}if\hspace{0.17em}}t\ne 0\hfill \end{array}\right.(31)$

with

${\dot{q}}_r=J_r^{w†}\left(q_r\right)\left[J_{rs}\left(p_r\right)K_s{J}_{hs}^{†}\left(p_h\right)I\mathrm{\Omega }\dot{p_h}+K_c\left({\tilde{p}}_r-p_r\right)+J_r^w\left[I-{\left(J_{rs}{K}_s{J}_{hs}I\mathrm{\Omega }\right)}^{†}\left(J_{rs}{K}_s{J}_{hs}I\mathrm{\Omega }\right)\right]u\right](32)$

FIGURE 6

FIGURE 6. Graphic representation of thumb indeterminacy in retractor method with two reachable positions, in the correct one in (A) the thumb is between middle and index to allow the hand to reach the position scoop-like while in wrong one, depicted in (B), the thumb is in opposition respect the other two fingers and the reached posed of MUSHA hand is equal to approach pose in power grasp method. This is a consequence of the high similarity between the virtual spheres generated by the contact points in the Rt case and in the Po case.

2.4 User Interface, Software and Communication

The camera system LeapMotion (Ultraleap inc., United States) is used to acquire active points of human hand. This camera system is composed of two monochromatic IR cameras and three LEDs that observe a hemispherical area and capture snapshots of up to 200 frames per second, and it returns information about positions of some fixed notable points of acquired hand as tips of fingers, the center of palm and knuckles. A ROS node acquires these from LeapMotion and publishes the values of active points, selecting only those useful for the mapping algorithm. To test the control law a virtual model of MUSHA hand is imported in V-rep (Coppelia Robotics GmbH, CH), and here its kinematic and dynamic behavior is simulated.

3 Results

First of all, a theoretical analysis of applied forces is conducted to evaluate the effect of contact between MUSHA and a generic organic tissue for each method of grasp, then a validation of the control algorithm is pursued to demonstrate its stability. An experimental campaign follows, during which naive participants are asked to complete a pick and place operation using the virtual model of the MUSHA hand.

3.1 Theoretical Grasping Forces Evaluation

During surgery, the grasping and manipulation of tissue may generate both internal and external forces. The former impresses deformation and allows to avoid the sliding of the manipulated tissue, the latter generates twists. In accordance with the grasping theory (Prattichizzo and Trinkle, 2016), it is possible to evaluate from the applied force (i.e., λ) the components that generate the squeezing (i.e., ξ), and the ones that generate twisting (i.e., ω). This can be done by exploiting the grasping matrix $G\in {\mathbb{R}}^{6\times 18}$ and the null space projector $N_G\in {\mathbb{R}}^{18\times 18}$, following the relationship

$\lambda =-G^{†}\omega +N_G\xi (33)$

with

$G^T=H{\tilde{G}}^T.(34)$

${\tilde{G}}^T$ is the transposed grasping matrix that maps sources of twist from contact point reference to world reference. H is a selection matrix depending on the contact finger model. Considering the consistency of organic tissues, it is possible to approximate the grasp with the Soft Finger (SF) contact model. Interested reader is referred to Prattichizzo and Trinkle (2016) for further details. Therefore, admitting the applicability of the principle of kineto-static duality, it is possible to apply generalized forces exclusively in the directions of constrained motions. Directions of force application are shown in Figure 7 in case of contact with one finger. We assumed SF contact model for each point, thus the selection matrix H can be defined as

$H=\left[\begin{array}{ccc}\hfill H_T\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill H_I\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill H_M\hfill & \hfill \hfill \end{array}\right]\in R^{18\times 18}(35)$

with

$H_T=H_I=H_M=\left[\begin{array}{cccccc}\hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill \end{array}\right].(36)$

FIGURE 7

FIGURE 7. In (A), the contact model with one finger. The red area surrounding the contact point c_j is the deformation zone. Red arrows display the directions in which motion is not allowed, thus it is possible to apply force or torques. In (B), the contact surface and the axes of the main reference systems are depicted.

Consequently, the complete grasp matrix G^T can be modeled as $G^T=H{\tilde{G}}^T$, being

${\tilde{G}}^T=\left[\begin{array}{ccc}\hfill {\overline{R_T}}^T\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill {\overline{R_I}}^T\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill {\overline{R_M}}^T\hfill \end{array}\right]\left[\begin{array}{c}\hfill P_T^T\hfill \\ \hfill P_I^T\hfill \\ \hfill P_M^T\hfill \end{array}\right],{\tilde{G}}^T\in {\mathbb{R}}^{18\times 6}(37)$

with

${\bar{R}}_j^T=\left[\begin{array}{cc}\hfill R_j^T\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill R_j^T\hfill \end{array}\right]\forall j\in \left\{T,I,M\right\},(38)$

and

$P_j^T=\left[\begin{array}{cc}\hfill I\hfill & \hfill -S\left(c_j-p_{ts}\right)\hfill \\ \hfill 0\hfill & \hfill I\hfill \end{array}\right]\forall j\in \left\{T,I,M\right\}(39)$

where R^T ∈ R^3×3 are the rotation matrices describing the orientation of each contact point reference frame Σ_j, j = T, I, M with respect to the world reference frame Σ_O, S(c_j − p_ts) is the cross-product matrix of the vector $\overline{c_j{p}_{ts}}$, and p_ts is center of mass of the grasped portion of tissue. Accounting the available grasping configurations, we can refine the general formula as follows:

$\lambda =\left\{\begin{array}{cc}\hfill N_G\xi \hfill & \hfill \text{if\hspace{0.17em}method}=Po\hfill \\ \hfill -G^{†}\omega +N_G\xi \hfill & \hfill \text{if\hspace{0.17em}method}=Pr\hfill \\ \hfill -G^{†}\omega \hfill & \hfill \text{if\hspace{0.17em}method}=Rt\hfill \end{array}\right.(40)$

At each time instant, the position of the centroid of the manipulated tissues and the positions of the contact points may be known, so the grasping matrix can be computed and a projector in its null space can be evaluated:

$N_G=I-G^{†}G.(41)$

Thus, given measure of contact force, $\tilde{\lambda }$, the value of ξ and ω can be evaluated as:

$\xi =\left\{\begin{array}{cc}\hfill N_G^{-1}\tilde{\lambda }\hfill & \hfill \text{if\hspace{0.17em}method}=Po\hfill \\ \hfill N_G^{†}\tilde{\lambda }\hfill & \hfill \text{if\hspace{0.17em}method}=Pr\hfill \\ \hfill 0\hfill & \hfill \text{if\hspace{0.17em}method}=Rt\hfill \end{array}\right.(42)$

$\omega =\left\{\begin{array}{cc}\hfill -{\left(G^{†}\right)}^{†}\tilde{\lambda }\hfill & \hfill \text{if\hspace{0.17em}method}\ne Po\hfill \\ \hfill 0\hfill & \hfill \text{if\hspace{0.17em}method}=Po\hfill \end{array}\right.(43)$

3.2 Lyapunov Stability Proof

The stability of the proposed control law was assessed by Lyapunov conditions. To this aim, we choose as positive definite Lyapunov functions the function of error:

$V=\frac{1}{2}{ϵ}^T{K}_c^Tϵ.(44)$

Where ϵ is the distance between the actual (p_r) and desired ($\widetilde{p_r}$) MUSHA fingertip.

$ϵ={\tilde{p}}_r-p_r(45)$

It is possible to evaluate its time derivative value as:

$\dot{ϵ}=\dot{\stackrel{̃}{p_r}}-\widehat{J_r^w}\dot{q_r}(46)$

Consequently, the time derivative of the Lyapunov function is

$\dot{V}={ϵ}^T{K}_c^T\left(\dot{\stackrel{̃}{p_r}}-\widehat{J_r^w}\dot{q_r}\right).(47)$

Where $\widehat{J_r^w}$ is the Jacobian matrix of model of MUSHA hand evaluated in real current position. Substituting (28) in (47), the derivative Lyapunov function becomes

$\dot{V}={ϵ}^T{K}_c^T\dot{\stackrel{̃}{p_r}}-{ϵ}^T{K}_c^T\widehat{J_r^w}\left[J_r^{w†}\left[\mathbb{M}\dot{p_h}+J_r^w{N}_{\mathbb{M}}u+K_c\left({\tilde{p}}_r-p_r\right)\right]\right].(48)$

Where $\mathbb{M}$ is the mapping action of control law

$\mathbb{M}=J_{rs}\left(p_r\right)K_s{J}_{hs}^{†}\left(p_h\right)I\mathrm{\Omega },(49)$

and $N_{\mathbb{M}}$ is the projection action in null space of $\mathbb{M}$ used to overcome the duality of thumb pose in retractor method:

$N_{\mathbb{M}}=I-{\left(J_{rs}\left(p_r\right)K_s{J}_{hs}^{†}\left(p_h\right)I\mathrm{\Omega }\right)}^{†}\left(J_{rs}\left(p_r\right)K_s{J}_{hs}^{†}\left(p_h\right)I\mathrm{\Omega }\right).(50)$

The desired velocity ${\dot{\stackrel{̃}{p}}}_r$ is chosen as the result of mapping $\mathbb{M}(q_r,p_r,p_h)$ and projection in null space of mapping $N_{\mathbb{M}}$,

${\dot{\stackrel{̃}{p}}}_r=\mathbb{M}{\dot{p}}_h+J_r^w{N}_{\mathbb{M}}u.(51)$

Introducing (48) in (51), it is obtained that:

$\dot{V}={ϵ}^T{K}_c^T\dot{\stackrel{̃}{p_r}}-{ϵ}^T{K}_c^T\widehat{J_r^w}{J}_r^{w†}\left[\dot{\stackrel{̃}{p_r}}+K_c\left({\tilde{p}}_r-p_r\right)\right].(52)$

Reintroducing (52) the error evaluated in accordance with (45), we obtain

$\dot{V}={ϵ}^T{K}_c^T\dot{\stackrel{̃}{p_r}}-{ϵ}^T{K}_c^T\widehat{J_r^w}{J}_r^{w†}\left(\dot{\stackrel{̃}{p_r}}+K_cϵ\right).(53)$

If MUSHA is accurately calibrated the Jacobian matrix evaluated on the real position of fingers and on the estimated one are approximately equal, $\widehat{J_r^w}\approx J_r^w$, and the time derivative Lyapunov function becomes

$\dot{V}\approx -{ϵ}^T{K}_c^T{K}_cϵ.(54)$

Since the Lyapunov function (47) is positive definite (V(t) > 0) and its time derivative is negative definite $(\dot{V}(t)<0)$, the system is Lyapunov stable.

It is possible assert the BIBO stability, too. In fact, substituting definition of desired position (51) in control law (28), we obtain that

${\dot{q}}_r=J_r^{w†}\left({\dot{\stackrel{̃}{p}}}_r-K_cϵ\right)(55)$

It is possible to evaluate the tendency of error, using the definition of time derivative of error metric:

$\dot{ϵ}=K_cϵ(56)$

The error evolves as first order linear system, so the Lyapunov stability implies the BIBO stability.

3.3 User Studies

Ten inexperienced participants (six males and four females, aged 24–59, all right-handed) took part in user studies. Each subject gave her/his consent to participate and was able to discontinue participation at any time during experiments. The experimental evaluation protocols followed the declaration of Helsinki, and there was no risk of harmful effects on subjects’ health. Participants performed 10 pick-and-place tasks of two cubes with 10 different initial scenarios pseudo-randomly sorted. The experimental set-up is shown in¹ Figure 8. The dimension of each cube is 0.02 m × 0.02 m×0.02 m and the weight is equal to 20 g. To help the participant to complete the task, force feedback is provided by two vibromotors NFP-C1234L to inform them about internal and external forces applied on cubes. The two components of force are evaluated starting from the results of Section 3.1 using (42) and (43), respectively, and their norms in ℓ² space, ‖ξ‖ and ‖ω‖, are used to calculate the motor PWMs as:

$PW{M}_{\xi }=⌈\frac{1}{k}\frac{\Vert \xi \Vert }{\mathrm{\Delta }F}255⌉,(57)$

$PW{M}_{\omega }=⌈\frac{\Vert \omega \Vert }{\mathrm{\Delta }F}255⌉.(58)$

where ΔF is the range of forces that can be acquired through MUSHA hand force sensors (ΔF = 4N). The dimension disparity between the internal force space ($\xi \in {\mathbb{R}}^{18}$) and the external one ($\omega \in {\mathbb{R}}^6$) introduces an imbalance of haptic stimulus intensity which has been compensated by scaling the ℓ² norm ‖ξ‖ with a k–factor equal to the ratio of the dimensions of the two spaces (k = 3).

FIGURE 8

FIGURE 8. The experimental set-up consisted in the LeapMotion controller, the haptic bracelet and a couple of screen to visualize the Virtual environment and the reconstructed model of tracked hand. In the virtual scene, cubes pseudo-randomly changed their position at the beginning of the single repetition of experiment.

At the end of each session, a NASA Task Load Index questionnaire (TLX) (Hart, 2006) was proposed to the participants, with the aim of assessing the perceived load in terms of Mental Demand (MD), Temporal Demand (TD), Physical Demand (PD), Performance (PE), Effort (EF), and Frustration (FR). This method assesses workload on a point-based scale. Each of the six questions has a scale of 21 levels, considering 1 as “very low” and 21 as “very high.” Questions are reported in what follows:

- Mental demand: “How mentally demanding was the task?”

- Physical demand: “How physically demanding was the task?”

- Temporal demand: “How hurried or rushed was the pace of the task?”

- Performance: “How successful were you in accomplishing what you were asked to do?”

- Effort: “How hard did you have to work to accomplish your level of performance?”

- Frustration: “How insecure, discouraged, irritated, stressed, and annoyed were you?”

All participants successfully completed the ten pick and place repetitions of the two cubes, so the completion times for each trial and the results of the questionnaires were considered as metrics for evaluation. Results are reported in Table 1 and Table 2. The mean time to complete the task is 16.49 s and no participant took more than 30s. The load of the task performed with the proposed control law is evaluated from answers of the NASA TLX questionnaire and its mean value is equal to 35%. According with the proposed task and experimental set-up we used the following weights: w_MD = 3, w_PD = 0, w_TD = 5, w_PE = 1, w_FR = 3, w_EF = 3. The physical demand was excluded from task load evaluation (PD = 0) because the physical load is due to the absence of arm support that is provided by the surgeon console in a real scenario.

TABLE 1

TABLE 1. Time to complete tasks (mean and ±std).

TABLE 2

TABLE 2. NASA TLX questionnaire results, tally of importance selections: w_MD = 3, w_PD = 0, w_TD = 5, w_PE = 1, w_FR = 3, w_EF = 3.

4 Discussion

In this work, we present a reliable and intuitive method for the bilateral asymmetric teleoperation of the MUSHA hand. A camera-based device is in charge of tracking the human hand, and a mapping algorithm projects the human motion in the end-effector actuation space through the virtual object domain. The developed technique consists in a remote control for the projection of the human gestures into the robot space of actuation by crossing the object domain and switching in the function of the current gripping method, which is recognized online by the posture of the user’s hand. A sphere is chosen as virtual objects due to the reduced number of parameters that describe its pose and deformation. A closed-loop inverse kinematic algorithm makes more robust the control action and guarantees the BIBO stability of the algorithm, while a projection in the null of the object space allows the correct estimation of indeterminate poses. Starting from the kinematics description of the MUSHA hand and its main grasping methods, we theoretically evaluated the grasping forces applied by the tool during each method and an experimental campaign assesses the functionality and the usability of the algorithm. All the participants completed the assigned task in less than 30 s (15 s per cube) with a limited cognitive load. The obtained results demonstrate the intuitiveness of the teleoperation system.

The proposed control algorithm has been designed, developed, and tested to control the MUSHA hand, nevertheless it can be generalized and extended to different grippers and robotic hands. Indeed the proposed methodology relies on a mapping algorithm that exploits contact points and grasping methods. Thus, a redefinition of the differential kinematic equations and of the related Jacobian Matrix of the gripper allow the reader adapting the algorithm to the hand of interest. A preliminary analysis of the instrument grasping capacity and the redefinition of the selection vector, Ω, allows to maintain the validity of the theoretical results regarding the evaluation of the applied forces. In future work, experiments will be conducted considering expert users asking them to complete elementary surgical procedures using the real MUSHA hand mounted on the da Vinci robot and we will test the compatibility of the algorithm with several standard end-effector during the same procedures. Finally, the entire system will be integrated with the surgeon console of the da Vinci robot.

Data Availability Statement

The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

FF and DP were project coordinators, they contributed to conception and design of the study. AV and TL worked on the experimental setup and runned the experiments. FF, AV, and DP proposed algorithm and supervised the experiments. FF, AV, and TL were principal investigators, and wrote the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.

Funding

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101017008.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frobt.2021.741807/full#supplementary-material

Footnotes

¹A video demonstration of the experimental task is available at the following link: https://youtu.be/6DYI1MfqI7A.

References

Beasley, R. A. (2012). Medical Robots: Current Systems and Research Directions. J. Robotics 2012, 401613. doi:10.1155/2012/401613

CrossRef Full Text | Google Scholar

Bebek, O., and Cavusoglu, M. C. “Model Based Control Algorithms for Robotic Assisted Beating Heart Surgery,” in Proceedings of the 2006 International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE), New York, NY, USA, August 2006, 823–828. doi:10.1109/IEMBS.2006.260562

CrossRef Full Text | Google Scholar

Bowyer, S. A., Davies, B. L., and y Baena, F. R. (2013). Active Constraints/virtual Fixtures: A Survey. IEEE Trans. Robotics 30, 138–157. doi:10.1109/TRO.2013.2283410

CrossRef Full Text | Google Scholar

Cavusoglu, M. C., Tendick, F., Cohn, M., and Sastry, S. S. (1999). A Laparoscopic Telesurgical Workstation. IEEE Trans. Robot. Automat. 15, 728–739. doi:10.1109/70.782027

CrossRef Full Text | Google Scholar

Chadi, S. A., Guidolin, K., Caycedo-Marulanda, A., Sharkawy, A., Spinelli, A., Quereshy, F. A., et al. (2020). Current Evidence for Minimally Invasive Surgery during the COVID-19 Pandemic and Risk Mitigation Strategies. Ann. Surg. 272, e118–e124. doi:10.1097/sla.0000000000004010

PubMed Abstract | CrossRef Full Text | Google Scholar

Cheng, L., and Tavakoli, M. (2018). Switched-impedance Control of Surgical Robots in Teleoperated Beating-Heart Surgery. J. Med. Robot. Res. 03, 1841003. doi:10.1142/s2424905x18410039

CrossRef Full Text | Google Scholar

Fontanelli, G. A., Ficuciello, F., Villani, L., and Siciliano, B. “Modelling and identification of the da vinci research kit robotic arms,” in Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE), Vancouver, BC, Canada, September 2017, 1464–1469.

Google Scholar

Fontanelli, G. A., Selvaggio, M., Ferro, M., Ficuciello, F., Vendittelli, M., and Siciliano, B. “A v-rep simulator for the da vinci research kit robotic platform,” in Proceedings of the 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob) (IEEE), Enschede, Netherlands, August 2018, 1056–1061.

Google Scholar

Gioioso, G., Salvietti, G., Malvezzi, M., and Prattichizzo, D. (2013). Mapping Synergies from Human to Robotic Hands with Dissimilar Kinematics: an Approach in the Object Domain. IEEE Trans. Robot. 29, 825–837. doi:10.1109/tro.2013.2252251

CrossRef Full Text | Google Scholar

Handini, D., Yeong, T. M., and Hong, L. V. “System Integration of Neurobot: a Skull-Base Surgical Robotic System,” in Proceedings of the IEEE Conference on Robotics, Automation and Mechatronics, 2004. (IEEE), Singapore, December 2004, 43–48.

Google Scholar

Hanley, D. F., Thompson, R. E., Rosenblum, M., Yenokyan, G., Lane, K., McBee, N., et al. (2019). Efficacy and Safety of Minimally Invasive Surgery with Thrombolysis in Intracerebral Haemorrhage Evacuation (Mistie Iii): a Randomised, Controlled, Open-Label, Blinded Endpoint Phase 3 Trial. Lancet 393, 1021–1032. doi:10.1016/S0140-6736(19)30195-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Hart, S. G. (2006). Nasa-task Load index (Nasa-tlx); 20 Years Later. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 50, 904–908. doi:10.1177/154193120605000909

CrossRef Full Text | Google Scholar

Hu, Z., Yoon, C.-H., Park, S. B., and Jo, Y.-H. (2016). Design of a Haptic Device with Grasp and Push-Pull Force Feedback for a Master-Slave Surgical Robot. Int. J. CARS 11, 1361–1369. doi:10.1007/s11548-015-1324-9

CrossRef Full Text | Google Scholar

Jintae Lee, J., and Kunii, T. L. (1995). Model-based Analysis of Hand Posture. IEEE Comput. Grap. Appl. 15, 77–86. doi:10.1109/38.403831

CrossRef Full Text | Google Scholar

Kim, C. Y., Kang, B. H., Lee, M. C., and Yoon, S. M. (2013). Design of Master Console Haptic Handle for Robot Assisted Laparoscopy. Intell. Autonomous Syst. 12, 3–10. doi:10.1007/978-3-642-33932-5_1

CrossRef Full Text | Google Scholar

Kulaylat, A. S., Mirkin, K. A., Puleo, F. J., Hollenbeak, C. S., and Messaris, E. (2018). Robotic versus Standard Laparoscopic Elective Colectomy: where Are the Benefits? J. Surg. Res. 224, 72–78. doi:10.1016/j.jss.2017.11.059

CrossRef Full Text | Google Scholar

Leven, J., Burschka, D., Kumar, R., Zhang, G., Blumenkranz, S., Dai, X. D., et al. (2005). “Davinci Canvas: a Telerobotic Surgical System with Integrated, Robot-Assisted, Laparoscopic Ultrasound Capability,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, Palm Springs, CA, USA, October 2005 (Springer), 811–818. doi:10.1007/11566465_100

CrossRef Full Text | Google Scholar

Lisini, B. T., Scheggi, S., Meli, L., Mohammadi, M., and Prattichizzo, D. (2017). Gesto: A Glove for Enhanced Sensing and Touching Based on Inertial and Magnetic Sensors for Hand Tracking and Cutaneous Feedback. IEEE Trans. Human-mach. Syst. 47, 1066–1076. doi:10.1109/thms.2017.2720667

CrossRef Full Text | Google Scholar

Liu, H., Ferrentino, P., Selvaggio, M., Pirozzi, S., and Ficuciello, F. “Design of the Musha Hand Ii for Robotic-Assisted Laparoscopic Surgery,” in Proceedings of the 12th The Hamlyn Symposium on Medical Robotics, London, United Kingdom, June 2019.

Google Scholar

Liu, H., Selvaggio, M., Ferrentino, P., Moccia, R., Pirozzi, S., Bracale, U., et al. (2020). The Musha Hand Ii: A Multi-Functional Hand for Robot-Assisted Laparoscopic Surgery. Ieee/asme Trans. Mechatron. 26, 1. doi:10.1109/TMECH.2020.3022782

CrossRef Full Text | Google Scholar

Madhani, A. J., Niemeyer, G., and Salisbury, J. K. “The Black falcon: a Teleoperated Surgical Instrument for Minimally Invasive Surgery,” in Proceedings of the 1998 IEEE/RSJ International Conference on Intelligent Robots and Systems. Innovations in Theory, Practice and Applications (Cat. No. 98CH36190) (IEEE), Victoria, BC, Canada, October 1998, 936–944.

Google Scholar

Manganelli, R., Chinello, F., Formaglio, A., and Prattichizzo, D. (2010). A Teleoperation System for Micro-invasive Brain Surgery. Paladyn, J. Behav. Robotics 1, 198–203. doi:10.2478/s13230-011-0006-8

CrossRef Full Text | Google Scholar

Meeker, C., Rasmussen, T., and Ciocarlie, M. “Intuitive Hand Teleoperation by Novice Operators Using a Continuous Teleoperation Subspace,” in Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA) (IEEE), Brisbane, QLD, Australia, May 2018, 1–7.

Google Scholar

Meli, L., Pacchierotti, C., and Prattichizzo, D. (2014). Sensory Subtraction in Robot-Assisted Surgery: Fingertip Skin Deformation Feedback to Ensure Safety and Improve Transparency in Bimanual Haptic Interaction. IEEE Trans. Biomed. Eng. 61, 1318–1327. doi:10.1109/tbme.2014.2303052

PubMed Abstract | CrossRef Full Text | Google Scholar

Moccia, R., Iacono, C., Siciliano, B., and Ficuciello, F. (2020). Vision-based Dynamic Virtual Fixtures for Tools Collision Avoidance in Robotic Surgery. IEEE Robot. Autom. Lett. 5, 1650–1655. doi:10.1109/lra.2020.2969941

CrossRef Full Text | Google Scholar

Mohareri, O., Schneider, C., and Salcudean, S. “Bimanual Telerobotic Surgery with Asymmetric Force Feedback: a Davinci® Surgical System Implementation,” in In Proceedings of the 2014 Ieee/Rsj International Conference on Intelligent Robots and Systems (IEEE), Chicago, IL, USA, September 2014, 4272–4277.

Google Scholar

Niemeyer, G., and Slotine, J.-J. E. (1991). Stable Adaptive Teleoperation. IEEE J. Oceanic Eng. 16, 152–162. doi:10.1109/48.64895

CrossRef Full Text | Google Scholar

Okamura, A. M. (2009). Haptic Feedback in Robot-Assisted Minimally Invasive Surgery. Curr. Opin. Urol. 19, 102–107. doi:10.1097/mou.0b013e32831a478c

PubMed Abstract | CrossRef Full Text | Google Scholar

Parisi, A., Reim, D., Borghi, F., Nguyen, N. T., Qi, F., Coratti, A., et al. (2017). Minimally Invasive Surgery for Gastric Cancer: A Comparison between Robotic, Laparoscopic and Open Surgery. Wjg 23, 2376. doi:10.3748/wjg.v23.i13.2376

PubMed Abstract | CrossRef Full Text | Google Scholar

Perigli, G., Cortesini, C., Qirici, E., Boni, D., and Cianchi, F. (2008). Clinical Benefits of Minimally Invasive Techniques in Thyroid Surgery. World J. Surg. 32, 45–50. doi:10.1007/s00268-007-9259-0

CrossRef Full Text | Google Scholar

Prattichizzo, D., and Trinkle, J. C. (2016). “Grasping,” in Springer Handbook of Robotics. Editors B. Siciliano, and O. Khatib (Berlin, Germany: Springer), 955–988. doi:10.1007/978-3-319-32552-1_38

CrossRef Full Text | Google Scholar

Sheetz, K. H., Claflin, J., and Dimick, J. B. (2020). Trends in the Adoption of Robotic Surgery for Common Surgical Procedures. JAMA Netw. Open 3, e1918911. doi:10.1001/jamanetworkopen.2019.18911

PubMed Abstract | CrossRef Full Text | Google Scholar

Song, J., Kang, W. H., Oh, S. J., Hyung, W. J., Choi, S. H., and Noh, S. H. (2009). Role of robotic gastrectomy using da vinci system compared with laparoscopic gastrectomy: initial experience of 20 consecutive cases. Surg. Endosc. 23, 1204–1211. doi:10.1007/s00464-009-0351-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Sood, A., Meyer, C. P., Abdollah, F., Sammon, J. D., Sun, M., Lipsitz, S. R., et al. (2017). Minimally Invasive Surgery and its Impact on 30-day Postoperative Complications, Unplanned Readmissions and Mortality. J. Br. Surg. 104, 1372–1381. doi:10.1002/bjs.10561

CrossRef Full Text | Google Scholar

Sugimoto, M., Yasuda, H., Koda, K., Suzuki, M., Yamazaki, M., Tezuka, T., et al. (2010). Image Overlay Navigation by Markerless Surface Registration in Gastrointestinal, Hepatobiliary and Pancreatic Surgery. J. hepato-biliary-pancreatic Sci. 17, 629–636. doi:10.1007/s00534-009-0199-y

CrossRef Full Text | Google Scholar

Sukhoon, P., Bin, L. K., Muk, J. J., and Yong-San, Y. “Design of Master Console Robot for Natural Orifice Transluminal Endoscopic Surgery,” in Proceedings of the 2009 ICCAS-SICE (IEEE), Fukuoka, Japan, August 2009, 1152–1157.

Google Scholar

Tabrizi, L.B., and Mahvash, M. (2015). Augmented Reality-Guided Neurosurgery: Accuracy and Intraoperative Application of an Image Projection Technique. Jns 123, 206–211. doi:10.3171/2014.9.jns141001

CrossRef Full Text | Google Scholar

Takács, Á., Rudas, I., and Haidegger, T. (2015). Open-source Research Platforms and System Integration in Modern Surgical Robotics. Acta Universitatis Sapientiae; Electr. Mech. Eng. 14, 20–34.

Google Scholar

Tang, A., Cao, Q., Tan, H., Fujie, M. G., and Pan, T. (2016). “Motion Control of a Master-Slave Minimally Invasive Surgical Robot Based on the Hand-Eye-Coordination,” Computer Aided Surgery. Editor M. Fujie (Berlin, Germany: Springer), 57–71. doi:10.1007/978-4-431-55810-1_5

CrossRef Full Text | Google Scholar

Tonutti, M., Elson, D. S., Yang, G.-Z., Darzi, A. W., and Sodergren, M. H. (2017). The Role of Technology in Minimally Invasive Surgery: State of the Art, Recent Developments and Future Directions. Postgrad. Med. J. 93, 159–167. doi:10.1136/postgradmedj-2016-134311

PubMed Abstract | CrossRef Full Text | Google Scholar

Van der Meijden, O. A., Broeders, I. A., and Schijven, M. P. (2010). The SEP "robot": a Valid Virtual Reality Robotic Simulator for the Da Vinci Surgical System? Surg. Technol. Int. 19, 51–58.

PubMed Abstract | Google Scholar

Yamamoto, T., Abolhassani, N., Jung, S., Okamura, A. M., and Judkins, T. N. (2012). Augmented Reality and Haptic Interfaces for Robot-Assisted Surgery. Int. J. Med. Robotics Comput. Assist. Surg. 8, 45–56. doi:10.1002/rcs.421

CrossRef Full Text | Google Scholar

Yin, X., Guo, S., and Wang, Y. “Force Model-Based Haptic Master Console Design for Teleoperated Minimally Invasive Surgery Application,” in 2015 IEEE International Conference on Mechatronics and Automation (ICMA) (IEEE), Beijing, China, August 2015, 749–754.

Google Scholar