In our instrument localization pipeline, we assume that the instruments are not completely occluded. Partial occlusions are supported, as long as there is a visible path from the entrypoint to the tip of the instrument. Note that with entrypoint we refer to the point located at the edge of the endoscopic content area where the instrument enters the image. This point is not to be confused with the incision point which is the point on the body wall where the incision is made through which the instrument enters the patient’s body. Now, if the tip is occluded, the tooltip will be estimated on the furthermost point of the shaft. When the entrypoint is completely covered, the instrument will not be detected in the current approach. Methods that exploit knowledge of the incision point could help in such a case (and could be explored in future work as they do not form the core of this work). The current limitations are illustrated in Figure 4. The assumption that instruments have to enter from the edge serves two purposes, 1) as a noise reduction technique for the segmentation, because false positive islands of pixels can be easily discarded, and 2) to detect whether the instrument is held by the right/left hand of the surgeon (as explained in Section 4.2.8). In most cases, the entrypoint of at least one of the instruments will be visible. Therefore, the benefits of the assumption that instruments will not be completely occluded largely outweigh its limitations. The proposal of surgical tool segmentation models that are robust to entrypoint occlusions (Figure 4, right) or complete occlusions is out of the scope of this work.

The stereo camera and endoscopy module of choice for this work were the Tipcam1 S 3D ORL (30° view on a 4 mm outer diameter shaft) and Image1 S D3-LINK respectively (both from Karl Storz, Germany). The DVI output of the endoscopy module is plugged into a DVI2PCIe Duo framegrabber (Epiphan, Canada). The endoscopy module produces images at 60  fps and at a resolution of 1920, ×, 1080 pixels, which turns into 1920, ×, 540 as each grabbed image contains a stereo pair with the left frame on even rows and the right frame on odd ones. These images are upsampled in the y-axis so that two images of 1920, ×, 1080 pixels are obtained. Upscaling is chosen (as opposed to downsampling to 960, ×, 540 pixels) to avoid degrading the depth resolution based on x-axis disparity. The left-right cameras are calibrated using a chessboard pattern of 1.1 mm-wide squares (Cognex Glass Calibration Plate Set 320-0015R, Applied Image Inc., NY, United States). Both frames, left and right, are rectified in real-time. Then, the black background of the images is cropped out, keeping just a square crop of the endoscopic circle content area (as shown in Figure 2), which results in an image of 495 × 495 pixels. Finally, the image where the 2D tooltip localization is going to be performed (either the left or right frame can be chosen without loss of generality) is downsampled to 256 × 256 pixels to speed up the subsequent processing steps (i.e. segmentation, graph extraction and 2D tooltip localization). Once the 2D tooltips have been estimated, they are extrapolated to the original image size and the disparity estimation and 3D tooltip reconstruction in Section 4.2.9 is performed on the original upsampled images of 1920, ×, 1080 pixels.

In this work, we trained a CNN to segment instruments in our experimental setup (see Figure 1). While having the necessary characteristics for a bimanual laparoscopic task, the visual appearance of the surgical training model we use is not representative of a real clinical environment. Therefore, we do not propose a new image segmentation approach but rather focus on the downstream computational questions. In order to translate our pipeline to the clinic, a newly annotated dataset containing real clinical images would need to be curated, and the images would need to contain artifacts typical of endoscopic procedures such as blood, partial occlusions, smoke, and blurring. Alternatively, an existing surgical dataset could be used. We have compiled a list of public datasets for tool segmentation ² where the data available includes surgical scenes such as retinal microsurgery, laparoscopic adrenalectomy, pancreatic resection, neurosurgery, colorectal surgery, nephrectomy, proctocolectomy, and cholecystectomy amongst others. The compiled list also includes datasets for similar endoscopic tasks such as tool presence, instrument classification, tool-tissue action detection, skill assessment and workflow recognition, and laparoscopic image-to-image translation. The unlabelled data in these other datasets could also be potentially helpful for tool segmentation.

Next, we provide the details on how we built the segmentation model for our particular proof-of-concept of the robotic endoscope control. The dataset we curated consists of 1110 image-annotation pairs used for training, and 70 image-annotation pairs employed for validation (hyperparameter tuning). These 1110, + ,70 image-annotation pairs were manually selected by the first co-authors so that the chosen images represent well the variety of scenes in the task. They have been extracted from the recording of a surgical exercise in the lab, prior to the user study, and in a different location. There is no testing set at this point because the segmentation is an intermediary step. In Section 7.1, we give more details about our testing set, which is used to evaluate the whole tooltip localization pipeline (as opposed to just the segmentation). The images in each stereo pair do not look the same: there is an observable difference in colour tones between them. Therefore, the data set has an even number of left and right frames such that either of them could be used as input for the surgical tool segmentation. In the training set, 470 images (42%) do not contain any tool. In them, the endoscope just observes the task setting under different viewpoints and lighting conditions (diverse intensities of the light source). The remaining 640 images of the training set, and all images of the validation set, have been manually labelled with delineations of the laparoscopic tools. The U-Net (Ronneberger et al., 2015) architecture showed superior performance in the tool segmentation EndoVis MICCAI challenge (Allan et al., 2019). Therefore, this was the architecture of choice employed for segmentation (32 neurons in the first layer and convolutional blocks composed of Conv + ReLU + BN). A minibatch of four images is used. Default conventional values and common practice was followed for setting the hyperparameters as detailled hereafter. The batch normalization (Ioffe and Szegedy, 2015) momentum was set to 0.1 (default value in PyTorch). Following the U-Net implementation in (Ronneberger et al., 2015), Dropout (Srivastava et al., 2014) was used. In our implementation, Dropout was employed in layers with ≥ 512 neurons (p = 0.5), as in (Garcia-Peraza-Herrera et al., 2017). Following Bengio (2012), the initial learning rate (LR) of choice was set to 1e − 2. The network was trained for a maximum of 100 epochs. As is common practice, LR decay was employed during training, multiplying the LR by 0.5 every 10 epochs. Data augmentation was limited to on-the-fly left-right flips. As we evaluate our segmentation using the intersection over union (IoU), our loss function L IoU is a continuous approximation to the intersection over union (Rahman and Wang, 2016) averaged over classes: I y ̂ , y , k = ∑ i = 1 P y ̂ i , k ⋅ y i , k , U y ̂ , y , k = ∑ i = 1 P y ̂ i , k + ∑ i = 1 P y i , k − ∑ i = 1 P y ̂ i , k ⋅ y i , k , L IoU y ̂ , y = 1 − 1 K ∑ k = 1 K I y ̂ , y , k + ϵ U y ̂ , y , k + ϵ , (2) Where P is the number of pixels, K = 2 is the number of classes (instrument and background), y ̂ represents the estimated probability maps, y represents the ground truth probability maps, y ̂ i , k is the estimated probability of the pixel i belonging to the class k, and y _i,k is the ground truth probability of the pixel i belonging to class k. A machine epsilon ϵ is added to prevent divisions by zero (e.g., in case that both prediction and ground truth are all background).

Once we have obtained a segmentation prediction from the trained convolutional model, we proceed to convert the segmentation into a graph, which is a stepping stone towards the tooltip detection.

The instrument segmentation prediction is skeletonized via medial surface axis thinning (Lee et al., 1994). The resulting skeleton is converted via the Image-Py skeleton network framework (Xiaolong, 2019) into a pixel skeleton graph G = (V, E) (see Figure 5E), where V is a set of vertices and E ⊆ { { x , y } : x , y ∈ V ∧ x ≠ y } is a set of edges. The nodes v _i ∈ V are defined as a tuple v _i = (i, p _i ) where i and p _i = {x _i , y _i } represent the node index and 2D point image coordinates, respectively.

As the size of the image is known, a circular segmentation mask (see Figure 5B) is used to detect the graph nodes that could potentially correspond to instrument entrypoints. That is, given G, we populate a set R containing those graph nodes that represent tool entrypoints into the endoscopic image. Those graph nodes contained within the intersection of the circle mask and the tool segmentation mask are collapsed into a single new entry node v _c = (n, p _c ) per instrument, where p _c = {x _c , y _c } is set to the centroid of all nodes captured within the aforementioned intersection. See Figures 5B–F for an example of entry node extraction.

A depth-first search is launched from each entry node to determine all the graph nodes that can be reached from entry nodes. Those that cannot be reached are pruned from the graph.

Let L = {v ∈ V : d _G (v) = 1 ∧ v∉R} be the set containing all leaf nodes, where d _G (v) = |{u ∈ V: {u, v} ∈ E}|. In this part of the instrument localization pipeline each leaf node in L is paired to an entrypoint node in R. This is solved by recursively traversing G, starting from each leaf. The criteria to decide which node to traverse next is coined in this work as dot product recursive traversal. It is based on the assumption that the correct path from a tip to a corresponding entrypoint is the one with minimal direction changes. The stopping condition is reaching an entry node.

Dot product recursive traversal operates as follows. Let v _i , v _j ∈ V be two arbitrary nodes, and {v _i , v _j } the undirected edge connecting them. Assuming v _i is previously visited and v _j being traversed, the next node v* to visit is chosen following: v * = argmax k , p k ∈ N v j − A p j − p i | | p j − p i | | 2 ⋅ p k − p j | | p k − p j | | 2 , (3) Where N(v _i ) = {w ∈ V: {v _i , w} ∈ E}, and A = {v _i } is the set of nodes previously traversed. The idea behind (3) is to perform a greedy minimization of direction changes along the path from tooltip to entrypoint (Figure 6A).

In the case of two tools present in the image, it is possible to have hidden leaves (Figures 6B,D), defined as graph nodes that represent an overlap between the tip of an instrument and another instrument. This situation can easily occur (Figure 6) in surgical tasks, including the task presented in the experiments from Section 6. There are two possible graph arrangements that can lead to hidden leaves. A node with exactly two (Figure 6B) or three neighbours (Figure 6D). Nonetheless, the number of neighbours alone does not facilitate the discovery of such hidden leaves (and subsequent disentanglement of tools), as it is also possible for a node with exactly two (could be a chain instead) or three (could be a bifurcation instead) neighbours to not be a hidden leaf (see Figure 6C). Hence, extra information is needed. Aiming in this direction, after each successful traversal from a normal leaf to an entry node, all the edges along the path are labelled with the index of the entry node. In addition, all the edges directly connected to an entry node are also labelled.

A node with exactly two or three neighbours whose edges are all labelled with different entry nodes is a hidden leaf. Labelling helps to solve some of the hidden leaf cases. Such leaves can be duplicated, effectively splitting the graph into two, and disentangling the overlapped instruments. After disentanglement, they become normal leaves which can be assigned to an entry node by dot product traversal (3). Although not a hidden leaf, a node with exactly four neighbours whose edges are labelled represents an overlap which can be trivially disentangled. Hidden leaves such as the ones presented in Figures 6B,D cannot be classified with certainty as such just with the graph/skeleton information. As shown in Figure 6, different tool configurations/occlusions could lead to the same graph configuration. As not all the tips can be unambiguously detected, entry nodes that are unmatched after dot product traversal (i.e., they were not reached after launching a traversal from each leaf node to a possible entry node) are paired to the furthest opposing node connected to them.

Although the traversal from tips to entrypoints has been illustrated in this section with one or two instruments (as it is the case in our setup, see Figure 1), the dot product traversal generalizes to setups with more instruments as the assumption that the path from tip to entrypoint is the one with less direction changes still holds.

Noisy skeletons can lead to inaccurate graphs containing more than two leaves matched to the same entry node, or more than two entry nodes connected to leaves. In our framework, unmatched entry nodes are deleted. In addition, due to the nature of our experimental setup, a maximum of two tools with two tips each can be found. Therefore, when more than two leaves are matched to the same entry node, only the two furthest are kept. Analogously, when more than two entry nodes are found and matched to leaves, the two kept are those with the longest chain (from entry node to leaves). That is, a maximum of two disentangled instrument graphs remain after pruning.

In the presence of a single instrument, left/right vertical semi-circles determine whether the instrument is left/right (see Figure 7, right), i.e. if the entrypoint of the tool is located in the right half of the image, it is assumed that the subgraph connected to this entrypoint is the right instrument, and viceversa. Note that this simple method is also generalizable to scenarios with three to five instruments, which are different from the two-instrument solo surgery setting examined in this work (see Figure 8), but still worth mentioning as there are some endoscopic procedures that may involve such number of tools (Abdi et al., 2016).

When two instruments are detected (i.e. two entrypoints with their corresponding subgraphs), a line segment connecting the entrypoints of both instruments is assumed to be the viewing horizon. A vertical line that is parallel to the vertical axis of the image and cuts through the central point of the viewing horizon defines whether the entrypoints (and subsequently the instruments) are left/right (see Figure 7, left).

Once the tips of the instruments have been detected and classified as right/left instrument, the disparity for each tooltip in the left and right stereo images is estimated using classical intensity-based template matching. As endoscope images are stereo-rectified, template matching with a sliding window of 64 × 64 pixels running over the same line (and only in one direction) suffices for the stereo matching. Normalized cross-correlation is the cost function of choice. Given the disparity measure for each tooltip, its depth can be reconstructed using the pinhole camera model and conventional epipolar geometry. The 3D reconstruction was performed with an extended Kalman filter (EKF). The EKF is valuable here, because of its capacity to bridge potential measurement gaps and to reduce the noise on the 3D position estimate, which is very sensitive to small disparity variations, as the lens separation of the Tipcam is only 1.6 mm. The details of the EKF are specified in Section 5.2.2.

Although in our proposed experimental setup we use stereovision because we have an stereo-endoscope, many centers still use monoscopic endoscopes. In this case, a method such as that presented by Liu et al. (Liu et al., 2020) could be used to estimate the 3D tip location directly from the 2D endoscopy.

Endoscopes come in different forms and sizes. Rigid endoscopes are typically straight, but can also be pre-bent. The camera can be oriented collinear with the longitudinal axis of the scope or can be positioned at an oblique angle. Some scopes are flexible over their entire length, others contain a proximal rigid straight portion with a distal bendable portion.

Figure 9 visualizes a general endoscope geometry that encompasses all the above configurations, along with the frames of interest. The incision frame {i} is defined at the incision point and is common for all robotic endoscope holders. The z-axis of {i} is the inward-pointing normal of the body wall. A frame {t} is connected to the distal tip of the straight portion of the endoscope shaft, with its z-axis pointing along the shaft axis. In the most general case, the camera frame {c} is located at an offset and rotated with respect to {t}. The offset can reproduce for tip articulation, for stereo camera lens separation. The rotation can account for oblique-viewing endoscopes. As such, this endoscope model can describe complex endoscopes, such as the articulating 3D video endoscope EndoEye Flex 3D (Olympus, Japan) (Figure 10).

Starting from this general endoscope model, different visual servoing approaches for REC will be detailed next. The visual servoing approaches strive to determine the endoscope motion that is needed to match the position s = x y z T of the feature of interest, expressed in the camera frame, with its desired position s * = x * y * z * T , while taking into account the presence of the incision point. The visual servoing approaches assume that the robot endoscope holder has three active DoFs that can produce any linear velocity of the endoscope tip. In order to obtain a fully determined endoscope motion, it is further assumed that the remaining rolling DoF about the endoscope axis is not controlled by the visual servoing controller, but by an external controller. Note that, as was pointed out in Section 3, this DoF could be employed to control the camera horizon.

The following notation will be used in the subsequent sections: a rotation of angle ξ about the axis i will be denoted by R _i (ξ). For a transformation from a frame {j} to a frame {i}, the notation T j i will be used, consisting of a rotation R j i and a translation P j i . Further, the twist vector t = v T ω T T is defined as the concatenation of a linear velocity v and an angular velocity ω . For all kinematic variables, the reference frames will be indicated with a trailing superscript. For the features s and the error e in the camera frame, the trailing superscript ^c is mostly omitted for brevity.

The instrument localization pipeline from Section 4.2 yields the tooltip image coordinates u _l , v _l and the disparity d _x . The 3D tooltip position, required for the visual servoing methods, is estimated from these measurement data, through an EKF. The state transition model describes a linear tooltip motion of exponentially decreasing velocity, partially expressed in frames {i} and {c} to limit the non-linearity, and the observation model implements the pinhole camera model: x k = g x k − 1 , t c , k c + ϵ k , = s k − 1 c + Δ T R i c s ̇ k − 1 i + L 3 D s k − 1 c t c , k c λ s s ̇ k − 1 i + ϵ k , (4) z k = h x k + δ k = x k f x z k + c x y k f y z k + c y b c f x z k + δ k , (5) Where x k = s k c T s ̇ k i T T is the state vector, s k c is the tooltip position in the camera frame {c}, s ̇ k i is the tooltip velocity in the incision frame {i}, t c c is the camera twist, L _3D is the 3D interaction matrix from Eq. 20, λ _s is a reduction factor < 1 (governing the exponential decrease of s ̇ i ), z k = u l , k v l , k d x , k T is the observation vector, f _x , f _y , c _x , c _y are the intrinsic camera parameters, b _c the distance between the optical centres of the (left and right) cameras, and ϵ _k and δ _k are the usual process and observation noises. The velocity reduction factor λ _s is introduced to scale down the contribution of dead reckoning during measurement gaps.

IBVS aims to determine the camera motion to move the 2D projection of the 3D feature point s to its desired position in the image plane. Assuming a pinhole camera model, the 2D projection s _n is obtained by expressing s in normalized camera coordinates: s n = x n y n T = x / z y / z T . (6)

Classically, the relation between the camera twist t c c and the 2D feature point velocity s ̇ n is expressed by the interaction matrix L _2D (Chaumette and Hutchinson, 2008): s ̇ n = L 2 D t c c , (7) Where L 2 D = − 1 / z 0 x n / z x n y n − 1 + x n 2 y n 0 − 1 / z y n / z 1 + y n 2 − x n y n − x n . (8)

In this equation, it is assumed that the camera has six DoFs, while only three are available for the endoscope control. To incorporate these constraints, the camera twist t c c needs to be mapped first to the twist of tip of the endoscope’s straight portion t t t : t c c = J t c t t t , (9) Where J t c is the well-known expression for a twist transformation: J t c = R t c − R t c p c t × 0 R t c . (10)

The operator [ ]_× is the operator for the skew-symmetric matrix. The incision constraint introduces a coupling between the linear tip velocity v t t and angular tip velocity ω t t and can be expressed as: t t t = J i v t t , (11) With the incision transformation J i = 1 0 0 0 1 0 0 0 1 0 − 1 / l 0 1 / l 0 0 0 0 0 , (12) And the inserted endoscope length l = ‖ p t i ‖ . Combining (7–12) yields the modified interaction matrix L 2 D ′ : L 2 D ′ = L 2 D J t c J i (13) Which maps v t t to s ̇ n . This matrix is a generalized form for the modified interaction matrix presented in (Osa et al., 2010).

As is customary in visual servoing, the error in the normalized image space is expressed as e n = s n − s n * (14) And the control law enforces an exponential decay of the error: e ̇ n = − λ e n , (15) Characterized by the time constant τ = 1/λ. For a constant s n * , this yields the desired endoscope tip velocity: e ̇ n = s ̇ n = L 2 D ′ v t t = − λ e n ⇒ v t t = − λ L 2 D ′ + e n . (16)

IBVS only seeks to optimize the 2D projected position s _n of the target point s in the image plane. As such IBVS alone is insufficient to control the 3D position of the endoscope. A decoupled depth controller can be added to control the third DoF. This was proposed in (Chen et al., 2018) and will be generalized here.

The depth controller acts along the z-axis of the camera frame {c} and uses the kinematic relation between the camera twist t c c and the change in the depth z of s : z ̇ = L z t c c , (17) Where L z = 0 0 − 1 − y x 0 . (18)

To reduce the depth error e _z = z − z*, concurrently with the image-space error e _n , a similar reasoning as with IBVS can be followed, yielding: e ̇ n e ̇ z = s ̇ n z ̇ = L 2 D L z J t c J i v t t = L 2 D ′ L z ′ v t t = − λ e n e z ⇒ v t t = − λ L 2 D ′ L z ′ + e n e z . (19)

To differentiate between directions, it is possible to define λ as a diagonal matrix, rather than as a scalar.

Instead of decoupling the control in the image plane and the depth control, the 3D feature s can also be used directly to define the 3D motion of the endoscope. This requires a 3D interaction matrix L _3D , which can be derived from the kinematic equations of motion for the stationary 3D point s in the moving camera frame {c}: s ̇ = − v c c − ω c c × s = L 3 D t c c , (20) With L 3 D = − I s × . (21)

As before, the modified interaction matrix L 3 D ′ can be obtained by including the offset of the tip frame with respect to the camera frame and the incision constraint. The desired endoscope velocity that ensures an exponential decay of the error e = s − s * follows then from: e ̇ = s ̇ = L 3 D J t c J i v t t = L 3 D ′ v t t = − λ e ⇒ v t t = − λ L 3 D ′ + e . (22)

PBVS identifies the camera pose, with respect to an external reference frame, that produces the desired view upon the 3D feature s and moves the camera towards this pose. As mentioned before, the camera pose is constrained to three DoFs due to the presence of the incision point and the separate horizon stabilization. Finding the desired camera pose, while taking into account its kinematic constraints, involves solving the inverse kinematics for the endoscope as defined in Figure 9.

The forward kinematics of the endoscope can be described as a function of three joint variables (θ ₁, θ ₂, l). Based on these variables, any endoscope pose can be reached by applying successive operations in a forward kinematics chains. When these joint variables are set to zero, the endoscope system is in a configuration where the incision frame {i} coincides with the distal tip frame {t}, while the camera frame is offset by p c t and rotated by R c t = R x ( α ) , with α the oblique viewing angle of the endoscope. Starting from this configuration, θ ₁ rotates {t} about its y-axis, then θ ₂ rotates it about its x-axis and finally l translates it along its z-axis. This leads to the following forward kinematic equations, expressed in the reference frame {i}: T c i s ̃ = T c i * s ̃ * = T t i * T c t s ̃ * (23) = R y θ 1 * R x θ 2 * 0 0 T 1 I l * e ̂ 3 0 T 1 R x α p c t 0 T 1 s ̃ * , With e ̂ 3 = 0 0 1 T the unit vector along the z-direction. The trailing * designates a desired value, different from the current value. The ̃ signifies the homogeneous representation of a 3D vector. Eq. 23 constitutes a system of three equations in the unknowns ( θ 1 * , θ 2 * , l * ) . 1 elaborates the analytic solution to this inverse kinematics problem.

The solution of the inverse kinematics can be inserted in the forward kinematics equations to obtain the desired position of the distal endoscope tip p t i * : p t i * = l * ⁡ sin θ 1 * cos θ 2 * − l * ⁡ sin θ 2 * l * ⁡ cos θ 1 * cos θ 2 * , (24) Which straightforwardly leads to the position error of the distal tip, expressed with respect to the incision frame {i}: e i = p t i − p t i * . (25)

When an exponential decaying error is required, the desired endoscope velocity becomes: v t i = e ̇ i = − λ e i (26) And can be expressed in the frame {t} as: v t t = − λ R i t e i . (27)

From the graphs, it is clear that IBVS differs from the other approaches in that, by construction, it does not attain the desired depth z* in the final endoscope pose. Moreover, IBVS also doesn’t guarantee a constant depth z. Consequently, z will drift towards undesired depths over time. In some configurations, this can be counteracted by separately controlling l to stay constant, but this does not hold for the general case. IBVS alone is thus unsuitable for REC and 3D information about s is a requirement.

Both IBVS+DC and 3D IBVS linearize the visual servoing problem in the camera frame. This enables a desired exponential decay of the targeted errors, but does not produce a well-controlled endoscope motion in Cartesian space. It can be seen from Figure 11 that the trajectories for these methods deviate from the straight trajectory that is accomplished by PBVS, and more so for large initial errors. As space is limited in REC and the environment delicate, the straight trajectory of PBVS appears favourable compared to its alternatives.

IBVS typically yields more accurate visual servoing results than PBVS, because the feedback loop in IBVS-based methods can mitigate camera calibration errors (excluding stereo calibration errors). However, the objective in REC often is to keep s inside a specific region of the endoscopic image (cf. position hysteresis), rather than at an exact image coordinate. The importance of the higher accuracy of IBVS is thus tempered by this region-based control objective: small calibration inaccuracies are acceptable. Therefore, and in contrast to the claims in (Osa et al., 2010), it can be argued that the predictability of a straight visual servoing trajectory outweighs the importance of the visual servoing accuracy. This argumentation points out why PBVS is the preferred approach for REC, especially when large initial errors exist.

If the accuracy of PBVS would need to be enhanced, e.g., when significant calibration errors exist, it is possible to apply a hybrid visual servoing method. PBVS can be used until the initial error drops below a certain threshold and from there, the visual servoing controller gradually switches to an IBVS-based approach for refinement, by applying a weighted combination of the desired tip velocities v t t computed by each visual servoing method. The curved shape of IBVS trajectories can thus be suppressed. In experiments that are not further documented here, it was observed that 3D IBVS, which ascertains an exponential Cartesian error decay, provided a more predictable and thus more desirable endoscope behaviour than IBVS+DC. To ensure robustness against potential calibration errors, the hybrid combination of PBVS and 3D IBVS was thus selected for the experiments in Section 6. Figure 11 shows the simulated performance of the hybrid visual servoing approach, which gradually transitions from PBVS to 3D IBVS when the error ‖ e _n ‖ in the normalized image space goes from 0.6 to 0.3.