Robotic-assisted surgery (RAS) has transitioned surgery from direct manual interaction with organs and tissues to digitally mediated interfaces. This transition has introduced a haptic void, where the surgeon’s perception of tissue tension and texture is mediated primarily through visual interfaces rather than physical feedback. This sensory decoupling presents substantial technical as well as cognitive challenges, as the absence of tactile information can compromise the surgeon’s ability to assess tissue and to detect mechanical stress. While the integration of haptic feedback is widely acknowledged as a rigorous technical challenge, it represents a deeper conceptual and design challenge rooted in human perception. Transitioning from visual-only feedback to a multisensory interface requires a precise understanding not only of how physical forces are simulated, but how intangible sensory cues are synthesized to inform the surgeon’s action and perception. Ultimately, aligning these distinct modalities is essential for achieving robotic embodiment to a point at which robotic systems may transcend their role as a remote mechanical interface to function as extensions of the surgeon’s own sensorimotor system.

To address these challenges, we developed the Haptic Interaction Toolkit (HIT), a mixed-reality (MR) robotic console designed for the conceptual investigation of haptic-guided surgical maneuvers. MR is particularly suitable for this purpose because it preserves the co-location of hands, instruments, and interfaces, keeping the users’ proprioception and postural behavior and the physical setup aligned with the corresponding virtual elements. Combined visual overlays and haptic cues are intended to support depth perception, tissue interaction judgments, and force modulation during experimentation–advantages that purely virtual environments, which decouple the user from any physical grounding, cannot provide in the same way. Rather than aiming for mechanical fidelity, the HIT serves as a low-cost, reproducible research tool that isolates and evaluates diverse haptic strategies across configurable visuo-haptic scenarios. By low-cost we refer to a setup based on commercially available haptic devices, off-the-shelf VR hardware and standard software licenses, which together cost far less than proprietary clinical robotic platforms. In contrast, current robotic consoles require substantial capital investment and have closed architectures, making it difficult and costly to implement and test alternative haptic modalities. By prioritizing modularity and ease of implementation, the toolkit is intended to democratize surgical haptic research, opening the field to communities beyond engineering including design, ergonomics, and human-computer interaction.

The toolkit comprises three visuo-haptic surgical scenarios that expose users to patterns of force-tissue interaction, offering insight into how different touch cues shape task performance and suggest new modes of thinking about how the sense of touch can affect the surgeon’s cognitive and perceptual experience in virtual reality (VR) and MR environments. We position the HIT as a versatile, reproducible platform for the investigation of haptic modalities, with the broader aim of refining the interface between surgeon, robot, and patient.

In contrast to existing VR/MR simulators and closed clinical robotic platforms, HIT is designed explicitly as an open, modular experimental infrastructure. It combines a MR console, dual force-feedback controllers, precise real/virtual registration and an open Unity-based software pipeline, enabling rapid reconfiguration of tasks, systematic manipulation of haptic variables and sharing of scenarios and code via a public repository. This open, hardware-agnostic design is the core novelty of HIT: it offers a dedicated, low-risk testbed for surgical haptics research that complements, rather than replaces, platforms such as the da Vinci Research Kit.

The development of robotic and image-guided surgery has profoundly reshaped intraoperative practice, yet the surgeon’s manual skills remain fundamentally rooted in finely tuned tactile perception. In both conventional, open and minimally invasive procedures, surgeons rely on force and texture cues to discriminate between tissue types and potential alterations, modulate instrument–tissue interaction, and maintain safe force application during dissection, suturing, and knot tying (Vleugels and Rahimi, 2024; Okamura, 2009; Othman et al., 2022). In this context, robotic systems for telemanipulation of instruments have increased technical capabilities in terms of dexterity, precision, and ergonomics (Marcus et al., 2024). However, they often do so at the cost of attenuated or absent haptic cues at the surgeon’s console. Despite recent integration of haptic or force-reflection capabilities in systems such as the latest Intuitive da Vinci 5 model, these solutions are often constrained by limited degrees of freedom and limited coverage of the full range of tool-tissue interactions encountered in complex surgery (Awad et al., 2024). Experimental evidence in RAS consistently indicates that restoring haptic information improves both safety and quality of key procedural steps. Studies demonstrate that adding force or tactile feedback improves performance in palpation, tissue manipulation, and suturing, leading to more consistent force control, reduced tissue trauma, and lower intraoperative blood loss (Colan et al., 2022; Bergholz et al., 2023). These findings suggest that the absence of haptic feedback is not merely an ergonomic drawback, but a substantive technical limitation in current RAS platforms, with direct implications for patient safety and the surgeon’s ability to execute fine motor tasks. Recent engineering reviews emphasize that integrating advanced haptic feedback technologies into surgical simulators and robot-assisted systems is essential to support safe, realistic manipulation of tissues and instruments during minimally invasive procedures, motivating richer force and tactile channels for training and intraoperative use (Laga Boul-Atarass et al., 2025). Despite these documented benefits, a marked gap remains between the importance of visuo-haptic feedback and its implementation in current simulators and robotic platforms. Existing systems often rely on proprietary hardware and closed architectures that limit systematic testing of alternative haptic rendering strategies or tailored force-feedback profiles (Gani et al., 2022). Many simulators also provide only coarse or non-contextual force cues, so subtle changes in tissue stiffness and elasticity are poorly conveyed (Huang et al., 2020; Neidhardt et al., 2023). In training settings this limits targeted manipulation of specific haptic features, such as directional cues, multi-finger interaction, or event-based feedback, and evaluation of their role in skill acquisition and safe tool–tissue interaction (Dawe et al., 2014; Lim and Yap, 2024).

Yet existing surgical simulators, even those incorporating haptic feedback, have predominantly been designed around specific engineering objectives or fixed training curricula. Research based on the da Vinci Research Kit has demonstrated the value of force reflection during grasping, palpation, and incision tasks (Saracino et al., 2019) and explored sensorless haptic feedback strategies for teleoperation (Yilmaz et al., 2024), while work based on the VerroTouch (a robotic-assisted minimally invasive surgery accessory that provides surgeons with audio and haptic feedback of instrument vibrations) has shown that instrument vibration feedback is preferred by surgeons and can reduce workload during procedures (Gomez et al., 2025). These studies provide strong evidence that haptic feedback shapes surgical performance but remain bound to specific robotic hardware and feedback modalities. In parallel, low-cost simulators using commercial haptic devices and VR have emerged as accessible alternatives for robotic surgery training (Neri et al., 2024), and high-fidelity physical platforms have been validated for procedural skill assessment (Gamberini et al., 2024). However, these systems are designed around task repetition and proficiency evaluation rather than experimental manipulation of haptic variables. Few platforms have been built with the architectural flexibility to isolate, systematically vary, and compare individual haptic parameters, such as stiffness rendering, graded spatial resistance, or vibrotactile cues, within a controlled perceptual framework that is independent of a specific clinical robot.

The HIT is developed to address these interconnected gaps identified in the current landscape: the hardware dependency of haptic feedback research, which limits findings to specific robotic platforms; the absence of systems designed for experimental manipulation of haptic variables; and the lack of an architecturally flexible infrastructure capable of isolating and systematically varying individual haptic parameters within a controlled perceptual framework. Conceived as a modular system for research and development, it targets immersive visuo-haptic experimentation in RAS scenarios. Building on the conceptual foundation (Akbal et al., 2026) and informed by a validated background investigation into the need for haptic feedback in RAS (Akbal et al., 2025), we offer the details of the technical implementation of HIT as a research tool. Rather than presenting yet another fixed-function simulator, the toolkit aims to provide an extensible visuo-haptic infrastructure in which sensing, actuation, and control strategies can be rapidly configured and empirically evaluated. It is designed to support realistic tissue interaction with controllable force profiles, enable fine-grained experimental manipulation of haptic channels, and facilitate integration into existing VR and RAS-style simulators through standardized interfaces.

The HIT serves as a research tool for next-generation surgical interfaces, enabling research teams to develop and evaluate novel haptic and tactile interaction strategies. The system facilitates the rendering of material properties, such as the shape, texture, and mass of anatomical structures as well as immaterial information, including virtual barriers for tissue protection or guided trajectories for enhanced precision. These interaction concepts are demonstrated within three initial experimental scenarios which function as proof-of-concept scenarios for representative clinical use cases as defined by Akbal et al. (2026).

The first scenario involves discrimination between soft and rigid regions of virtual and physical liver models representing healthy and tumorous tissue. This simulates intraoperative palpation and explores how combined visual and haptic cues support safe margin definition and resection planning. The second scenario examines instrument navigation and tissue preparation near vulnerable structures, such as the ureter, where graded haptic resistance conveys safety boundaries and restricted movement within a virtual “guardian” zone to improve spatial awareness. The third scenario simulates fatty tissue dissection. It integrates dual haptic interfaces and vibrotactile sensors that signal instrument activation states. Here, the interaction between continuous tissue feedback and distributed vibration patterns was used to study how multimodal haptic information informs precise control during energy-based tissue dissection.

The HIT illustrates how a configurable MR robotic console can make complex visuo-haptic interaction experiments both technically feasible and practically reproducible. By combining a custom physical console, dual force-feedback devices, and an open Unity-based software pipeline, the system supports precise real–virtual registration, structured calibration, and systematic manipulation of haptic conditions. The three proof-of-concept experimental scenarios (liver stiffness discrimination, ureter boundary protection, and fatty tissue dissection) illustrate how variations in stiffness rendering, safety-zone feedback, and distributed vibrotactile cues can be prototyped and studied without access to clinical robotic systems. The documented limitations in hardware fidelity, latency, and SDK transparency point to concrete engineering challenges that must be addressed to mature haptic feedback in surgical simulation; at the same time, these constraints clarify the toolkit’s intended scope: it is a platform for interaction design and perceptual research, not a biomechanical ground truth. We deliberately chose organ-specific scenarios rather than fully abstract tasks to preserve recognizable surgical structure and maintain face validity for clinicians. Given the limited biomechanical and visual realism of these simulations, studies conducted on HIT should therefore be interpreted as conceptual and comparative evaluations of haptic interaction strategies under controlled conditions, rather than as direct evidence about absolute performance or outcomes in real robotic surgery. A key next step is to extend the current implementation. The toolkit’s modular architecture and open asset library make it straightforward to extend the scenario set beyond the three visceral surgery use cases presented here, incorporating additional organ systems and task types including bowel, vascular, and urologic targets, as well as traction, weightlifting, and safe retraction, identified as priorities by robotic surgeons (Akbal et al., 2025). The MR workflow that links physical phantoms to spatially registered virtual counterparts is particularly well suited to testing new physical and virtual organ models and investigating how varying degrees of real–virtual correspondence affects presence and haptic judgment.

More broadly, the framework offers a foundation for systematic exploration of further multisensory integration in robotic surgery, where future work can combine force feedback and vibrotactile feedback with additional modalities such as spatialized audio communication or olfactory cues, to approximate the sensory contingencies on which surgeons rely, with the longer-term aim of transferring refined interaction strategies into next-generation clinical robotic systems.