Abstract
Robotic-assisted surgery provides superior fine motor control of instruments but typically deprives surgeons of haptic cues. We present the Haptic Interaction Toolkit, a mixed-reality robotic console that integrates dual force-feedback devices, and a Unity-based software pipeline, along with details of its configuration and implementation. The toolkit is a reproducible technology platform that enables real/virtual registration, configurable visuo-haptic interactions, and replication via source code. Three exemplar experimental scenarios (discrimination of liver stiffness, protection of critical structures, and fatty tissue dissection next to sensitive structures) demonstrate how the system can be used to prototype and evaluate novel haptic interaction concepts for future robotic systems.
1 Introduction
Robotic-assisted surgery (RAS) has transitioned surgery from direct manual interaction with organs and tissues to digitally mediated interfaces. This transition has introduced a haptic void, where the surgeon’s perception of tissue tension and texture is mediated primarily through visual interfaces rather than physical feedback. This sensory decoupling presents substantial technical as well as cognitive challenges, as the absence of tactile information can compromise the surgeon’s ability to assess tissue and to detect mechanical stress. While the integration of haptic feedback is widely acknowledged as a rigorous technical challenge, it represents a deeper conceptual and design challenge rooted in human perception. Transitioning from visual-only feedback to a multisensory interface requires a precise understanding not only of how physical forces are simulated, but how intangible sensory cues are synthesized to inform the surgeon’s action and perception. Ultimately, aligning these distinct modalities is essential for achieving robotic embodiment to a point at which robotic systems may transcend their role as a remote mechanical interface to function as extensions of the surgeon’s own sensorimotor system.
To address these challenges, we developed the Haptic Interaction Toolkit (HIT), a mixed-reality (MR) robotic console designed for the conceptual investigation of haptic-guided surgical maneuvers. MR is particularly suitable for this purpose because it preserves the co-location of hands, instruments, and interfaces, keeping the users’ proprioception and postural behavior and the physical setup aligned with the corresponding virtual elements. Combined visual overlays and haptic cues are intended to support depth perception, tissue interaction judgments, and force modulation during experimentation–advantages that purely virtual environments, which decouple the user from any physical grounding, cannot provide in the same way. Rather than aiming for mechanical fidelity, the HIT serves as a low-cost, reproducible research tool that isolates and evaluates diverse haptic strategies across configurable visuo-haptic scenarios. By low-cost we refer to a setup based on commercially available haptic devices, off-the-shelf VR hardware and standard software licenses, which together cost far less than proprietary clinical robotic platforms. In contrast, current robotic consoles require substantial capital investment and have closed architectures, making it difficult and costly to implement and test alternative haptic modalities. By prioritizing modularity and ease of implementation, the toolkit is intended to democratize surgical haptic research, opening the field to communities beyond engineering including design, ergonomics, and human-computer interaction.
The toolkit comprises three visuo-haptic surgical scenarios that expose users to patterns of force-tissue interaction, offering insight into how different touch cues shape task performance and suggest new modes of thinking about how the sense of touch can affect the surgeon’s cognitive and perceptual experience in virtual reality (VR) and MR environments. We position the HIT as a versatile, reproducible platform for the investigation of haptic modalities, with the broader aim of refining the interface between surgeon, robot, and patient.
In contrast to existing VR/MR simulators and closed clinical robotic platforms, HIT is designed explicitly as an open, modular experimental infrastructure. It combines a MR console, dual force-feedback controllers, precise real/virtual registration and an open Unity-based software pipeline, enabling rapid reconfiguration of tasks, systematic manipulation of haptic variables and sharing of scenarios and code via a public repository. This open, hardware-agnostic design is the core novelty of HIT: it offers a dedicated, low-risk testbed for surgical haptics research that complements, rather than replaces, platforms such as the da Vinci Research Kit.
2 Related work
The development of robotic and image-guided surgery has profoundly reshaped intraoperative practice, yet the surgeon’s manual skills remain fundamentally rooted in finely tuned tactile perception. In both conventional, open and minimally invasive procedures, surgeons rely on force and texture cues to discriminate between tissue types and potential alterations, modulate instrument–tissue interaction, and maintain safe force application during dissection, suturing, and knot tying (; ; ). In this context, robotic systems for telemanipulation of instruments have increased technical capabilities in terms of dexterity, precision, and ergonomics (). However, they often do so at the cost of attenuated or absent haptic cues at the surgeon’s console. Despite recent integration of haptic or force-reflection capabilities in systems such as the latest Intuitive da Vinci 5 model, these solutions are often constrained by limited degrees of freedom and limited coverage of the full range of tool-tissue interactions encountered in complex surgery (). Experimental evidence in RAS consistently indicates that restoring haptic information improves both safety and quality of key procedural steps. Studies demonstrate that adding force or tactile feedback improves performance in palpation, tissue manipulation, and suturing, leading to more consistent force control, reduced tissue trauma, and lower intraoperative blood loss (; ). These findings suggest that the absence of haptic feedback is not merely an ergonomic drawback, but a substantive technical limitation in current RAS platforms, with direct implications for patient safety and the surgeon’s ability to execute fine motor tasks. Recent engineering reviews emphasize that integrating advanced haptic feedback technologies into surgical simulators and robot-assisted systems is essential to support safe, realistic manipulation of tissues and instruments during minimally invasive procedures, motivating richer force and tactile channels for training and intraoperative use (). Despite these documented benefits, a marked gap remains between the importance of visuo-haptic feedback and its implementation in current simulators and robotic platforms. Existing systems often rely on proprietary hardware and closed architectures that limit systematic testing of alternative haptic rendering strategies or tailored force-feedback profiles (). Many simulators also provide only coarse or non-contextual force cues, so subtle changes in tissue stiffness and elasticity are poorly conveyed (; ). In training settings this limits targeted manipulation of specific haptic features, such as directional cues, multi-finger interaction, or event-based feedback, and evaluation of their role in skill acquisition and safe tool–tissue interaction (; ).
Yet existing surgical simulators, even those incorporating haptic feedback, have predominantly been designed around specific engineering objectives or fixed training curricula. Research based on the da Vinci Research Kit has demonstrated the value of force reflection during grasping, palpation, and incision tasks () and explored sensorless haptic feedback strategies for teleoperation (), while work based on the VerroTouch (a robotic-assisted minimally invasive surgery accessory that provides surgeons with audio and haptic feedback of instrument vibrations) has shown that instrument vibration feedback is preferred by surgeons and can reduce workload during procedures (). These studies provide strong evidence that haptic feedback shapes surgical performance but remain bound to specific robotic hardware and feedback modalities. In parallel, low-cost simulators using commercial haptic devices and VR have emerged as accessible alternatives for robotic surgery training (), and high-fidelity physical platforms have been validated for procedural skill assessment (). However, these systems are designed around task repetition and proficiency evaluation rather than experimental manipulation of haptic variables. Few platforms have been built with the architectural flexibility to isolate, systematically vary, and compare individual haptic parameters, such as stiffness rendering, graded spatial resistance, or vibrotactile cues, within a controlled perceptual framework that is independent of a specific clinical robot.
The HIT is developed to address these interconnected gaps identified in the current landscape: the hardware dependency of haptic feedback research, which limits findings to specific robotic platforms; the absence of systems designed for experimental manipulation of haptic variables; and the lack of an architecturally flexible infrastructure capable of isolating and systematically varying individual haptic parameters within a controlled perceptual framework. Conceived as a modular system for research and development, it targets immersive visuo-haptic experimentation in RAS scenarios. Building on the conceptual foundation () and informed by a validated background investigation into the need for haptic feedback in RAS (), we offer the details of the technical implementation of HIT as a research tool. Rather than presenting yet another fixed-function simulator, the toolkit aims to provide an extensible visuo-haptic infrastructure in which sensing, actuation, and control strategies can be rapidly configured and empirically evaluated. It is designed to support realistic tissue interaction with controllable force profiles, enable fine-grained experimental manipulation of haptic channels, and facilitate integration into existing VR and RAS-style simulators through standardized interfaces.
3 System design/method description
3.1 System overview
The HIT is built as an open-source research platform in which the user is seated at a physical console that replicates the ergonomics of a clinical robotic surgery system and performs physical tasks using physical instruments (Figure 1A). These physical interactions are continuously matched in a MR environment, where the user is situated within a virtual operating room but can see their virtual hands and virtual counterparts of the physical tools, maintaining visuo-proprioceptive correspondence between the real and virtual workspaces (Figure 1B). The system integrates a physical master console coupled to a Unity-based VR simulation and data-logging framework hosted in a public GitHub repository for community-driven development1. The reference setup comprises a custom-built console housing two 3D Systems Touch haptic devices that are fixed to a stationary base, paired with a Meta Quest 3 HMD, and a high-performance host PC running the Unity application. This architecture supports real-time bidirectional communication between the physical console and the virtual scene ensuring visuo-haptic coupling (Figure 2).
FIGURE 1
FIGURE 2
3.2 Hardware configuration
3.2.1 Physical console design and setup
The physical robotic console (Figures 3A,B) was designed to reproduce the ergonomics and key geometric measurements of the Intuitive da Vinci Xi surgical system console installed at Charité–Universitätsmedizin Berlin, Campus Virchow-Klinikum (Figure 3C). The frame is constructed from high-grade metal components, combined with laser-cut and 3D-printed parts that must conform to tight tolerances to maintain structural rigidity and accurate positioning of the haptic devices (Figure 4). Any deviation in the manufacturing or assembly of these parts risks introducing spatial misalignments that can degrade the fidelity of both force rendering and positional tracking. A detailed 3D twin of the console is implemented in the VR environment (Figure 5).
FIGURE 3
FIGURE 4
FIGURE 5
The console itself is mounted on a mobile base with lockable wheels and incorporates two recessed haptic-device mounts mimicking the layout of the da Vinci surgeon console, alongside ergonomically contoured forearm support (Figure 3A). To preserve visuo-haptic congruence, a calibration protocol is executed before each experimental session (see Section 3.3.3 for the full procedure). Accurate fabrication and assembly of the metal and 3D-printed components are therefore critical not only for mechanical stability, but also for achieving consistent tactile and proprioceptive feedback, so that users experience their body posture and applied forces as reliably mirrored in VR.
The physical console can be replicated using the provided digital fabrication resources and assembly instructions, which include all required CAD files and fabrication drawings for 3D-printed and machined metal components2,3. The main console elements comprise the console frame and table surface, mounting bases for the Touch devices, the liver model and liver platform, a calibration platform, and a pyramid-shaped, 3D-printed stylus tip that is press-fitted onto the distal nub of the Meta Quest 3 controller to enable precise controller-based calibration, as well as arm-rest components, all of which are manufactured according to the supplied specifications and then assembled following an illustrated, step-wise construction guide provided as supplementary material. After assembling the frame and supporting structures, the two 3D Systems Touch devices are mounted at their specified positions and orientations, with particular attention to lateral spacing, height, and alignment relative to the arm rests, so that the physical workspace matches its virtual counterpart in Unity. Finally, the Meta Quest 3 HMD and the host PC are installed and connected according to the system’s hardware schematic, completing the physical console setup and enabling seamless integration with the VR simulation environment and associated calibration procedures.
3.2.2 Haptic devices
The toolkit employs two 3D Systems “Touch” devices (Figure 6), each providing 6-degrees-of-freedom (DOF) positional sensing and 3-DOF force feedback at the end-effector. The adjustable stylus of each device conveys a range of haptic cues, including stiffness, viscosity, damping, and friction, as well as programmable force thresholds and feedback gains that can be tuned to represent different tissue properties and interaction scenarios. When mounting the devices in the console, the lateral spacing between the two Touch units on the physical frame is set to 30 cm, corresponding to 0.30 units in the Unity coordinate system, to maintain spatial congruence between the physical and virtual workspaces. This spatial matching is essential, as offsets between real and virtual device positions can significantly alter users’ proprioceptive perception and degrade task performance. Using two identical haptic devices therefore requires careful configuration of their relative placement to avoid excessive workspace overlap while keeping both styluses within the user’s comfortable reach. The devices are connected to the host PC via USB 2.0/3.0 links, supporting update frequencies of up to 2 kHz for position sensing and force rendering, depending on driver and plugin configuration. The Haptics Direct plugin maintains a dedicated haptic servo thread, decoupled from Unity’s rendering frame rate, that updates device position sensing and impedance-based force output at up to 1 kHz.
FIGURE 6
The HIT further integrates the Hapticlabs sensor-actuator DevKit, which enables rapid prototyping of vibrotactile cues by mapping external sensor events into configurable haptic patterns. In our implementation, the sensor module is used exclusively in the third experimental scenario, where it functions as an instrument on/off indicator, providing a discrete haptic signal to notify users about changes in tool state. The actuator is mounted on a custom 3D-printed wristband, while the sensor elements are wrapped around the user’s dorsal side of the hand, to maintain stable skin contact and consistent signal transmission during interaction (Figure 7).
FIGURE 7
3.2.3 VR head-mounted display and host computer
The VR display is provided by a Meta Quest 3 HMD (Meta Platforms Inc., USA, California), operated via the appropriate Meta/OpenXR integration for wired PC-VR use. For the reference implementation, the host computer AMD Ryzen 3950X (Advanced Micro Device Inc., USA, California) is equipped with Nvidia GeForce RTX 3090 GPU, and 64 GB of RAM, providing sufficient resources for high-fidelity graphical rendering and low-latency haptic computation. Haptic interaction is delivered through the two 3D Systems Touch devices (3D Systems Corporation, USA, North Carolina), which are interfaced with the Unity application using dedicated haptic middleware and custom plugins for synchronized data logging. A consolidated list of hardware and software requirements, together with step-by-step setup instructions, is provided in the README of the repository (see footnote 1).
3.3 Software architecture
3.3.1 Game engine and core libraries
The reference implementation is provided as a Unity project (Unity 6000.0.58f2, Windows target, Unity Technologies Inc., USA, California) that can be accessed from the public repository4. The project is configured to interface with the two 3D Systems Touch devices via the 3D Systems driver stack and the Unity-compatible “Haptics Direct for Unity V1” plugin5,6, which enables force feedback within the simulation. Meta Quest 3 and OpenXR integration is provided through Meta’s Unity SDK and OpenXR packages, which maintain PC-VR functionality, including hand tracking and passthrough, for the Meta Quest 3 HMD7. The application can be executed either as a standalone PC-VR build or via Quest Link or Air Link, with the HMD acting as the primary display for the MR environment.
The software stack further integrates the Hapticlabs sensor-actuator DevKit (Hapticlabs, Germany, Dresden) as a vibrotactile feedback channel. Hapticlabs is connected to Unity through a dedicated Unity package8 and the Hapticlabs Manager prefab, which can be configured to communicate with Hapticlabs Studio9 via TCP or directly with a Hapticlabs Satellite over a serial link after uploading haptic tracks. This integration provides a complementary vibrotactile channel alongside the force feedback of the Touch devices.
3.3.2 Main menu and scene navigation
The main menu functions as the central navigation interface of the application, enabling users to select and load individual experimental scenarios. The user interface (UI) is presented as a fixed panel positioned at eye level to ensure consistent spatial alignment and ergonomic accessibility.
3.3.3 Calibration
Upon startup, the user completes a calibration procedure to align the physical console with the virtual environment and to calibrate the haptic devices. The calibration scene, operating in mixed reality, is the first mandatory step before accessing experimental scenarios. Through a dynamic UI panel, step-by-step guidance is provided. The procedure includes two stages: first, the user aligns the virtual scene with the physical calibration platform by placing four virtual reference points into corresponding physical sockets using the right Meta Quest controller (Figure 8). This step establishes global spatial alignment between real and virtual environments. Next, the user calibrates the haptic devices by aligning four reference points using the right and left Touch styluses, ensuring precise correspondence between physical and virtual tools.
FIGURE 8
The alignment computation is implemented using the Kabsch algorithm, which calculates the optimal rigid transformation between corresponding point sets10. All instructions are presented sequentially through the Meta Quest 3 HMD.
In practice, calibration procedure yields a physical–virtual registration that is accurate to within a few millimeters and a few degrees over most of the workspace, with larger deviations near steep curvatures and at phantom edges. These residual errors arise from manufacturing tolerances and small deformations of the organ models and mounting structures, which cannot be fully corrected by a single global calibration transform.
The full calibration takes 1–2 min to complete and can be carried out by non-expert users after a brief familiarization. Residual misalignment mainly arises from misplacement of reference points and small variations in how firmly devices and phantoms are seated. Registration accuracy was assessed qualitatively by visual inspection of tool-phantom correspondence and test movements across salient anatomical landmarks. Because the Meta Quest 3 uses inside-out tracking, spatial drift over longer sessions can degrade real–virtual registration. In the current implementation, we mitigate this by recalibrating before each experimental block and limiting task segments to a few minutes, during which no console-relative drift was observed on visual inspection. Long-term tracking stability and its impact on coordinate alignment have not yet been quantified and remain a target for future technical validation.
3.3.4 Touch device and Hapticlabs integration in unity
The transformation from the Touch device’s native coordinate space to Unity’s coordinate system is handled by the device’s SDK and plugin. Spatial correspondence between the physical device and its virtual counterpart is further established through the session-specific calibration procedure (Section 3.3.3). Haptic materials and effects are configured through the SDK’s exposed interfaces; only high-level parameters such as material properties and effect presets are accessible to the developer. Based on observed behavior, haptic interaction is driven by Unity colliders rather than visual mesh representations, meaning physical interaction occurs when the haptic stylus intersects with colliders defining the properties of virtual objects. Hapticlabs integrates into Unity via a dedicated package, through which haptic patterns defined in Hapticlabs Studio are triggered at runtime with configurable parameters for track selection, queuing, and amplitude. In the present implementation, the DevKit functions exclusively as an instrument on/off indicator in the third experimental scenario, emitting discrete vibrotactile signals to notify users of tool-state changes.
3.3.5 Assets and virtual environment
The VR operating room () (Figure 9) and anatomical assets are derived from openly available or institutionally hosted 3D models11,12,13, all accessible through project’s GitHub repository14. For texturing, the models are UV-unwrapped and processed using Adobe Substance 3D Painter (Adobe Inc., San Jose, CA, USA) before being exported and imported into Unity as FBX or OBJ files. To determine the correct scale, assets representing digital counterparts of physical objects are dynamically resized during MR play mode until they match the real console and environment. Once the scale of these reference objects is established, all remaining virtual assets are scaled relative to them, ensuring spatial coherence between physical and virtual elements, which is essential for accurate haptic feedback and consistent visuo-proprioceptive integration.
FIGURE 9
4 Experimental scenarios and clinical use cases
The HIT serves as a research tool for next-generation surgical interfaces, enabling research teams to develop and evaluate novel haptic and tactile interaction strategies. The system facilitates the rendering of material properties, such as the shape, texture, and mass of anatomical structures as well as immaterial information, including virtual barriers for tissue protection or guided trajectories for enhanced precision. These interaction concepts are demonstrated within three initial experimental scenarios which function as proof-of-concept scenarios for representative clinical use cases as defined by .
The first scenario involves discrimination between soft and rigid regions of virtual and physical liver models representing healthy and tumorous tissue. This simulates intraoperative palpation and explores how combined visual and haptic cues support safe margin definition and resection planning. The second scenario examines instrument navigation and tissue preparation near vulnerable structures, such as the ureter, where graded haptic resistance conveys safety boundaries and restricted movement within a virtual “guardian” zone to improve spatial awareness. The third scenario simulates fatty tissue dissection. It integrates dual haptic interfaces and vibrotactile sensors that signal instrument activation states. Here, the interaction between continuous tissue feedback and distributed vibration patterns was used to study how multimodal haptic information informs precise control during energy-based tissue dissection.
4.1 Experimental scenario 1: Tissue elasticity and palpation
This scenario focuses on the discrimination between compliant and rigid regions in virtual and physical liver models. By simulating intraoperative palpation of tumorous versus healthy parenchyma, we explore how synchronized visuo-haptic cues can facilitate the definition of oncological resection margins. Participants first palpate in MR mode using a physical liver phantom with an embedded tumor placed on the console platform, which they manually explore to familiarize themselves with its tactile properties and spatial orientation (Figure 10A). Pressing the dark grey button on the stylus transitions the system into VR mode, where a spatially registered digital replica of the console and liver is displayed, and participants probe the virtual liver using the Touch device styluses, receiving visual deformation feedback corresponding to contact (Figure 10B). After the physical platform is removed, haptic feedback for the virtual liver is activated via the light grey stylus button, and participants then discriminate between soft (healthy parenchyma) and rigid (tumor) regions under controlled visual conditions, approximating intraoperative palpation and enabling assessment of how visuo-haptic cues support safe margin definition and resection planning.
FIGURE 10
4.2 Experimental scenario 2: Virtual barriers for critical structure protection
The scenario addresses instrument navigation in the proximity of vulnerable anatomy, using the ureter as an exemplar risk region in pelvic surgery. We implemented graded haptic resistance acting as a “guardian zone” or virtual barrier to restrict movement and enhance the surgeon’s situational awareness near high-risk boundaries. Users interact entirely in VR, where a stylized anatomical model of a virtual patient is presented: non-interactive organs are rendered semi-transparent, the ureter appears as an opaque structure, and surrounding adipose tissue is visualized as a semi-transparent, noisy layer partially covering the ureter (Figure 11). Haptic feedback is activated via the light grey stylus button, after which participants delineate the ureter’s borders using force feedback cues to distinguish it from the surrounding tissue, relying primarily on haptic rather than visual information. Within the simulation, a virtual safety zone is defined around the ureter, analogous to a VR “guardian” boundary; as users advance the instrument toward this zone, increasing friction is rendered through the Touch device, creating a tangible barrier that discourages further penetration and functioning as a physical counterpart to conventional visual boundary warnings.
FIGURE 11
4.3 Experimental scenario 3: Multimodal feedback in electrosurgical dissection
The third scenario simulates the dissection of fatty tissue by combining continuous force feedback from the Touch devices with cutaneous vibrotactile feedback from a forearm-mounted actuator. This setup demonstrates how force feedback cues, together with discrete tactile feedback signaling instrument activation and use, inform precise control during the application of electrosurgical energy. Users start in VR mode, where they need to dissect fatty tissue, a frequent step in abdominal and pelvic procedures (Figure 12). Pressing and holding the dark grey stylus button activates a dissection mechanic, so that continuous stylus movements gradually remove the semi-transparent fat layer and reveal the underlying anatomical structures, simulating a focused dissection task. In addition to the dual Touch devices, participants wear a forearm-mounted tactile actuator linked to the instrument state via the Hapticlabs sensor-actuator system: when the virtual instrument is switched on or off, a brief vibrotactile cue is delivered to the arm, providing an informational alert about the current motor state (Figure 13). During active dissection, continuous force feedback from the Touch devices represents tissue separation, while the intermittent vibrotactile signals convey instrument activation, enabling study of how distributed haptic cues support control during energy-based tissue dissection.
FIGURE 12
FIGURE 13
5 Technical and methodological limitations
5.1 Hardware and SDK constraints
The current implementation depends on proprietary middleware provided by 3D Systems, which limits transparency and controllability at the driver level. Error handling and parameter tuning are restricted to SDK interfaces, complicating troubleshooting and limiting direct manipulation of control loops or force rendering algorithms. Interruptions in device communication are difficult to diagnose, as the SDK provides minimal logging and documentation. Future implementations may benefit from open-source haptic middleware, such as that offers greater transparency and flexibility.
Beyond these software constraints, the employed 3-DOF force-feedback architecture inherently limits torque rendering and full six-dimensional force reflection. While sufficient for many exploratory interaction paradigms, this restricts the emulation of complex instrument-tissue interactions involving torsion, distributed contact forces or rotational maneuvers that characterize realistic operative scenarios. These characteristics also influence which interaction types can be meaningfully explored on HIT. The relatively low maximum force output and lack of torque rendering limit the realism of high-resistance interactions, such as contact with bone or sharply defined hard-tissue margins, which in clinical systems can generate substantial reactive forces and moments at the instrument tip. The present implementation is therefore targeted primarily at softer tissue interactions, virtual safety zones and guidance fields, where modest, well-controlled forces suffice to convey the intended perceptual cues.
5.2 Rendering and haptic constraints
Haptic rendering in Unity is collider-based and sensitive to mesh complexity. High-polygon, non-convex anatomical models produce unstable collision detection and inconsistent force output. As a result, anatomical realism must be balanced against real-time computational constraints. Practically, this means that haptic objects require simplified colliders that may not faithfully represent the geometry of the underlying visual mesh.
A second constraint concerns temporal coherence. Minor desynchronization between frame-based visual rendering and the high-frequency haptic updates loop may occur under computational load. Although not prohibitive for exploratory studies, such timing offsets can weaken perceived visuo-haptic coupling during rapid movements. Finally, while the Touch devices can in principle render nonlinear force profiles, the soft-tissue models and rendering pipeline used here provide only simplified approximations of biological mechanics, constrained by the need to update forces in real time. As a result, simulated stiffness and damping should be interpreted as qualitative cues rather than realistic biomechanical models. This distinction is important: the HIT is not intended to replicate surgical realism at a biomechanical level. Instead, it is designed to explore basic perceptual haptic variables.
5.3 Conceptual limitations
As a technology and code paper, the present work does not include quantitative user studies, force validation measurements, or statistical performance analyses. It therefore makes no claim of empirical verification of task improvement, clinical outcome simulation or controlled comparison against established robotic systems. Instead, the contribution lies in providing a configurable and openly documented experimental infrastructure that enables future systematic exploration of haptic interaction strategies under controlled conditions. The three presented experimental scenarios function as proof-of-concept demonstrations of this infrastructure, illustrating the range of interaction paradigms the toolkit can support rather than constituting formal experiments. While the three experimental scenarios demonstrate the technical feasibility and configurability of the HIT, they do not yet constitute a formal study of the platform or of specific feedback strategies. A logical next step will be to conduct at least one experimental study on one of these scenarios, using HIT to document the complete workflow from implementation of a novel feedback strategy through experimental design, data collection, and analysis.
6 Discussion: Implications for haptic research in RAS
6.1 HIT as an experimental infrastructure
The HIT is conceived as a research tool rather than a simulation platform. Its central contribution is to enable controlled experimental manipulation of visuo-haptic variables without requiring access to, or modification of proprietary clinical robotic systems. By decoupling experimentation from commercial robotic platforms, the system reduces financial, regulatory, and technical barriers that often restrict haptic research to well-resourced engineering laboratories. The open-source release of both the software pipeline and the hardware components and fabrication resources is intended to lower the threshold for adoption across disciplines including design, ergonomics, psychology, science and technology studies and human-computer interaction. These disciplines have much to contribute to surgical haptic research yet have lacked accessible entry points to do so. HIT is a configurable testbed whose capabilities are illustrated through three technical examples. However, in its current state, it is a proof-of-concept, as it is not yet validated through the study of haptic feedback strategies. A first standardized experiment based on one of these scenarios is planned as the next step, using HIT to document the complete workflow from implementation to analysis.
6.2 Haptic design beyond mechanical fidelity
A recurring assumption in surgical haptics research is that higher biomechanical fidelity produces better outcomes. The HIT is built on a different premise: that the perceptual and cognitive dimensions of haptic feedback deserve investigation independently of physical realism. The three experimental scenarios deliberately foreground this distinction. Rather than asking whether simulated stiffness numerically matches ex vivo tissue data, they are structured to ask how stiffness differences are perceived and acted upon; how a haptic boundary modifies spatial decision-making near vulnerable anatomy; and how distributed vibrotactile cues interact with continuous force feedback to inform instrument control. Whether this perceptual reframing produces measurable differences in task performance, decision quality, or workload remains to be established through formal user studies. The toolkit’s contribution at this stage is to provide the experimental infrastructure through which such questions can be rigorously and reproducibly investigated. This is a capability that high-fidelity biomechanical simulators, precisely because of their fixed architectures, are not well suited to offer. At the same time, the present implementation imposes important constraints. Device workspace, update-rate limits and the need for simplified collision geometries restrict the complexity, duration and realism of tasks that can be implemented without degrading stability or visuo-haptic coherence. These factors must therefore be considered explicitly when designing and interpreting experiments on HIT.
6.3 Multisensory integration and embodiment
The MR approach of the HIT creates conditions that are particularly favorable for studying presence, embodiment, and sensorimotor integration. By aligning the physical console, the virtual environment, and the force feedback devices within a single coherent setup, the HIT produces a setting in which the user’s proprioceptive and postural experience (i.e., leaning forward, hand movement, applying force) merges with the virtual tasks. This ongoing sensorimotor correspondence reinforces a sense of embodied presence that goes beyond what either virtual or fully physical setups can achieve (). Whether this can apply to a surgical interaction context, and to what extent real–virtual registration quality and physical interaction modulate presence, embodiment, and haptic judgment, are open empirical questions. The MR configuration therefore does not merely add realism; it constitutes a methodologically distinct condition in which the boundaries between physical and virtual action are deliberately blurred, enabling the study of how such perceptual ambiguity shapes haptic judgment and task performance. At present, the system has only undergone qualitative assessment of spatial registration accuracy and visuo-haptic coherence, and we do not yet report formal engineering benchmarks. Future work will therefore include instrumented measurements of end-to-end latency from physical input to visual and haptic response, tests of haptic loop stability and refresh-rate behavior under high computational load, and quantitative evaluation of physical–virtual registration error after Kabsch-based alignment, to establish quantitative bounds within which experimental results obtained on HIT can be interpreted. A stepwise validation of the simulator can follow established approaches in surgical simulation, incorporating extensive face validity, content validity, and usability similar to the framework used by .
7 Conclusion and future work
The HIT illustrates how a configurable MR robotic console can make complex visuo-haptic interaction experiments both technically feasible and practically reproducible. By combining a custom physical console, dual force-feedback devices, and an open Unity-based software pipeline, the system supports precise real–virtual registration, structured calibration, and systematic manipulation of haptic conditions. The three proof-of-concept experimental scenarios (liver stiffness discrimination, ureter boundary protection, and fatty tissue dissection) illustrate how variations in stiffness rendering, safety-zone feedback, and distributed vibrotactile cues can be prototyped and studied without access to clinical robotic systems. The documented limitations in hardware fidelity, latency, and SDK transparency point to concrete engineering challenges that must be addressed to mature haptic feedback in surgical simulation; at the same time, these constraints clarify the toolkit’s intended scope: it is a platform for interaction design and perceptual research, not a biomechanical ground truth. We deliberately chose organ-specific scenarios rather than fully abstract tasks to preserve recognizable surgical structure and maintain face validity for clinicians. Given the limited biomechanical and visual realism of these simulations, studies conducted on HIT should therefore be interpreted as conceptual and comparative evaluations of haptic interaction strategies under controlled conditions, rather than as direct evidence about absolute performance or outcomes in real robotic surgery. A key next step is to extend the current implementation. The toolkit’s modular architecture and open asset library make it straightforward to extend the scenario set beyond the three visceral surgery use cases presented here, incorporating additional organ systems and task types including bowel, vascular, and urologic targets, as well as traction, weightlifting, and safe retraction, identified as priorities by robotic surgeons (). The MR workflow that links physical phantoms to spatially registered virtual counterparts is particularly well suited to testing new physical and virtual organ models and investigating how varying degrees of real–virtual correspondence affects presence and haptic judgment.
More broadly, the framework offers a foundation for systematic exploration of further multisensory integration in robotic surgery, where future work can combine force feedback and vibrotactile feedback with additional modalities such as spatialized audio communication or olfactory cues, to approximate the sensory contingencies on which surgeons rely, with the longer-term aim of transferring refined interaction strategies into next-generation clinical robotic systems.
Statements
Data availability statement
The data presented in the study are deposited in the Zenodo repository, DOI: 10.5281/zenodo.21240409. The corresponding source code and configuration files are available in the GitHub repository “Haptic Interaction Toolkit in Robotic-Assisted Surgery” (https://github.com/ExperimentalSurgery/Haptic-Interaction-Toolkit-in-Robotic-Assisted-Surgery).
Author contributions
ZA: Conceptualization, Investigation, Methodology, Project administration, Writing – original draft, Writing – review and editing, Supervision. AY: Data curation, Methodology, Software, Visualization, Writing – review and editing. CR: Data curation, Investigation, Methodology, Software, Writing – review and editing. JB: Conceptualization, Visualization, Writing – review and editing. JP: Supervision, Writing – review and editing, Resources. IS: Conceptualization, Funding acquisition, Investigation, Project administration, Supervision, Validation, Visualization, Writing – original draft, Writing – review and editing, Resources. MQ: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Validation, Visualization, Writing – original draft, Writing – review and editing, Resources.
Funding
The author(s) declared that financial support was received for this work and/or its publication. This work was supported by the Cluster of Excellence “Matters of Activity. Image Space Material” funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy–EXC 2025.
Acknowledgments
We would like to thank Michelle Mantel for the photos of the physical console (Figures 3, 6) and Dominic Eger Domingos for the custom design and 3D printing of the finger grips for the Touch devices.
Conflict of interest
The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declared that generative AI was used in the creation of this manuscript. The authors acknowledge that they occasionally used DeepL and Perplexity to “proofread” sentences in English.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Footnotes
1.^Haptic Interaction Toolkit repository: https://github.com/ExperimentalSurgery/Haptic-Interaction-Toolkit-in-Robotic-Assisted-Surgery
2.^BWG Cloud project files: https://bwgcloud.hu-berlin.de/d/4a507cf93cee46f996ea/
3.^Haptic Interaction Toolkit repository “Assets” folder: https://github.com/ExperimentalSurgery/Haptic-Interaction-Toolkit-in-Robotic-Assisted-Surgery/tree/main/Assets
4.^Haptic Interaction Toolkit repository: https://github.com/ExperimentalSurgery/Haptic-Interaction-Toolkit-in-Robotic-Assisted-Surgery
5.^3D Systems Touch device: https://www.3dsystems.com/haptics-devices/touch
6.^Haptics Direct for Unity: https://assetstore.unity.com/packages/tools/integration/haptics-direct-for-unity-v1-197034
7.^Meta XR Unity developer documentation: https://developers.meta.com/horizon/develop/unity
8.^Hapticlabs Unity integration docs: https://docs.hapticlabs.io/integrations/unity/
9.^Hapticlabs download page: https://www.hapticlabs.io/download
10.^MathUtilities GitHub and stable rotation paper: https://github.com/zalo/MathUtilities, https://animation.rwth-aachen.de/media/papers/2016-MIG-StableRotation.pdf
11.^Human base meshes (Blender demo files): https://www.blender.org/download/demo-files/
12.^Abdomen anatomy model (Sketchfab): https://sketchfab.com/3d-models/abdomen-anatomy-ed05d3b7b49b4014a09d7a9d62e4f421
13.^Liver model (BWG Cloud, Leber Neu): https://bwgcloud.hu-berlin.de/d/b499550961b94e62b692/
14.^Haptic Interaction Toolkit repository: https://github.com/ExperimentalSurgery/Haptic-Interaction-Toolkit-in-Robotic-Assisted-Surgery
References
1
AkbalZ.DaneshgarA.MorgülM. H.MoosburnerS.PratschkeJ.SauerI. M.et al (2025). Exploring the need for haptic feedback and improved communication in robotic-assisted surgery: a surgeon-centered survey. IEEE Access13, 197889–197898. 10.1109/access.2025.3633553
2
AkbalZ.SauerI.YadyginaA.RemdeC.BlumenthalJ.QueisnerM. (2026). “Haptic interaction toolkit for robotic-assisted surgery,” in International XR-Metaverse Conference 2025, Maastricht, Netherlands. Cham, (Forthcoming: Springer Nature).
3
AwadM. M.RaynorM. C.Padmanabhan-KabanaM.SchumacherL. Y.BlatnikJ. A. (2024). Evaluation of forces applied to tissues during robotic-assisted surgical tasks using a novel force feedback technology. Surg. Endosc.38, 6193–6202. 10.1007/s00464-024-11131-z
4
BergholzM.FerleM.WeberB. M. (2023). The benefits of haptic feedback in robot assisted surgery and their moderators: a meta-analysis. Sci. Rep.13, 19215. 10.1038/s41598-023-46641-8
5
CHAI3D (2016). The Open Source Haptic Framework. Version 3.3.0. CHAI3D.
6
ColanJ.DavilaA.HasegawaY. (2022). A review on tactile displays for conventional laparoscopic surgery. Surgeries3, 334–346. 10.3390/surgeries3040036
7
DaweS. R.WindsorJ. A.BroedersJ. A.CreganP. C.HewettP. J.MaddernG. J. (2014). A systematic review of surgical skills transfer after simulation-based training: laparoscopic cholecystectomy and endoscopy. Ann. Surg.259 (2), 236–248. 10.1097/SLA.0000000000000245
8
GamberiniG.MaglioS.MarianiA.MazzottaA. D.ForgioneA.MarescauxJ.et al (2024). Design and preliminary validation of a high-fidelity vascular simulator for robot-assisted manipulation. Sci. Rep.14 (1), 4779. 10.1038/s41598-024-55351-8
9
GamberiniG.MazzottaA. D.DuranteA.TognarelliS.PetruccianiN.MenniniG.et al (2026). Sensorized vascular high-fidelity physical simulator for robot-assisted surgery training: a multisite pilot evaluation. J. Clin. Med.15 (3), 1054. 10.3390/jcm15031054
10
GaniA.PickeringO.EllisC.SabriO.PucherP. (2022). Impact of haptic feedback on surgical training outcomes: a randomised controlled trial of haptic versus non-haptic immersive virtual reality training. Ann. Med. Surg. (Lond).83, 104734. 10.1016/j.amsu.2022.104734
11
GomezE. D.Mat HusinH.DumonK. R.WilliamsN. N.KuchenbeckerK. J. (2025). Simulation training with haptic feedback of instrument vibrations reduces resident workload during live robot-assisted sleeve gastrectomy. Surg. Endosc.39, 1523–1535. 10.1007/s00464-024-11459-6
12
HuangK.ChitrakarD.MitraR.SubediD.SuY. H. (2020). Characterizing limits of vision-based force feedback in simulated surgical tool-tissue interaction. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc.2020, 4903–4908. 10.1109/EMBC44109.2020.9176658
13
Laga Boul-AtarassI.RubioM. D. M.Padillo-EguíaA.Racero-MorenoJ.Eguía-SalinasI.Pereira-ArenasS.et al (2025). Role of haptic feedback technologies and novel engineering developments for surgical training and robot-assisted surgery. Front. Robot. AI12, 1567955. 10.3389/frobt.2025.1567955
14
LimZ. J.YapK. M. (2024). Haptic VR simulation for surgery procedures in medical training. Report No.: arXiv:2411.05148.Available online at: https://arxiv.org/abs/2411.05148 (Accessed November, 2024).
15
MarcusH. J.RamirezP. T.KhanD. Z.Layard HorsfallH.HanrahanJ. G.WilliamsS. C.et al (2024). The IDEAL framework for surgical robotics: development, comparative evaluation and long-term monitoring. Nat. Med.30, 61–75. 10.1038/s41591-023-02732-7
16
NeidhardtM.MielingR.BengsM.SchlaeferA. (2023). Optical force estimation for interactions between tool and soft tissues. Sci. Rep.13 (1), 506. 10.1038/s41598-022-27036-7
17
NeriA.CoduriM.PenzaV.SantangeloA.OliveriA.TurcoE.et al (2024). A novel affordable user interface for robotic surgery training: design, development and usability study. Front. Digit. Health2, 1428534. 10.3389/fdgth.2024.1428534
18
OkamuraA. M. (2009). Haptic feedback in robot-assisted minimally invasive surgery. Curr. Opin. Urol.19 (1), 102–107. 10.1097/MOU.0b013e32831a478c
19
OthmanW.LaiZ. A.AbrilC.Barajas-GamboaJ. S.CorcellesR.KrohM.et al (2022). Tactile sensing for minimally invasive surgery: conventional methods and potential emerging tactile technologies. Front. Robot. AI8, 705662. 10.3389/frobt.2021.705662
20
QueisnerM.PogorzhelskiyM.RemdeC.PratschkeJ.SauerI. M. (2022). VolumetricOR: a new approach to simulate surgical interventions in virtual reality for training and education. Surg. Innov.29 (3), 406–415. 10.1177/15533506211054240
21
SaracinoA.DeguetA.StaderiniF.BoushakiM. N.CianchiF.MenciassiA.et al (2019). Haptic feedback in the da Vinci Research Kit (dVRK): a user study based on grasping, palpation, and incision tasks. Int. J. Med. Robot.15 (4), e1999. 10.1002/rcs.1999
22
SlaterM.Sanchez-VivesM. V. (2016). Enhancing our lives with immersive virtual reality. Front. Robot. AI3, 74. 10.3389/frobt.2016.00074
23
VleugelsM. P. H.RahimiM. (2024). Feeling of pulsations in artificial arteries with a real time haptic feedback laparoscopic grasper: a validation study. Surg. Endosc.38 (8), 4222–4228. 10.1007/s00464-024-10877-w
24
YilmazN.BurkhartB.DeguetA.KazanzidesP.TumerdemU. (2024). Enhancing robotic telesurgery with sensorless haptic feedback. Int. J. Comput. Assist. Radiol. Surg.19 (6), 1147–1155. 10.1007/s11548-024-03117-y
Summary
Keywords
haptic interaction, haptics, mixed reality, robotic-assisted surgery, surgical simulation, virtual reality
Citation
Akbal Z, Yadygina A, Remde C, Blumenthal J, Pratschke J, Sauer IM and Queisner M (2026) Haptic Interaction Toolkit: a mixed reality-based robotic console for experimental investigation of haptic feedback in robotic-assisted surgery. Front. Virtual Real. 7:1830167. doi: 10.3389/frvir.2026.1830167
Received
13 March 2026
Revised
08 June 2026
Accepted
19 June 2026
Published
06 August 2026
Volume
7 - 2026
Edited by
Arnaud Leleve, Institut National des Sciences Appliquées de Lyon (INSA Lyon), France
Reviewed by
Alessandro D. Mazzotta, Oncologique et Métabolique de l'Institut Mutualiste Montsouris, Italy
Hamed Jamshidifar, Honeywell, Canada
Updates
Copyright
© 2026 Akbal, Yadygina, Remde, Blumenthal, Pratschke, Sauer and Queisner.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Zeynep Akbal, zeynep.akbal@charite.de
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.