With the accelerating population aging, the number of patients suffering from spinal degenerative diseases, such as lumbar disc herniation and lumbar spinal stenosis, is increasing annually. However, the anatomical structure of the spine is complex and involves critical components such as the spinal cord, nerve roots, and blood vessels. Surgery in this area is unsafe and requires extremely high precision. Rapid advances in surgical robotics can help spine surgeons improve the precision and stability of their surgeries. In recent years, surgical robots employed in spine surgery have been primarily used to assist in pedicle screw placement, such as the TiRobot (TINAVI Medical Technologies Co. Ltd., Beijing, China), Mazor (Mazor Robotics Ltd., Caesarea, Israel), Da Vinci (Intuitive Surgical, Sunnyvale, CA, USA), ROSA (Zimmer Biomet Robotics, Montpellier, France), Excelsius GPS (Globus Medical, Inc., Audubon, PA, USA), and Orthbot (Xin Junte, Shenzhen, China) (1–6). Robotic-assisted lumbar fusion is more precise in pedicle screw placement, has a shorter average operative time, less bleeding, faster postoperative recovery, and reduces radiation injuries to patients and medical staff than the freehand manipulation (7–11).

Currently, spinal surgical robots are mainly used for pedicle screw placement. By importing preoperative CT data and using registration technology to match the actual intraoperative position, surgeons perform the surgery under robotic guidance (3). However, surgical robots rely on unimodal CT images as baseline images, which can only display vertebral bone structure and are incapable of visualizing soft tissue structures such as intervertebral discs and nerves. This has led to a relatively limited scope of application of surgical robots and cannot assist surgeons in the establishment of surgical channels, nerve decompression, and other operations (12). With the continuous development of medical imaging and computer image processing technology, multimodal image fusion has shown significant advantages in the medical field. Multimodal fusion images are valuable for disease diagnosis and preoperative planning (13). In spine surgery, CT/MR image fusion is particularly practical because fused images can accurately provide information about the positions of both bones and soft tissues. However, no studies have reported on preoperative multimodal spine models that can be used for intraoperative registration and navigation of spine surgery robots.

With the rapid development of multimodal image fusion technology and in response to the current limitations in the application scope of spinal surgical robots, this study aims to develop a software system for surgical robots based on multimodal image fusion, which can provide key imaging data for surgical robots to perform operations such as surgical channel establishment and nerve decompression. For spine surgeons and patients, the use of accurate, intuitive, and anatomically rich spine multimodal fusion images to guide surgery can significantly improve surgical efficiency, shorten operation time, and reduce intraoperative complications. Patients will have better postoperative outcomes, reduced medical costs, and significantly improved quality of life.

The surgical robot software system based on multimodal image fusion consists of six major modules: a data reading and writing management platform, an image visualization platform, a vertebral bone segmentation algorithm platform for CT and MR images, an intervertebral disc segmentation algorithm platform for MR images, a nerve segmentation algorithm platform for MR images, and a CT/MR image registration fusion algorithm platform. The spinal model is constructed using the software system, as follows: input the preoperative spinal CT and MR images of the patient into the software system, and after sequential processing through the four modules of vertebral bone segmentation, intervertebral disc segmentation, nerve segmentation, and CT/MR image registration fusion, construct a spinal model containing independent structures such as vertebrae, intervertebral discs, and nerves. Figure 1 illustrates the development and operation process of the system.

This study aims to establish and optimize image segmentation and registration fusion algorithms and subsequently develop an integrated software system that includes vertebral segmentation algorithms for CT and MR images, intervertebral disc segmentation algorithms for MR images, nerve segmentation algorithms for MR images, and CT/MR image registration fusion algorithms. This integrated software system will be used to construct a multimodal spine model containing independent modules of vertebral bones, intervertebral discs, and nerves and then conduct clinical application research. Developing a software system for surgical robots based on multimodal image fusion is important. The clinical application of this software system will provide key imaging data for surgical robots to perform nerve decompression and other operations, which will further improve surgical accuracy and reduce surgical complications. Our group has made some research progress in the preoperative planning system, navigation system, and vertebral shaping system, which is a critical step toward the successful application of this software system in spinal surgery.

Pedicle screw placement is a critical step in spinal internal fixation surgery because misplaced screws may lead to neurological deficits or vascular injury (17, 18). The reported rates of screw misplacement vary significantly across different studies. Castro et al. (19) reported a misplacement rate as high as 40%. The complication rate due to misplaced screws ranges from 0% to 54% (20–23). Many studies have confirmed that robot-assisted pedicle screw placement has significant advantages over traditional fluoroscopic techniques, with an accuracy rate ranging from 93% to 100% (10, 11, 24–26). Kantelhardt et al. (27) first reported an accurate placement rate of 94.5% for robot-guided pedicle screw placement compared with a 91.4% accuracy rate for traditional screw placement. Several subsequent studies have also demonstrated that robot-assisted screw placement is superior in accuracy to traditional methods and results in less tissue damage during surgery, thus providing a better prognosis for patients (28–32). Moreover, undoubtedly, the future of spinal robotics should not be limited to assisting surgeons in pedicle screw placement alone. They should also assist in performing different surgeries for various spinal disorders. Wang et al. (33) and Lin et al. (34) reported that robot-assisted percutaneous kyphoplasty has advantages such as higher puncture accuracy, shorter channel establishment time, reduced radiation exposure, and lower bone cement leakage. Currently, percutaneous endoscopic discectomy (PTED) is a commonly used minimally invasive surgery for the treatment of lumbar disc herniation, and the difficulty of this surgery lies in the establishment of the surgical channel, which requires multiple fluoroscopy and punctures to reach the target position (35). Yang et al. (36) found that robot-assisted guided PTED, compared with conventional c-arm fluoroscopy-guided PTED, had fewer punctures (1.20 ± 0.42 vs. 4.84 ± 1.94), fewer fluoroscopy (10.49 ± 2.16 vs. 17.41 ± 3.23), and shorter surgery time (60.69 ± 5.63 vs. 71.19 ± 5.11 min). Importantly, our surgical robot software system, through multimodal image fusion technology, achieves three-dimensional visualization of the surgical area's anatomical structures, providing better guidance for surgery and maximizing the safety of the surgery.

Currently, the localization and navigation functions of spinal surgery robots are mainly based on CT images. CT images provide excellent visualization of bone structures but post challenges in distinguishing soft tissues such as nerves and ligaments. In contrast, MR images offer good visualization of soft tissues. Owing to the lack of local anatomical information, the application scope of robots in spinal surgery is extremely limited (12). However, some researchers have attempted to address this issue using CT/MRI fusion images. Their research results indicate that fusion images can better present the relationship between the bony and soft tissue components of lesions (37–39). This can serve as a valuable tool to enhance the accuracy of surgical planning.

The construction and application of spine models based on CT/MR image fusion technology involves several key technologies, such as vertebral bone segmentation, intervertebral disc segmentation, nerve segmentation, and multimodal image registration and fusion. ① Vertebral bone segmentation. Previous studies using traditional CT vertebral segmentation techniques have achieved precise segmentation of simple vertebral regions. However, challenges remain in segmenting complex areas such as tightly connected facet joints. Although some progress has been made in deep learning-based vertebral segmentation technology, it also suffers from the problems of long consumption time and poor accuracy (40–42). To address this issue, our research team designs an interactive dual-output vertebral instance segmentation algorithm. The results of algorithm training and optimization show relatively high overall and local segmentation accuracy. Compared with existing methods, the training speed of the algorithm is improved by 2.7 times, and the segmentation speed is improved by 4 times (14). ② Intervertebral disc segmentation. Current research on intervertebral disc segmentation technology requires further investigation. Recently, a research team developed a fully automated intervertebral disc recognition and segmentation algorithm based on iterative neural networks (43–46). However, local accuracy within the surgical area, which is crucial for spinal robot surgeries, remains unclear. To address this problem, our group designed an active interactive multimodal intervertebral disc fine-tune algorithm, which achieved an I9-Dice coefficient of 92.50% ± 1.82% for high-resolution intervertebral disc data in the pre-experiment. ③ Nerve segmentation. In addition, fewer studies have focused on nerve segmentation. Existing segmentation algorithms struggle to achieve complete and high-precision segmentation of neural tissues in routine clinical images. Although intraoperative neural automatic extraction algorithms have been designed, they are not suitable for preoperative planning (47). This study applies a super-resolution semi-supervised segmentation algorithm to nerve segmentation to achieve more precise nerve segmentation. ④ Multimodal image registration and fusion. Many researchers have explored the possibility of spinal CT/MR image registration fusion to integrate the anatomical information of bone and soft tissues. However, the results from these studies have not been ideal. Challenges include poor accuracy, reliance on manual vertebral joint matching, and overall long registration times (48, 49). Some scholars have applied the MIND Demons algorithm to deformably register soft tissues from MRI and fuse them into CT images; however, its similarity measure is unstable in multimodality at the time of registration (50). Our group designs a self-correcting S-PAC registration algorithm with vertebrae prior, which may help solve the current CT/MR image registration algorithm, in which combining accuracy, stability, reliability, and practicality is difficult. In this study, we used sheep spine specimen to validate and optimize the registration fusion algorithm for CT and MR images. Although the sheep spine specimen is a little different from the human body structure, it can meet our current study requirements. Certainly, we will further optimize and validate the algorithm on human cadavers in the future when conditions permit.

The current study protocol aims to preliminarily develop a software system based on multimodal image fusion, but its practical application to spinal surgery robotics still needs to overcome many challenges. Firstly, we only included patients who met the diagnosis of lumbar disc herniation or spinal stenosis, which was aimed at guaranteeing the validity and stability of the algorithm lightweight. Thereafter, we will further incorporate various types of patients such as spinal deformity, history of spinal surgery, spinal fracture, spinal infection, and spinal tumour, to continuously improve the clinical applicability of the system. Furthermore, contemporary deep neural network models commonly employed are characterized by large size and parameter count, requiring long computing time and high GPU computing power, which limits the practical application. In recent years, lightweight of algorithms has become an important optimization direction. He et al. (51) constructed a lightweight algorithm by reducing the parameters related to the algorithm, which has the advantages of fewer parameters, smaller size and faster training speed. In the future, in addition to upgrading our equipment and network on time, we plan to use lightweight algorithms to make our system faster and more clinically applicable. Finally, it's noteworthy that currently, no automated or semi-automated spinal surgery robot capable of soft tissue manipulation has been developed. Consequently, in the short term, the system will primarily provide navigation functions for the surgeon's manual operation. We believe that this system will play a greater role in the future with the further development of robotics.

Until now, spinal surgical robots have commonly utilized single-modal spinal CT images as intraoperative baseline images. No studies have reported on multimodal, multi-independent module spine models that can be employed for intraoperative registration and navigation of spinal surgery robots. This study faces numerous challenges that must be overcome, with many issues urgently requiring resolution. Nevertheless, this study has vast prospects for application. The application of the outcomes of this study to spinal surgical robots can significantly expand the scope of their use. It can facilitate robot-assisted operations in spinal surgery, such as the establishment of surgical channels and neural decompression. This would notably enhance the precision, effectiveness, and minimally invasive nature of intraoperative surgeries. This makes spine surgery robots more clinically applicable, ultimately benefiting a larger population of patients.