Introduction

In orthopedic oncology, achieving negative margins during tumor resections is associated with better oncologic outcomes [1, 2]. Positive resection margins are associated with increased local recurrence and compromised limb function. Clear spatial awareness of the pathoanatomy is crucial for achieving negative-margin resections in patients with complex anatomy and varying tumor extents. Assistive tools such as computer navigation or 3D-printed resection guides may replicate planned bone resections with an average error of 2 mm [3–6], but not at the soft-tissue margin [7, 8]. The overall oncologic outcome also depends on the soft-tissue margin, which can be managed only with conventional surgical techniques, as assistive tools are unavailable.

Conventionally, surgeons dissect around the tumor, leaving a cuff of normal tissue, separating vital neurovascular structures, or exposing the bone near planned osteotomies, where assistive tools (computer navigation or 3D-printed guides) help replicate bone resections. Before making a skin incision, surgeons plan the procedure using visual assessments of preoperative 2D images or 3D models reconstructed from them, displayed on a PC, and mentally translate these plans and 3D decisions onto the patient. While 3D-printed models may partly improve spatial awareness of tumors as static visualization aids, their usefulness is limited by the number of segmented volumes that can be physically printed, and the assessment remains disconnected from the actual patient. This process presents significant cognitive challenges, requiring constant mental reconstruction, especially when tumors vary in size and location, involve complex anatomy, or are close to vital structures.

Mixed Reality (MR) is an immersive technology that combines spatial computing to display both real and virtual environments, allowing users to interact with them in real time [9, 10]. In MR, virtual objects appear as 3D holograms that preserve the depth and parallax perception of physical objects. Overlaying digital 2D or 3D holograms on the patient’s body may help surgeons visualize the procedure more intuitively, providing a clear 3D spatial awareness of the anatomy, tumor boundaries, and nearby vital structures. This enhances surgical judgment and decision-making before the operation [10]. A systematic review of MR in orthopedics indicates that MR technology could improve surgical outcomes by reducing postoperative complications, increasing spatial awareness and precision, decreasing intraoperative time, and minimizing invasiveness [11].

To date, only two brief case series have been reported on MR in orthopedic oncology [12, 13]. One study demonstrated that MR technology enhanced 3D spatial understanding of 9 bone tumors and aided surgical planning prior to incision [12]. The other study, involving MR use in four pediatric bone tumors before and during surgery, found MR to be a useful adjunct for preoperative planning and surgical routine without increasing operative time [13]. Incorporating new technology into clinical workflows presents practical challenges, including technical problems, longer surgical durations, and a learning curve that may temporarily hinder clinical utility [14].

This study builds on the previous proof-of-concept publication [12] by including more bone tumor cases and other orthopedic tumor cases. We assessed 1) the surgeon’s spatial awareness of pathoanatomy and the cognitive load from using MR technology for preoperative assessment of primary and revision cases in orthopedic oncology; 2) the quality of surgical margins in sarcoma cases; 3) technical issues encountered during the integration of MR into clinical workflows; and 4) the average time for hologram-to-patient registration, as well as improvements observed over the implementation period.

Materials and methods

Between July 2021 and November 2025, we retrospectively reviewed 72 patients with bone or soft tissue tumors or failed tumor reconstructive surgeries (Table 1). There were 44 males and 28 females, with a mean age of 50 (range 14–91). The diagnoses included 19 (26.4%) bone sarcomas, 23 (31.9%) soft tissue sarcomas, 20 (27.8%) benign bone tumors, and 7 (9.7%) revision prostheses following previous tumor reconstructive surgeries. Tumor locations were: 37 (51.4%) in the lower limbs, 15 (20.8%) in the pelvis, 6 (8.3%) in the foot and ankle, 5 (6.9%) in the upper limbs, 4 (5.6%) in the chest wall, and 5 (6.9%) in other locations. Seven of 19 bone sarcoma patients (36.8%) who underwent bone resections required assistance with 3D-printed resection guides, and four patients (21.1%) required computer navigation assistance. One patient (5.3%) needed both tools due to the complexity of multiplanar osteotomies and nearby pelvic soft tissue constraints. The remaining 7 patients (36.8%) did not require assistive tools for less complex resections. Among the 19 bone sarcoma patients undergoing reconstruction, three (15.8%) received off-the-shelf modular tumor prostheses, five (26.3%) received 3D-printed patient-specific prostheses, two (10.5%) received iliac crest grafts, two (10.5%) had cementation, and no reconstruction was performed in seven (36.8%) patients. Surgeons used MR technology for preoperative clinical assessment before skin incision. MR clinical assessment was considered necessary due to anticipated difficulties in spatial awareness of patients’ pathoanatomy and nearby vital neurovascular structures; determining tumor boundaries such as intraosseous involvement, infiltrating soft-tissue peripheral borders, recurrent tumors, or effects of prior neoadjuvant radiotherapy; locating sites for planned osteotomies; or placing patient-specific guides and prostheses.

Preparation of MR holographic application

Before December 2024, the study followed the clinical workflow described in a previous publication [12]. This proprietary workflow was further refined to more effectively integrate MR technology into orthopedic oncology surgery (Figure 1). Since December 2024, the MR platform (Syngular, version 2, Syngular Technology Limited, Hong Kong SAR, China) has been in use. Unlike the earlier version (Syngular, version 1), which required an engineer to generate MR holograms, the updated platform enables surgeons to manually import medical images and CAD models from medical imaging and reconstruction software such as 3D Slicer (version 5.8, http://www.slicer.org) or MIMICS (version 16.0, Materialise NV, Leuven, Belgium) and automatically generate MR holographic contents for each patient. Surgeons view different 3D holograms through an MR headset (HoloLens 2, Microsoft Corporation, Redmond, WA, US) equipped with an improved interface that uses a more intuitive, quicker thumb-abduction gesture instead of pressing or gazing at virtual buttons as in the previous MR platform, version 1 (Figure 2). Patient-specific 3D holograms of muscle, bone, tumors, neurovascular structures, and 3D-printed guides or implants could then be spatially visualized and sequentially assessed on actual patients during preoperative surgical planning. A total of 36 cases were performed using the previous MR platform (version 1) before December 2024, and another 36 cases were performed using the newer platform (version 2) afterward.

Display full size

Figure 1. The final clinical workflow for implementing Mixed Reality (MR) in orthopedic oncology surgery, as of December 2024, shows the technological evolution of MR throughout the study period. *: Holographic content involves importing 3D models and textures into Unity, setting up model display, and creating a patient’s digital asset bundle.

Display full size

Figure 2. Holograms control with finger gestures or gaze. (A) shows the thumb abduction gesture (yellow arrow) used to control different 3D holographic scenes in a patient with leg soft tissue sarcoma. (B) displays gaze control of the scene button (yellow arrow), which activates the hologram of a bone tumor (red arrow) in a patient with a benign talar tumor.

CT images were acquired during diagnostic bone tumor biopsies, CT angiograms to assess the tumor’s proximity to blood vessels, or PET-CT scans for systemic soft tissue sarcoma staging. MRI scans helped define tumor boundaries and nearby neurovascular structures. Because the images were acquired in different body positions, CT-MRI fusion was performed manually in 3D Slicer (version 5.8; http://www.slicer.org). MRI images were transformed to match the CT images using the respective bone contours (Figure 3), and the transformation matrix of the fused images was recorded for subsequent MR case preparation on the MR platform. The tumor (low-grade or high-grade components) and other critical structures, such as nearby vessels and nerves, were mapped on the fused images and then exported as stereolithography (STL) files to MIMICS for planning bone resection in bone sarcoma cases (Figure 4). Surgeons defined bone resection planes and annotated surgical requirements, then exported the planning project file to the engineer for designing and manufacturing patient-specific implants to reconstruct bone defects after planned bone tumor resections. 3D-printed resection or drill guides were also manufactured to replicate the planned bone resection and ensure accurate implant placement. The DICOM images from CT/MRI, the transformation matrix for image fusion, and CAD models (including tumor, neurovascular structures, bone resection planes, and patient-specific guides or implants) were imported into the MR platform. This platform generated cinematic-rendered models and displayed them within the Unity engine (Unity Technologies, Unity Software Inc., San Francisco). The patient’s holographic data was assembled into a digital package (patient’s asset bundle) and loaded into the MR headset. After June 2023, surgeons accessed MR technology by wirelessly downloading the data from the hospital’s PC server to the headset over a secure hospital Wi-Fi network. The hospital network was designed to protect patient data privacy and cybersecurity during MR clinical deployment (Figure 5). Behind the hospital firewall, the Mobile Network Interface (MNI) manages secure, controlled, and efficient connections for mobile devices and internal PCs, ensuring data protection and streamlined access. The surgeon’s view from the MR headset and the audio signal could also be streamed to a mobile device, enabling surgeons online to join case discussions or offer assistance.

Display full size

Figure 3. In a patient with high-grade soft tissue sarcoma of the thigh, multimodal image fusion (B–D) was performed, and 3D models (A) were created after mapping the tumor, vessels, and nerve onto the fused images in 3D Slicer. For CT-MRI fusion, the MRI images were matched with the CT images using the positional controller (yellow arrows).

Display full size

Figure 4. Virtual surgical planning in pelvic sarcoma. (A) In the patient with left pelvic chondrosarcoma (red arrows), the surgeons planned tumor resection with multiplanar osteotomies (B) using MIMICS software. Engineers then created the 3D-printed resection and drill guides (C, D) to replicate the surgical plan, and produced 3D-printed custom pelvic implants (E, F) for bone reconstruction to restore limb function.

Display full size

Figure 5. The hospital IT infrastructure was established to deploy Mixed Reality (MR) to support patient care at the point of care. Surgeons accessed the PC server and downloaded the patient’s digital asset bundle to the MR headset (Data¹). The surgeon’s view from the MR headset and the audio signal (Data²) can be streamed to a mobile device for online discussions.

Surgical planning and preoperative clinical examination

The surgeon (KCW) performed surgical planning by palpating and evaluating each patient using the conventional 2D method first, and then applied the MR 3D hologram method (Figure 6, Movie S1), as described in the previous study [12]. In the 2D method, the surgeon mentally superimposed 2D images and various 3D models printed on paper onto the patient’s body. For the MR hologram method, the surgeon overlaid 3D holograms by manipulating them with finger gestures and manually aligning them with the patient’s anatomy. After April 2025, the upgraded MR platform enabled more precise hologram spatial positioning through incremental buttons, allowing fine translational and rotational adjustments (Figure 7, Movie S2). Since the accuracy of manual spatial registration could not be reliably measured due to differences in soft-tissue deformation between the patients’ operative and image-scanning positions, the registration was considered acceptable when the holograms aligned best with palpable nearby bony landmarks or with the tumor’s geometry beneath the skin. Special information of 3D tumor, such as nonpalpable fascial tumor extension in infiltrating myxofibrosarcoma and peritumoral edema in high-grade soft tissue sarcoma, along with 2D MRI slices, was also evaluated in the patients (Figure 8). This helped determine the correct circumferential tumor margin during preoperative surgical planning. By sequentially viewing the various superimposed 3D holograms on the patients’ bodies, the surgeon further identified the surgical incision sites, soft-tissue exposures, tumors with nearby neurovascular structures, planned osteotomies, and patient-specific guides or implants during the assessments (Figure 9, Movie S3). The intraoperative soft tissue and tumor exposure was performed using conventional surgical dissection techniques, and MR guidance was not used because the soft tissue deformed after surgical exposure, and the MR platform used in the study could not support real-time intraoperative tracking of pathoanatomy.

Display full size

Figure 6. An adolescent patient, with a prior resection of proximal tibial osteosarcoma and prosthetic reconstruction, was assessed preoperatively for resection of local tumor recurrence using the conventional 2D method (A) first and then the Mixed Reality 3D method (B). (Movie S1).

Display full size

Figure 7. Mixed Reality interface with the position tuner for precise manual adjustment of the spatial positions of 3D holograms during hologram-to-patient registration in a patient with back soft-tissue sarcoma (A) and another patient with femoral osteosarcoma (B). (Movie S2).

Display full size

Figure 8. Tumor information, including nonpalpable fascial tumor extension (A–C, yellow arrow), peritumoral edema (D–G, red arrow), and 2D axial contrast-enhanced MRI images (H), was evaluated in the patient with thigh high-grade soft-tissue sarcoma.

Display full size

Figure 9. Various 3D holographic models, including tumors (red arrows), vessels, planned bone resection planes, and 3D-printed guides/implants, were overlaid on patients’ bodies to enhance spatial awareness of pathoanatomy and assist in surgical planning for patients with psoas nerve tumor (A), posterior tibia exostosis (B), wrist synovial sarcoma (C), thigh soft tissue sarcoma (D–F), and pelvic sarcoma (G–I). (Movie S3).

Results

The qualitative survey using the LS questionnaire showed that the MR 3D hologram group outperformed the 2D group across all aspects of spatial awareness related to the patient’s pathoanatomy and was considered a more effective tool for preoperative planning (Figure 10A, Table 2). Regarding NASA-TLX scores, the overall cognitive workload during preoperative assessment was lower in the MR 3D hologram group (median 170 [IQR 30]) compared to the 2D group (median 310 [IQR 40]). In the 2D group, ‘Mental Demand,’ ‘Effort,’ and “Frustration” were the main contributors to cognitive load. Conversely, in the early version 1 of the MR platform, ‘Physical Demand’ was the main factor in the MR 3D hologram group, but it decreased in the later, improved version 2 as familiarity with MR use increased (Figure 10B, Table 2). Surgeons reported no discomfort, such as motion sickness, headaches, fatigue, or eye strain, when using MR headsets.

Display full size

Figure 10. Comparison between the conventional 2D and Mixed Reality 3D hologram groups. (A) Qualitative results from a Likert Scale (LS) questionnaire assessing spatial awareness of pathoanatomy and surgical planning effectiveness during the initial study period (July 2021–November 2024) with the Mixed Reality platform (version 1), the later study period (December 2024–November 2025) with version 2, and the overall study duration. (B) The NASA-TLX score indicates the surgeon’s subjective cognitive workload when using Mixed Reality technology during different phases of the study.

For the histological analysis of the resected sarcoma specimens, 27 (64.3%) patients had R0 resection margins, 10 (23.8%) had R1 resection margins at the soft tissue, and 5 (11.9%) had R1 resection margins at the bone (Table 1). No R2 resections were observed.

Since December 2024, generating cinematic-rendered models with the upgraded MR software platform (version 2) has averaged 61 minutes (range: 49–156 minutes). Engineer intervention was needed in 4 of 36 cases (11.1%), primarily due to coordinate mismatches between imported CAD models and bone models or network issues. All patients’ MR digital asset bundle data were accessible and could be downloaded from the hospital’s PC server to the MR headsets in the operating room via the hospital Wi-Fi network. The MR clinical assessments were completed in all cases. During the initial study, the MR headsets required rebooting in 3 cases after more than 30 minutes of use due to overheating. This issue was resolved with software updates, and headsets were connected to an external battery. Spatial registration between holograms and patients was successful in all cases, with no significant deviations. The average time for hologram-to-patient registration in the last 26 cases was 2.3 minutes, ranging from 0.95 to 5.17 minutes. Linear regression showed a slight decrease in registration time as the number of cases increased, but the correlation was minimal (slope of –0.02; 95% CI, −0.07 to 0.03; R² = 0.04; p = 0.35) (Figure 11).

Display full size

Figure 11. An average registration time of 2.3 minutes (range: 0.95–5.17) for hologram-to-patient registration using the upgraded version 2 of the MR platform. The registration time slightly decreases as the case number increases, but the correlation is not statistically significant.

Discussion

Compared with conventional preoperative assessment based on 2D imaging, MR offers significant advantages by enabling immersive 3D visualization with preserved depth and parallax, thereby enhancing surgeons’ spatial understanding of tumor boundaries and adjacent vital structures directly on the patients’ bodies. Conventional 2D-based workflows require extensive mental reconstruction and translation of complex anatomy, imposing a significant cognitive burden, particularly in cases involving anatomical variability, complex tumor morphology, or proximity to neurovascular structures. In contrast, MR allows patient‑specific virtual 2D and 3D holograms to be directly overlaid onto the surgical field, supporting intuitive, real-time interaction during preoperative planning. This integrated visualization paradigm may facilitate more confident surgical decision‑making while reducing reliance on mental reconstruction. While tools such as computer navigation and 3D-printed guides assist with precise bone cuts, they are only effective after the surgical site is exposed. Tasks such as soft-tissue dissection, tumor mobilization, and separation from neurovascular structures still rely on conventional surgical skills and surgeons’ 3D understanding of anatomy. Building on a previous proof-of-concept study using MR technology [12], additional cases with diverse diagnoses were examined to evaluate the surgeons’ 3D spatial awareness of patients’ anatomy and pathology, as well as the practical challenges of integrating MR into the surgical workflow.

The study has several limitations. 1) Although it included more patients, the diverse diagnoses made meaningful statistical comparisons among clinical subgroups difficult. The primary aim was to explore potential clinical uses of MR technology. 2) Only one surgeon with over 15 years of experience in orthopedic oncology participated in the assessment. Studying multiple surgeons with varying levels of surgical or MR experience could reduce bias. 3) The manual registration error for the hologram-to-patient matching was not measured because it could not be reliably quantified due to differences in soft-tissue deformation between the patients’ operative and image-scanning positions. Although manual registration may not be 100% accurate, we believe that improved spatial understanding of tumors and nearby vital structures in complex anatomy before skin incision will aid surgical judgment and decision-making in sarcoma surgery. 4) Clinical outcomes were not evaluated because patients with different diagnoses underwent various surgical procedures. Factors such as tumor grade, response to chemotherapy or radiotherapy, and surgical planning tools also influence oncologic outcomes alongside MR technology. The enhanced 3D spatial awareness of anatomy may not directly improve surgical success. 5) The evaluation relied solely on subjective, qualitative measures (the LS questionnaire and NASA-TLX), as objective metrics for MR assessment are unavailable. These methods provide a quick and simple way to evaluate MR’s effectiveness and the surgeon’s cognitive effort during planning. 6) Since the MR platform was developed from scratch and periodically updated, this may have affected the qualitative results related to MR use. Therefore, the assessment of surgical effectiveness and cognitive load was divided between the early (V1) and later (V2) versions, with MR implementation only evaluated on the more stable version (V2). 7) The results and strategies for implementing MR technology may not be directly reproducible at other institutions due to differences in clinical and technological infrastructure. However, the study may serve as a valuable reference for establishing and integrating MR services into other clinical workflows.

This study is currently the largest to involve patients with various orthopedic oncology conditions, utilizing MR technology to assess surgeons’ 3D spatial understanding of tumors and related structures within patients’ anatomies during preoperative evaluations. The LS questionnaire results aligned with those of other studies [12, 13], confirming that overlaying patient-specific 3D holograms enhanced tumor visualization and spatial awareness, thereby reducing cognitive load during surgical planning. In addition to bones, this was the first study to employ the full range of 3D holograms, including skin, muscle, neurovascular structures, tumors, and CAD models such as planned bone resection planes and patient-specific guides or implants, superimposed on patients. This enabled surgeons to “see through” the skin and better understand the true spatial relationships of anatomy and pathology before skin incision. Our findings suggest that MR technology could be useful for soft-tissue sarcomas, benign bone tumors, and complex revision surgeries. Unlike conventional 2D mental reconstructions and continuous translation during preoperative assessments, overlaying 3D MR holograms onto patients offers potential benefits such as optimizing skin incisions, mapping nonpalpable fascial tumor extensions in infiltrative soft-tissue sarcoma subtypes (e.g., myxofibrosarcoma and undifferentiated pleomorphic sarcoma) [20–22], determining suitable cortical bone windows in benign bone tumors for intralesional curettage [23, 24], or locating old implant sites in revision surgeries.

R0 resection offers better local disease control than R1 resection; however, it does not improve distant metastasis-free survival or overall survival [25, 26]. In this study, 5 of 22 (22.7%) patients with high-grade soft-tissue sarcoma had R1 margins, a rate comparable to that reported in the literature, which ranges from 11.7% to 23% [26–28]. Patients received neoadjuvant radiotherapy before definitive tumor resection to preserve uninvolved neurovascular structures or bones, aiming to maintain limb function. As a result, close resection margins were anticipated. R0 resection was achieved in soft-tissue sarcoma subtypes with infiltrative growth patterns (1 myxofibrosarcoma, 2 undifferentiated pleomorphic sarcoma), which are known to have a high risk of incomplete resection and local recurrence [20–22]. Since no assistive tools currently exist to guide soft-tissue sarcoma resection, our findings suggest that MR technology may support surgically challenging resections in infiltrative subtypes but may not significantly lower R1 resection rates, given tumor proximity to preserved neurovascular structures. Further studies with larger cohorts or focusing on specific sarcoma subtypes are needed to evaluate the potential clinical benefits. Future research might also explore whether it improves surgical efficiency by reducing operative time or enhances patient safety by minimizing blood loss.

Since December 2024, no significant technical issues have been reported during the implementation of MR technology on the upgraded platform. In the last year of the 4.5-year study, 35 out of 72 patients (48.6%) were enrolled. The higher adoption rate suggests that the enhanced MR system—featuring planning automation, network case access in the operating room, and an intuitive user interface—facilitated integration into clinical workflow. Like any new technology, there is a learning curve, and surgical times decreased as surgeons gained experience with navigated procedures [29, 30]. For MR-assisted preoperative assessment prior to skin incision, the current setup was simple and minimally disrupted the natural surgical workflow in the operating room. Hologram-to-patient registration averaged 2.3 minutes and exhibited a minimal learning curve. This contrasts sharply with computer navigation, which requires extensive facilities, a system operator, and dedicated bone-to-patient registration steps after the bone tumor is exposed.

The MR technology described in this study may improve visualization and surgical judgment before skin incision by directly overlaying digital content onto the surgeon’s field of view. Future development should also extend to intraoperative use, including real-time MR tracking of bone tumor location [31], resection planes, and 3D printed guides and implants. The MR user interface may be enhanced by integrating AI and Large Language Models (LLMs) to enable context-aware voice-controlled assistants, alongside traditional input modalities such as hand gestures or eye gaze [32]. The Microsoft HoloLens 2 has limited processing capacity, with 4 GB of Dynamic Random Access Memory (DRAM). More powerful MR headsets are required to support the continued development of MR with additional and faster features. Exploring how human factors affect the integration of MR technology into clinical practice, as well as the technology itself, would constitute another research area [33]. Hospital network infrastructure and security should be designed to support emerging MR technology, protecting patient data privacy and ensuring compliance with the Health Insurance Portability and Accountability Act (HIPAA) regulations [34].

This study demonstrates that, compared with conventional 2D assessment methods, MR 3D holograms generated through the proposed clinical workflow can be directly visualized and overlaid on patients during preoperative surgical planning. The results indicate that MR technology could enhance surgeons’ 3D spatial awareness in various orthopedic tumor surgeries and reduce cognitive load. It may be beneficial for guiding soft-tissue exposure and determining soft-tissue margins in sarcoma resections. No significant technical issues were encountered that would hinder MR use throughout the study period. Further research is necessary to evaluate clinical outcomes, intraoperative applications, and to compare MR with existing assistive tools. The ongoing development of software and hardware should address clinical needs. Establishing network infrastructure and regulatory frameworks is crucial for efficient deployment and wider clinical adoption of the new MR technology.