Image-guided surgical systems are increasingly becoming established tools for visual aid in several interventional procedures. In this paper, we introduce a prototypic add-on system for enhancing the intraoperative visualization within a navigated spine surgery utilizing an extended reality approach. In essence, operation-speci c important anatomical structures are segmented from preoperative patient data and superimposed on the video stream of the operation eld. In addition, slices of the anatomy data, as well as shape and depth information of targeted structures, like spinal nerves or herniated discs, can be blended, which allows for a better protection of risk anatomy and accurate identication of the structures under consideration, and thus raises the safety and accuracy factors of the intervention.
One of the challenging tasks in a surgeon’s life is to transfer the information
displayed in 2D diagnostic images to the 3D situation in the real world of the
patient’s anatomy. This challenge is mastered in a continuous learning process,
still with significant obstacles. In this regard, medical navigation systems help
physicians to establish correspondences between locations in an acquired
patient dataset and the patient’s physical body during navigated surgeries. This
is highly advantageous for surgeries in regions with high density of critical and
vital structures; like the brain, skull base, and spine. However, this requires the
surgeon to switch between the operation field, i.e. microscope or endoscope view,
and wall-mounted or computer displays. Introducing augmented reality
facilitates the transfer of the diagnostic imaging to the individual patient anatomy
in a straightforward fashion. In this context, some research works provided
enhanced endoscopic views that are paired with synthesized virtual renderings
generated from the same view, e.g. [
In this paper, we present a prototype for improving intraoperative visualization by augmenting the video stream of the operation field with relevant patient data from different diagnostic and intraoperative imaging modalities. This will facilitate the identification of vital structures like blood vessels and nerves or landmarks like bony structures. This would potentially make surgical procedures safer and easier. In addition, the presented approach may serve as a teaching tool for surgical procedures, since critical anatomical relations can be identified and discussed before the actual procedure starts. 2
In this section, we adapt and extend our intraoperative visualization method [
For tracking, we implement a marker-based optical tracking server, which continuously captures a video stream of the trackable region with a high resolution camera (Logitech QuickCam Pro 9000; 1280x720 pixel at 30 fps). The tracking camera has to be calibrated once, the same way as the tablet PC camera. At each frame, the system searches for predefined markers and, for each detected marker, computes its pose in the camera coordinate system. Pose information of all detected markers are transmitted to the tablet PC using the VRPN (Virtual Reality Peripheral Network) protocol over a WLAN connection. Technically, this design allows for connecting to several commercial tracking systems.
Prior to the operation, a one-time, fully-automatic hand-eye calibration step is performed. In essence, the tablet PC camera is calibrated with respect to the tracker using a common reference marker that is visible (only at the calibration step) to both cameras. A second tracked marker is fixed to the tablet PC. Following the chain of transformations, the transformation from the tablet PC marker to its video camera is calculated as Tmc = Tcr:Tkr1:Tkm, where the transformations are video-camera to reference, tracker to reference, and tracker to tablet PC marker, respectively. The transformation Tmc remains valid as long as the marker does not move with respect to the video camera. At operation time, the patient is fixed with respect to the reference marker, which can then be removed from the view field of the camera. Thereafter, the transformation Tcr can be computed using Tcr = Tmc:Tkm1:Tkr. Since Tmc and Tkr are fixed from the previous step, Tcr is only dependent on the tracking data of the marker.
To register the patient/phantom with the scanned dataset (and hence with the anatomical models reconstructed from it), the reference marker is attached to a rectangular plastic board that is scanned with the patient. The corners of the marker are interactively selected from the scanned dataset, and their absolute positions are calculated, considering image spacing, and defined as registration points. 3D positions of the corresponding points in the patient coordinate system are precisely defined using the tracking system. The two sets of points are finally registered adapting a paired-point rigid registration scheme, applying a least square fitting approach. 2.2
The rendering module, running on the tablet PC, continuously captures a video
stream and renders it as a background. At each frame, relevant virtual objects
are rendered/overlaid using a two-pass rendering algorithm that highlights
objects silhouettes for better shape perception. Several objects can be overlaid
according to the current operation conditions. These include 3D reconstructions
of segmented structures from the anatomy dataset. Additionally, tomographical
slices can be superimposed. For this purpose, we adapt an optimized slicing
algorithm [
In minimally-invasive endoscopic or microscope-based spine surgery, augmentation should be performed on the video stream of the endoscope/microcope camera. However, this requires tracking these devices and a more complicated calibration of their cameras. 3
The software modules have been implemented with C++, OpenGL, VTK, and Qt. For our current implementation, we relied on marker-based tracking provided by the ARToolkit, which allows for multiple-marker tracking in real time. However, due to the inaccurate calibration, we calibrate camera parameters using MATLAB, as stated in Section 2.1. As a result, the new calibration was significantly more accurate regarding marker detection and pose estimation.
For our simulated OP scenario, a phantom model of the upper body is scanned with the mounted reference marker. After extrinsic calibration of the video camera, the phantom model is registered to the scanned data, using the corners of the reference marker as the set of correspondence point pairs. From a co-registered patient dataset, three vertebrae, inter-vertebral discs, and the spinal canal are segmented and 3D models are reconstructed. Finally, the visualization module starts video stream augmentation.
The upper-left part of Figure 1 depicts a snapshot of the GUI of the visualization module in a simulated spine surgery scenario. Figure 2 (left) shows a left posterior oblique (LPO) view, with augmented models of the lumbar vertebrae L2-L4 (cyan), inter-vertebral discs (green), and spinal canal (pink). Objects silhouettes are slightly highlighted for enhanced shape perception. In Figure 2 (right), a side view with an additional transparent overlay of a tomographical slice from the patient data is shown. The slicing algorithm allows for on-the-fly computation and rendering of slices at a near real-time rate. 4
A prototypic tablet PC based add-on for enhancing intra-operative visualization has been introduced, with the focus on spine surgeries. The hardware setup and functional components of the prototype have been depicted. We aim at raising the safety and accuracy factors of the intervention by superimposing relevant operation-specific anatomical structures on the video stream of the operation field, which obviously allows for the protection of risk structures like spinal nerves, blood vessels, and the spinal canal. In addition, targeted structures like tumors or herniated discs are better localized and information about the shape and extend of such structures can be conveyed at different depth levels. From surgical point of view, we have received positive feedback regarding the relevance and applicability of the presented approach. Our future work will focus on transferring the prototype for application in real operation suits and evaluating the applicability of the concept to other surgery scenarios. Acknowledgement. This work is funded by the German Ministry of Education and Science (BMBF) within the ViERforES project (No. 01IM08003C).