In general, minimally invasive procedures are less stressful for the patient. However, the surgeon does not have a clear view of the surgical field. Technical systems could assist the surgeon in orientation based on preoperative images. We developed a surgical assistance system for Transcatheter Aortic Valve Implantation to address these issues. The system is based on a Magic Lens concept. It combines tracking technology and visualization on a mobile display in real-time. A prototype was implemented to demonstrate the context and focus dependent presentation of patient information. The system allowed an intuitive interaction with preoperatively acquired patient data during the intervention. A preliminary user study with seventeen cardiac surgeons was conducted to evaluate the interaction concept and potential acceptance of such a system. The study results indicated the strong potential of the proposed concept and provided important hints for further development of the technique.
occlusion in the field of view. The visualization on the lens display finally combined the tracking information with preoperative patient data, i.e. CT, to provide a real-time rendering that respects the field of view of the surgeon. The context aspect required a clear visualization of the patient anatomy. Especially, occlusion of anatomical structures had to be resolved by adapted visualization techniques. Furthermore, an easy adaptation of the visualization to different surgical tasks was required. The Magic Lens system was designed for transfermoral as well as transapical approaches. Hence, the visualization had to address various surgical issues: provide an overview, present structure and course of vessels in relation to bone structures and show calcifications.
Preoperatively, a segmentation of the patient image data had to be generated including all relevant anatomical structures. Skin, bones, heart and main vessels with calcifications were required at least. Additional segmentations could be displayed if present. However, the preoperative pipeline is currently not in the focus of the project. A freely available dataset of the Institute of Research against Digestive Cancer at the University of Strasbourg was used for development and evaluation.
The intraoperative setup consisted of three major components: A tracking system, a workstation and an active display used as a lens. An accurate tracking of the lens and a robust tracking of the surgeon were required. Markers were not applicable for person tracking in the OR. Hence, a standard marker based tracking (NDI Polaris) was combined with a consumer Time-of-Flight camera (Microsoft Kinect). Both devices were attached and registered to each other. This allowed a marker-based lens tracking and a robust surgeon tracking in a shared coordinate system. The acquired tracking data were streamed to the workstation using a TiCoLi based OR bus implementation [8]. The workstation calculated the view to be rendered on the lens display. Thereto the tracking data were combined with a patient registration and the display size to generate a sheared camera in the virtual patient anatomy scene. The rendered image was streamed to the lens over a wireless network and displayed there.
The visualization pipeline was implemented based on the Visualization Toolkit (VTK). The reconstructed 3D models of the anatomical structures were used to render the view of the Magic Lens. Four presets were defined to adapt the visualization to the surgical situation. The default preset showed all available models of anatomical structures to provide an overview. None of the structures, except the skin, were displayed opaque for a clear presentation of their position to one another.
The transapical preset focused on the apex of the heart. The bones and the apex were displayed opaque whereas all the other structures were displayed semi-transparent. The visualization was combined with silhouette rendering to support the differentiation of structures and depth impression. A detail of the patient model displayed semi-transparent with silhouettes is depicted in figure 1.
The transfemoral preset was designed to present the vascular system. The vessels and the bones were rendered opaque. The rest of the structures were displayed semi-transparent again with silhouettes. This allowed a clear view on the vascular structures and their course. The preset was useful if the vessel planned for access could not be palpated manually.
Finally, a calcification preset was defined. The preset emphasized the calcifications with semi-transparent vessel structures. Heavy calcification at the incision is a risk factor for transcatheter procedures. Hence, the calcification preset supported finding a suitable incision point.
The selection of anatomical structures to display as well as all their visualization parameters, including color, opacity and silhouettes could be intraoperatively changed at the workstation. However, a sterile interaction was required for the surgeon. The surgeon could directly control the focus by lens and head movement. The person with a hand closest to the lens was expected to be the one interacting with the lens. The visualization was automatically adapted to the corresponding viewpoint. This increased the flexibility because it was possible to hand over the lens.
Additionally, the lens itself had a touch display to grab user input. The interaction needed to be simple and intuitive. Hence, the display was split into three areas. The presets could be changed at the left and right borders, switching forward and backward through the four presets. At the center of the display, the surgeon could toggle another mode. The mode allowed setting a clipping plane by moving the lens. The dataset, except the vessels and the heart were clipped transversal. This provided an additional possibility to avoid disturbing occlusions. A prototype of the Magic Lens for cardiac surgery was implemented. The setup in the demonstration OR is shown in figure 2 on the left. The infrared tracking camera with the attached Microsoft Kinect (A) is shown in the background. Next to that, the graphical user interface of the workstation (B) is visible. The lens (C) is implemented using a tablet located at the patient dummy (D). The right hand side of figure 2 shows an image of the Microsoft Kinect camera with the tracked skeleton of the user. The lens view adapted to the users head and lens movements in real-time on standard hardware. The view also sheared according to the viewing angle of the user. Hence, the Magic Lens system provided the field of view the user would have through the lens. The person tracking was very robust against partial occlusion caused by the OR table and the patient. Additionally, the marker-based tracking of the lens was very stable. The mobile view practically did not show any disturbing jitter. A photo of the Magic Lens in use and the corresponding visualization with the default preset is depicted in figure 3.
A preliminary user study was conducted to evaluate the potential of the proposed intraoperative assistance. Seventeen cardiac surgeons from the Herzzentrum Leipzig tested the prototype under laboratory conditions. A subsequent questionnaire focused on visualization quality, user interaction and general acceptance. Visualization and interaction were rated well on average. All surgeons indicated they would use such a type of assistance system at least in complicated cases. However, the results also indicated additional features that might be useful, such as see-through functionalities and overlay with preoperative planning data [9] and intraoperative imaging modalities.
We proposed a novel surgical assistance system for minimally invasive cardiac surgery based on the concept of a Magic Lens. The current prototype provided context and focus dependent anatomical information to the surgeon. Thus, it fulfilled the two basic requirements of the Magic Lens concept. The prototype demonstrated a way of interactive integration of preoperative patient data into the surgical area. The interaction is intuitive and reduces the workload for the surgeon to mentally integrate the information, in contrast to common stationary systems. However, there are still some limitations that need to be overcome. The preoperative segmentation workload needs to be reduced for clinical routine. Additionally, a surface-based registration technique is required because of the lack of stable anatomical landmarks. Nonetheless, the preliminary user study indicated the potential usefulness of the proposed concept. The intuitive way of interaction by movements and switching through the presets contributed to the acceptance of the designed system. The next important step for use in operating rooms will be an enhanced patient registrations and accuracy measurements. The implemented assistance system demonstrates a promising approach to interact with preoperative patient datasets during the intervention. The surgeon directly interacts with the data in an intuitive way. The surgeon is relieved of the task of integrating the data with the surgical area mentally. Thus, the system might contribute to patient safety, if the additional workload can be minimized to allow the use in clinical routine. Although the prototype was designed for minimally invasive cardiac surgery, the basic concept can be applied to different use cases in several clinical disciplines. ICCAS is funded by the German Federal Ministry of Education and Research (BMBF) and the Saxon Ministry of Science and Fine Arts (SMWK) in the scope of the Unternehmen Region with grant number 03Z1LN12 and by the European Regional Development Fund (ERDF) and the state of Saxony within the frame of measures to support the technology sector.