=Paper=
{{Paper
|id=None
|storemode=property
|title=Towards Mobile Augmented Reality for On-Patient Visualization of Medical Images
|pdfUrl=https://ceur-ws.org/Vol-715/bvm2011_80.pdf
|volume=Vol-715
}}
==Towards Mobile Augmented Reality for On-Patient Visualization of Medical Images==
https://ceur-ws.org/Vol-715/bvm2011_80.pdf
Towards Mobile Augmented Reality for
On-Patient Visualization of Medical Images
L. Maier-Hein1 , A. M. Franz1 , M. Fangerau1 , M. Schmidt2 , A. Seitel1 ,
S. Mersmann1 , T. Kilgus1 , A. Groch1 , K. Yung1 , T. R. dos Santos1 ,
H.-P. Meinzer1
1
German Cancer Research Center, Div. of Medical and Biological Informatics
2
Heidelberg Collaboratory for Image Processing, University of Heidelberg
l.maier-hein@dkfz-heidelberg.de
Abstract. Despite considerable technical and algorithmic developments
related to the fields of medical image acquisition and processing in the
past decade, the devices used for visualization of medical images have
undergone rather minor changes. As anatomical information is typically
shown on monitors provided by a radiological work station, the physician
has to mentally transfer internal structures shown on the screen to the
patient. In this work, we present a new approach to on-patient visual-
ization of 3D medical images, which combines the concept of augmented
reality (AR) with an intuitive interaction scheme. The method requires
mounting a Time-of-Flight (ToF) camera to a portable display (e.g., a
tablet PC). During the visualization process, the pose of the camera and
thus the viewing direction of the user is continuously determined with a
surface matching algorithm. By moving the device along the body of the
patient, the physician gets the impression of being able to look directly
into the human body. The concept can be used for intervention planning,
anatomy teaching and various other applications that require intuitive
visualization of 3D data.
1 Introduction
Visualization of medical images for disease diagnosis, surgical planning or as
orientation means during interventional therapy is an integral part of today’s
health care. As only few medical imaging modalities provide real-time images of
the patient’s anatomy, a common procedure involves acquiring static 3D images,
e.g. with a computed tomography (CT) or magnetic resonance imaging (MRI)
scanner, and visualizing the data on a regular monitor provided by a radiological
work station. In consequence, it is the task of the physician to transfer these
images mentally to the patient. Furthermore, navigation in the 3D data set
using the provided standard input devices may not be intuitive and thus requires
experience with the system.
In related work, some promising methods have been proposed for improv-
ing visualization during interventional therapy via augmented reality (AR), e.g.
by applying head-mounted displays [1] or intra-operative projector systems [2].
390 Maier-Hein et al.
In general, however, these methods require relatively bulky equipment such as
optical tracking systems and are thus not appropriate or too costly to be used
in the context of diagnosis, teaching, or surgical planning. To address this is-
sue, we present a new approach to on-patient visualization of anatomical data
that provides an augmented reality (AR) view of the medical data as well as an
intuitive interaction scheme without requiring expensive or bulky equipment.
2 Materials and Methods
2.1 Visualization Concept
The proposed visualization concept is based on a portable device such as a
tablet PC or an iPad with a mounted Time-of-Flight (ToF) camera as shown in
Fig. 1. The ToF camera is able to generate dense range images and corresponding
grayscale intensity images from a given scene in real-time [3]. Optionally, a color
camera calibrated with the ToF camera can be attached to the device to provide
high-resolution color images of the scene.
To enable on-patient visualization of a given static 3D data set, the images
stored in the Picture Archiving and Communications System (PACS) of the
clinic are transferred to the portable viewer (e.g., via a wireless connection).
Next, the skin as well as additional structures of interest are segmented (semi-)
automatically with the medical image processing software provided by the device.
Note in this context that the computations can optionally be performed remotely
on an external server. The physician may then move the portable monitor freely
along the body of the patient with the mounted ToF camera pointing in the
viewing direction of the user as shown in Fig. 1. During this procedure, the pose
of the ToF camera is continuously estimated by registering the ToF data with the
surface extracted from the static image. In consequence, the viewing direction
of the physician relative to the patient is known, and the virtual camera showing
the 3D data may be set accordingly. By looking onto the screen, which provides
a view on the internal anatomical structures using e.g. volume rendering or
surface rendering techniques, the physician obtains a kind of “x-ray vision” into
the patient. Furthermore, navigation through the 3D data set becomes more
intuitive because it is performed directly at the object of interest.
Given the hardware, which is already commercially available, implementation
of the concept thus requires real-time capable algorithms for (1) preprocessing
of the camera data, (2) camera pose estimation, and (3) visualization.
2.2 Feasibility Study
To investigate the feasibility of the proposed visualization concept, we imple-
mented a prototype based on the CamCube 2.0 (PMD Technologies, Siegen,
Germany) and the open-source software framework The Medical Imaging Inter-
action Toolkit (MITK) (www.mitk.org) using the following methods to process
each image:
Mobile Augmented Reality 391
Fig. 1. Concept for on-patient visualization: the pose of a portable device relative
to the patient is continuously estimated by registering time-of-flight range images to
static volume data.
– Preprocessing: After distortion correction based on a standard calibration
procedure for the intrinsic camera parameters, the range image is denoised
using a variant of the bilateral filter that takes into account the intensity
and distance measured in the individual pixels [4]. Next, a threshold filter
is applied to the distance image to segment those parts of the image that
correspond to the patient surface. Finally, the range image is converted
into Cartesian coordinates, and a Delaunay based triangulation method is
applied to generate a surface from the resulting point cloud.
– Camera pose estimation: In the current implementation, the pose of the
camera is continuously computed by matching the surfaces generated from
the ToF range images to the patient surface extracted from the static data
set. Initially, i.e., upon start of the visualization process, a two-stage reg-
istration process is performed to get an alignment: (1) First, both meshes
are segmented into regions sharing similar curvature properties, and a graph
representation is generated, where each node represents a region and each
edge connects neighboring regions. Next, a graph matching procedure [5] is
applied to find a rough alignment of the surfaces. (2) Based on this align-
ment, an anisotropic variant of the iterative closest point (ICP) algorithm [6]
is applied for the purpose of fine surface registration. The latter accounts for
the high noise along the viewing direction of the camera. After convergence,
the current transformation is used to get an initial alignment for the next
image frame, i.e., step (1) has to be performed only once at the beginning of
the procedure. Note in this context, that the registration problems arising
from symmetric body parts are potentially eliminated by applying a reg-
istration procedure that converges only to the nearest local optimum with
respect to a given cost function. Ambiguities can thus be compensated for,
given that the alignment for the previous image frame was correct. This as-
pect should be accounted for upon initialization of the visualization process,
i.e., by maximizing the area of the surface used for the initial alignment.
392 Maier-Hein et al.
– Visualization: In the current implementation, the physician may choose one
of four visualization modes: Volume rendering, surface rendering, virtual x-
ray view, and slice view, as shown in Fig. 2. Similar as in [1, 7], internal
structures are presented in relation to the skin surface by virtually cutting a
sphere-shaped hole into the skin as shown in Fig. 2. To avoid manipulation
of the skin surface mesh, the cutting is realized during the rendering process
using the concept of shading.
In a related contribution, we presented an evaluation framework which al-
lows for assessing the performance of ToF registration methods by providing
physically realistic ToF range data generated from a virtual scence [6]. In this
work, the framework was applied to perform first experiments on the feasibility
of continuous surface registration based on the anisotropic ICP: A sequence of
five ToF images was simulated from the CT image of a human face, and the
proposed registration method was applied to continuously register the simulated
ToF range data with the static 3D data set starting with a small known mis-
alignment. For each image frame (except for the first one), the estimated camera
pose of the previous frame was used to initialize the fine registration algorithm.
After convergence, the target registration error (TRE) for a set of virtual targets
distributed in the brain of the patient was determined to quantify registration
accuracy.
3 Results
A prototype for the visualization concept has successfully been implemented
with the following results: (i) preprocessing and surface generation is performed
robustly at 10 frames/sec on a standard PC; (ii) camera pose estimation based on
surface registration with the anisotropic ICP was successful in our experiments,
yielding a mean root-mean-square (RMS) TRE of 0.3 ± 0.1 mm (final value:
0.2 mm) compared to 2.1 ± 0.5 mm (final value: 2.3 mm) with the standard
ICP; (iii) visualization using the different modes represented in Fig. 2 works in
real-time.
4 Discussion
Compared to other visualization concepts introduced in the literature, the pro-
posed method has several major advantages: It (i) is markerless, (ii) involves no
bulky equipment, (iii) provides an intuitive mechanism for navigating through a
3D data set, and (iv) poses no restrictions on image acquisition (such as using
a tracked imaging modality, attaching markers, etc.). A seamless integration
into the clinical workflow, however, requires addressing several remaining issues.
Firstly, improvement of the ToF surfaces by calibration of systematic distance
errors is necessary to ensure robust registration results in practice. Secondly,
a real-time implementation of the fine surface registration method based on
Mobile Augmented Reality 393
(a) volume rendering (b) surface rendering
(c) x-ray view (d) slice view
Fig. 2. Visualization options. The x-ray view shows a simulated x-ray from the viewing
direction of the camera. In the slice view, the user may navigate through reconstructed
image planes whose normals point in the viewing direction of the user.
optimizing and parallelizing the weighted nearest neighbor computation is nec-
essary. In the long run, the registration method should further be able to cope
with varying angles between joints as well as with changing clothing. Finally,
the accuracy, run-time, and robustness of the method must be extensively eval-
uated with simulated and real data. The proposed concept could then become a
valuable tool for AR visualization of 3D images in the context of port planning,
anatomy teaching, and many other fields.
References
1. Navab N, et al. Action- and workflow-driven augmented reality for computer-aided
medical procedures. IEEE Comput Graph Appl. 2007;27(5):10–4.
2. Sugimoto M, et al. Image overlay navigation by markerless surface registration in
gastrointestinal, hepatobiliary and pancreatic surgery. J Hepatobiliary Pancreat Sci.
2010;17(5):629–36.
3. Kolb A, et al. Time-of-Flight Sensors in Computer Graphics. In: Proc Eurographics;
2009. p. 119–34.
4. Seitel A, et al. Adaptive bilateral filter for image denoising and its application to
in-vitro Time-of-Flight data. Proc SPIE. 2011;(in press).
5. dos Santos TR, et al. Correspondences search for surface-based intra-operative
registration. Lect Notes Computer Sci. 2010;6362:660–7.
6. Maier-Hein L, et al. Accounting for anisotropic noise in fine registration of Time-
of-Flight range data with high-resolution surface data. Lect Notes Computer Sci.
2010;6361:251–8.
7. Tietjen C, et al. GPU-basierte Smart Visibility Techniken für die Planung von
Tumor-Operationen. Proc BVM. 2009; p. 272–276.