Multi-modal Multi-Atlas Segmentation using Discrete Optimisation and Self-Similarities Mattias P. Heinrich Oskar Maier Heinz Handels heinrich@imi.uni-luebeck.de Institute of Medical Informatics, University of Lübeck, Germany Abstract This work presents the application of a discrete medical im- age registration framework to multi-organ segmentation in different modalities. The algorithm works completely auto- matically and does not have to be tuned specifically for dif- ferent datasets. A robust similarity measure, using the local self-similarity context (SSC), is employed and shown to out- perform other commonly used metrics. Both affine and de- formable registration are driven by a dense displacement sam- pling (deeds) strategy. The smoothness of displacements is enforced by inference on a Markov random field (MRF), using a tree approximation for computational efficiency. Consen- sus segmentations for unseen test images of the VISCERAL Anatomy 3 data are found by majority voting. 1 Introduction Organ segmentations are an important processing step in medical image analysis, e.g. for image- guided interventions, radiotherapy, or improved radiological diagnostics. General solutions are preferable over organ specific models for large scale image processing. Machine learning approaches, in particular the popular random decision forests (RDF), have been recently used for multi-organ localisation [CSB09] and segmentation [GPKC12], yet for more challenging modalities (e.g. struc- tural MRI) they have had limited success. This is partly due to the inhomogeneous intensity variations within and across MR scans. Registration-based multi-atlas segmentation can provide more robustness by using contrast-invariant similarity measures to guide the alignment of atlas to patient data. Here, we propose to employ a discrete registration model, which can capture large de- formations to accurately segment volumes with large differences in patient anatomy and geometry. Combined with a robust multi-modal similarity metric (self-similarity context) it can be applied to registering both CT and MRI scans reliably. The method is briefly reviewed in the next section Copyright c by the paper’s authors. Copying permitted only for private and academic purposes. In: O. Goksel (ed.): Proceedings of the VISCERAL Anatomy Grand Challenge at the 2015 IEEE International Symposium on Biomedical Imaging (ISBI), New York, NY, Apr 16th , 2015 published at http://ceur-ws.org and the experimental setting detailed thereafter. The results on both training and test datasets are discussed in Sec. 4 and compared to some state-of-the-art approaches. 2 Method Discrete optimisation can capture large motions by defining an appropriate range of displacements u. It enables a flexible choice of different similarity terms, since no derivative is required. We use the framework presented in [HJBS13], which defines a graphical model with nodes p ∈ V (with spatial location xp ) that correspond to control points in a uniform B-spline grid. For each node, the hidden labels fp (from a large quantised set L) are defined as potential 3D displacements fp = up = {up , vp , wp } between a control point p in the fixed image F and moving image M . Edges between nodes used for inference of the pair-wise regularisation costs R(fp , fq ) (p, q ∈ E) are modelled by a minimum spanning tree (MST) for computational efficiency. The displacement field is regularised using the squared differences of the displacements of neighbouring control points: X ||up − uq ||2 R(fp , fq ) = (1) ||xp − xq || (p,q)∈E For the image similarity (data term) self-similarity descriptors are used [HJP+ 13]. The self- similarity context is based on local patch distances within each image and invariant to contrast change, robust to noise and modality independent. The dissimilarity metric D, the L1 norm be- tween 64 bit binary descriptor representations SSCF (for fixed image) and SSCM (for moving image) at two locations x and x + u, can be efficiently calculated in the Hamming space: X D(xp , up ) = 1/|P| Ξ{SSCF (xp + y) ⊕ SSCM (xp + up + y)} (2) y∈P where ⊕ defines an exclusive OR, Ξ a population count and y ∈ P the local patch Pcoordinates. The P combined energy function with regularisation parameter α becomes: E(f ) = p∈V D(fp ) + α (p,q)∈E R(fp , fq ). Belief propagation [FH06] on the MST (our relaxed graphical) is employed to find the global minimum without iterations in only two passes. Prior to the deformable registration, a block-matching based linear registration using also the SSC metric is employed as detailed in [HPSH14].1 3 Experiments The deformations between different anatomies make a large number of degrees of freedoms nec- essary. As pre-processing the images are resampled to an isotropic resolution of 1.8 × 1.8 × 1.8 mm3 and padded or cropped to have same dimensions. For the affine pre-registration, three scales of control-point grids with spacings of [9, 8, 7] voxels are used. The displacement label space is defined by two parameters: number of steps lmax and quantisation step q, which together define the label space L = q · {0, ±1, . . . , ±lmax }3 voxels. We used lmax = [6, 5, 4] and q = [5, 4, 3] voxels. For the deformable registration four scale levels with spacings of [8, 7, 6, 5], numbers of steps of lmax = [6, 5, 4, 3] and quantisations of q = [4, 3, 2, 1] voxels were used. The number of random sam- ples and the regularisation weight were left at their default parameters 50 and 2. Inverse consistent is improved by employing a symmetric calculation of deformations (see [HJP+ 13]). To asses the impact of the similarity metric, we additionally performed experiments using mutual information (MI) and normalised gradient fields (NGF) [HM05]. For a more detailed comparison of the optimisation, we also applied the popular continuous-optimisation based framework NiftyReg [MRT+ 10] (which uses a B-spline parameterisation) with an affine initialisation [ORPA00]. 1 Our software is publicly available for download at www.mpheinrich.de (deedsRegSSC) Table 1: Experimental results for training dataset of VISCERAL Anatomy 3 challenge. Dice volume overlap for the 7 most common organs (psoas major muscles are abbreviated by pmm) in abdominal and thorax scans when using majority voting. The results of [GSG14] are from a different subset of the challenge (Anatomy 2), so there are not directly comparable. method liver spleen bladder r kidney l kidney r pmm l pmm avg deeds+SSC CT-CT 0.92 0.84 0.82 0.91 0.91 0.85 0.84 0.872 deeds+MI MR-CT 0.77 0.66 0.16 0.52 0.85 0.69 0.64 0.610 deeds+NGF MR-CT 0.77 0.74 0.31 0.55 0.86 0.75 0.73 0.673 deeds+SSC MR-CT 0.82 0.78 0.44 0.62 0.88 0.80 0.79 0.732 NiftyReg+MI MR-MR 0.81 0.79 0.05 0.58 0.77 0.52 0.36 0.554 [GSG14] MR-MR 0.83 0.66 0.21 0.88 0.85 0.64 0.677 deeds+SSC MR-MR 0.80 0.82 0.63 0.55 0.88 0.79 0.76 0.744 proposed Test MR-MR 0.79 0.71 0.36 0.78 0.83 0.76 0.78 0.714 4 Results Our results are summarised in Table 1 for a subset of 10 training scans of the contrast enhanced (ce) abdominal MRI modality (or thorax/adominal ceCT) and a leave-one-out validation. It can be seen that MRI segmentation is substantially more challenging yielding average results of Dice overlap for 7 organs of at most 0.744, while the results for the same setting for CT scans are ≈0.13 higher. Either of the two compared discrete optimisation strategies, by Gass et al. [GSG14] and our framework [HJBS13], outperforms the continuous optimisation approach of [MRT+ 10]. Using SSC as similarity metric improves the segmentation by 0.12 compared to MI and by 0.06 compared to NGF within the same framework. The multi-modal segmentation, for which we used MRI scans as fixed and CT scans as moving atlas scans, shows nearly identical accuracy to using same modality priors. This is an interesting finding, which could be employed for generating synthetic CT scans from MRI scans, e.g. for MR-PET reconstruction [HSS+ 08]. Due to time limitations only preliminary results for the hidden test datasets could be computed (last row of Table 1), for which we employed only three atlas scans each. We anticipate further improvements for our final results, which will subsequently be published on the VISCERAL leaderboard. The run-time of our algorithm on the virtual machine was on average 4 minutes per registration, which can be reduced with an optimised CPU implementation to less than a minute. 5 Conclusion We have demonstrated that deformable registration using discrete optimisation enables accurate automatic MRI organ segmentation. Choosing both a robust similarity metric and optimisation strategy has been found to be important for achieving high overlap. Local similarity-weighted atlas performance estimation and advanced label fusion [AL13] may further improve the results. While machine learning techniques alone may not achieve the same accuracy as registration-based approaches for MRI segmentation, the combination of both can boost the performance. In initial experiments, we found that an RDF trained with both atlas-based priors and intensity features [MWG+ 15] improves the segmentation overlap of liver, spleen and kidneys by ≈0.06. References [AL13] Andrew J Asman and Bennett A Landman. Non-local statistical label fusion for multi- atlas segmentation. Medical Image Analysis, 17(2):194–208, 2013. [CSB09] Antonio Criminisi, Jamie Shotton, and Stefano Bucciarelli. Decision forests with long- range spatial context for organ localization in CT volumes. MICCAI workshop on Probabilistic Models for Medical Image Analysis, pages 69–80, 2009. [FH06] Pedro Felzenszwalb and Daniel Huttenlocher. Efficient belief propagation for early vision. Internation Journal of Computer Vision, 70:41–54, 2006. [GPKC12] Ben Glocker, Olivier Pauly, Ender Konukoglu, and Antonio Criminisi. Joint classification-regression forests for spatially structured multi-object segmentation. In ECCV 2012, pages 870–881. Springer, 2012. [GSG14] Tobias Gass, Gabor Szekely, and Orcun Goksel. Multi-atlas segmentation and land- mark localization in images with large field of view. In Medical Computer Vision: Algorithms for Big Data, pages 171–180. Springer, 2014. [HJBS13] Mattias P. Heinrich, Mark Jenkinson, Sir Michael Brady, and Julia A. Schnabel. MRF- based deformable registration and ventilation estimation of lung CT. IEEE Transac- tions on Medical Imaging, 32(7):1239–1248, 2013. [HJP+ 13] Mattias P. Heinrich, Mark Jenkinson, Bartlomiej W. Papież, Sir Michael Brady, and Julia A. Schnabel. Towards realtime multimodal fusion for image-guided interventions using self-similarities. In MICCAI, LNCS, pages 187–194. Springer, 2013. [HM05] Eldad Haber and Jan Modersitzki. Beyond mutual information: A simple and robust alternative. In Bildverarbeitung für die Medizin 2005, pages 350–354. Springer, 2005. [HPSH14] Mattias P. Heinrich, Bartlomiej W. Papież, Julia A. Schnabel, and Heinz Handels. Mul- tispectral image registration based on local canonical correlation analysis. In MICCAI, LNCS, pages 202–209. Springer, 2014. [HSS+ 08] Matthias Hofmann, Florian Steinke, Verena Scheel, Guillaume Charpiat, Jason Far- quhar, Philip Aschoff, Sir Michael Brady, Bernhard Schölkopf, and Bernd J. Pichler. MRI-based attenuation correction for PET/MRI: a novel approach combining pattern recognition and atlas registration. Journal of Nuclear Medicine, 49(11):1875–1883, 2008. [MRT+ 10] Marc Modat, Gerard R Ridgway, Zeike A Taylor, Manja Lehmann, Josephine Barnes, David J Hawkes, Nick C Fox, and Sébastien Ourselin. Fast free-form deformation using graphics processing units. Computer Methods and Programs in Biomedicine, 98(3):278–284, 2010. [MWG+ 15] Oskar Maier, Matthias Wilms, Janina von der Gablentz, Ulrike M Krämer, Thomas F Münte, and Heinz Handels. Extra tree forests for sub-acute ischemic stroke lesion segmentation in MR sequences. Journal of Neuroscience Methods, 240:89–100, 2015. [ORPA00] Sébastien Ourselin, Alexis Roche, Sylvain Prima, and Nicholas Ayache. Block match- ing: A general framework to improve robustness of rigid registration of medical images. In MICCAI 2000, pages 557–566. Springer, 2000.