=Paper=
{{Paper
|id=None
|storemode=property
|title=A Workflow for Improving Medical Visualization of Semantically Annotated CT-Images
|pdfUrl=https://ceur-ws.org/Vol-930/p3.pdf
|volume=Vol-930
|dblpUrl=https://dblp.org/rec/conf/semweb/BaranyaLCV12
}}
==A Workflow for Improving Medical Visualization of Semantically Annotated CT-Images==
https://ceur-ws.org/Vol-930/p3.pdf
A Workflow for Improving Medical Visualization
of Semantically Annotated CT-Images
Alexander Baranya1,2 , Luis Landaeta1,2 , Alexandra La Cruz1 , and
Maria-Esther Vidal2
1
Biophysic and Bioengeneering Applyed Group
2
Semantic Web Group
Simón Bolı́var University, Caracas, VENEZUELA
{abaranya,llandaeta,alacruz,mvidal}@ldc.usb.ve
Abstract. RadLex and Foundational Model of Anatomy (FMA)
ontologies represent anatomic and image characteristics, and they are
commonly used to annotate and describe contents of medical images
independently of the image acquisition method (e.g., CT, MR, or US).
We present ANISE, a framework that implements workflows to combine
these ontologies and image characteristics into Transfer Functions (TFs)
that map volume density values into optical properties. Semantics
encoded in the image annotations is exploited by reasoning processes
to improve accuracy of TFs and the quality of the resulting image.
1 Introduction
In the Life and Health Sciences domains large ontologies have been defined, e.g.,
SNOMED3 , MesH4 , RadLex5 , and Foundational Model of Anatomy (FMA) [9].
These ontologies are commonly applied to encode scientific knowledge through
annotations of concepts, e.g., MeSH terms have been used by curators to
annotate and describe PubMed6 publications and clinical trials published at
the Clinical Trials website7 . Knowledge encoded in these annotations as well as
the properties derived from reasoning tasks are used to recovery or discovery
properties of the annotated concepts. In this paper we propose a workflow to
annotate medical images with terms from RadLex and FMA, and illustrate the
benefits of exploiting these annotations during image visualization. We aim at
enriching transfer functions (TFs) with semantics encoded in these annotations
and provide more precise renderings of the volumetric data of a medical image.
A transfer function (TF) maps density values of volumetric data or voxel
into optical properties (e.g., opacity and color) used by rendering algorithms to
produce a final image. TFs allow to pre-classify different tissues in an image, and
3
http://www.nlm.nih.gov/research/umls/Snomed/snomed mail.html
4
http://www.nlm.nih.gov/mesh
5
http://www.rsna.org/radlex/
6
http://www.ncbi.nlm.nih.gov/pubmed
7
http://clinicaltrials.gov/
II
they are based on existing characterizations of the organs that relate a medical
image acquisition modality, a tissue, and a density range [7]. Nevertheless,
some tissues belonging to different organs may have overlapped densities, and
specifying a TF will normally require a robust segmentation technique and
specialized segmentation processes to produce a precise tissue classification able
to distinguish tissues with overlapped densities. Recently, the problem of tissue
classification by semantically annotating volumetric data has gained attention in
the literature [2, 3, 5, 8]. Rautek et al. [8] present a fuzzy rule-based system that
maps volumetric attributes to visual styles; rules are defined by users without
representing special knowledge about the rendering technique. Gerl et al. [5]
overcomes this limitation and propose a rule-based system for semantic shader
augmentation; this system automatically adds rule-based rendering functionality
to static visualization mappings in a shader program. Although both systems
rely on rule-based systems to characterize TFs, they do not exploit knowledge
encoded in ontologies to improve the quality of the visualization process. Möller
et al. [6] present a technique for annotating and searching medical images using
ontological semantic concepts for retrieving images from a Picture Archiving
and Communication System (PACS); ontologies as FMA and RadLex are
used to retrieve data, however, they are not exploited during visualization or
tissue classification from the image data. Although applications of semantic
annotations have been illustrated, nothing is said about the benefits of using
these annotations and the encoded semantics during the definition of TFs.
We present ANISE (an ANatomIc SEmantic annotator), a framework for
specifying TFs based on semantic annotations. TFs are based on pre-elaborated
semantic annotations of volumetric data which are validated against existing
medical ontologies. ANISE relies on a customized reasoner to infer the bounding
boxes which contain organs or tissues of a given sub-volume area, as well
as its main properties, e.g., density and opacity. Knowledge encoded in the
ontologies contribute to characterize and locating tissues by applying specific
organ selection algorithms; thus, voxels that are not part of the organ of interest
are not considered during the classification process.
This paper contains four additional sections. Section 2 describes ANISE
and Section 3 illustrates the ANISE workflow. Section 4 discusses the observed
results, and we conclude in Section 5 with an outlook to future work.
2 Architecture
Achieving high quality image rendering requires interpreting each intensity
value according to a given tissue. In consequence, a correct representation
of information through semantic annotations should ensure: i) minimal error
tissue classification due to reasoning and inference, and ii) an accurate visual
representation. Figure 1 shows the main components of ANISE: an annotator,
a rule-based system, and a visualization module. The Annotator extends
an image original annotations with terms that encode the properties of the
classified tissues. The rule-based system relies on inference tasks to process
III
ANNOTATED IMAGE Rule-Based System
s2
s1
v
b
s3 Ontology
Annotator a
RULES
r1(c1,o1) ->p1(c1, o1)
Volumetric data r2(c1,o1) , r3(c1,v3)->p2(c2, v3)
ANISE r1(c1,01), ¬r2(c1,02)->p3(c1,03)
VISUALIZATION
Ontology e.g.,
RadLex, FMA
FACTS
Density Values (range) TF Volume Rendering
Estimated location
(boundingbox)
Seep point inside the organ
Fig. 1. The ANISE architecture.
original annotations and derive facts that will be used to annotate an image.
Annotations regarding to visualization methods and anatomic parts are inferred
using Ontology relations (e.g., Subclass) for specific classes (e.g., the Anatomical
Set). Finally, the Visualization module executes visualization algorithms on the
annotated volumetric data.
Annotator: annotates an image with information about: i) resource authoring,
type and identification; ii) acquisition modality; iii) acquisition characteristics
like patient orientation in the image; iv) structural and anatomic elements
presented and identified in the image; v) regions and points of particular interest;
and vi) rendering information. ANISE relies on the following ontologies to extend
original image annotations:
– Foundational Model of Anatomy : FMA allows to describe membership
and spatial relationships among voxels in the volume to infer new facts.
Furthermore, there are terms in this ontology that can be used for annotating
non-anatomical elements, e.g., bounding boxes around particular anatomical
organs or some particular points of interest.
– RadLex : RadLex is an ontology defined for radiologist; it is composed
of terms required to annotate medical images. ANISE relies on RadLex
terms to describe characteristic from the image itself such as modality, and
other acquisition related characteristics that may alter the interpretation
and visualization of an image, e.g., orientation.
Rule-Based System: annotations are used during the inference process to
derive new annotations. First, it analyses the image acquisition characteristics
and correlates body structures of particular interest in order to normalize
information for further processing. A bounding box method is used to model
anatomic information [3]. Then, combining this information with tissue pre-
classification, the inference process is expressed in Probabilistic Soft Logic
(PSL) [1]; this process determines the likelihood for a given tissue to be included
in a particular region. Closely located tissues with similar intensity values are
usually treated as the same values; thus, spatial and anatomic information
is used to discriminate by annotating specific points; segmentation based on
IV
voxels neighborhood represent these tissues considering the associated semantic
annotations. Ontology classification reasoning tasks are performed with Jena8 .
Visualization Module: derived annotations are used by rendering algorithms
to visualize the classified tissues. Partial piece-wise transfer functions are used to
select appropriate color and opacity values and rendering them. Default transfer
functions are only applied on non-annotated voxels and regions.
3 Applying an ANISE Workflow- A Use Case
We illustrate the ANISE workflow in three different datasets (Table 1), to
visualize the FMA term dentition from a CT-Head volume data.
Volume Data Dimensions (voxels) Voxel size (mm) File size (MB)
skewed head.dat 184x256x170 1x1x1 16.0
visible head.dat 512x512x245 1x1x1 128.0
ct head.dat 256x256x113 1x1x2 14.8
Table 1. Datasets used for illustrating the utility of using semantic annotations on
Medical Images. These datasets are available in [10], [11] and [4] respectively.
Figure 2(a),(d),(g) illustrate the rendering of the images applying a simple
TF that maps density values to visualize the tissues that have the same density
that dentition; these tissues are colored in green. Although data were properly
pre-classified, it is not possible to discriminate only dentition by just considering
the corresponding densities, i.e., some other tissue were painted, and it was
not possible further tuning the TF. In this case the density value range for
identifying the dentition overlaps with density value range of other tissues like
bone for example. Nevertheless, if semantic annotations are used in conjunction
with knowledge encoded in the FMA and RadLex ontologies, ANISE can
determine that only the teeth should be colored different than the rest (green
in our example); this is done by selecting appropriate set of points, applying
Normalization rules, and considering the Image Modality taxonomy. Thus,
a better classification for different tissues can be done in an automatic way.
– Image Modality: supports a generic tissue classification process which is
independent on the image modality. The RadLex term used for Tomography
is RID288409 and the term RID10311 (imaging modality) can be reached
by using the SubClass relationship. Further, whenever the image is an MRI
the term RID10312 from the same taxonomy is used to annotate the image,
i.e., terms RID28840 and RID10312 share an ancestor RID10311. Tissues’
density ranges are represented as facts and used during the inference process
in conjunction with these annotations to pre-classify the image voxels.
– Volume format: ANISE current version receives images in raw format,
i.e., data correspond to a sequence of intensity values. This information is
recovered from the attribute format from DCMI10 metadata.
8
http://jena.apache.org/
9
http://purl.bioontology.org/ontology/RID/RID28840
10
http://dublincore.org/documents/dcmi-terms/
V
(a) (b) (c)
(d) (e) (f)
(g) (h) (i)
Fig. 2. Results of running the proposed approach with three different datasets: (a)
skewed head.dat, (d) visible head.dat and (g) ct head.dat. Images (b), (e), (h) results
from rendering without annotation and using a simple TF. Images (c), (f), (i) results
from rendering with the application of rule (1) and a semantically enhanced TF.
– Normalization rules: are used to transform volumes into a uniform scale
considering orientation, voxel size, and modality. Default values are assumed
if they are not given. In our use case, we used the term voxel geometry
RID2903 from RadLex and its ancestors in the subClass branch, i.e., non-
isotropic voxels, near-isotropic voxels, isotropic voxels.
– Dimension: we used the term location (RID39038) from RadLex to represent
header size, and dimensions in x, y and z of the volume.
– Tissue: dentition from FMA is the most relevant term in our use case.
We chose dentition because it is characterized as the tissue with the higher
density value, and the challenge consists on separating the dentition tissue from
tissues around it. PSL rules are used to compute the degrees of membership of a
voxel to the tissue of interest (dentition); it is mainly based on the density value
range. The rules that comprise the rule-based system are as follows; they specify
VI
TFs that better visualize the tissue of interest:
tissue(X, Y, Z, I) ∧ inside(X, Y, Z, R) ∧ inOrgan(X, Y, Z, I) → opacity(X, Y, Z).
(1)
where, truth values of opacity(X, Y, Z) are determined by the sum of truth values
of the following predicates:
– tissue(X, Y, Z, I) describes truth values of the voxel X, Y, Z with intensity
I that belong to the objective tissue. This value is defined by:
baseV oxel(X, Y, Z, I) ∧ tissueM ap(D, I) → tissue(X, Y, Z, I). (2)
where, baseVoxel(X,Y,Z,I) is a fact; tissueMap(D,I) is a PSL predicate that
assigns to an objective tissue D (e.g., dentition) the probability of the voxel
X,Y,Z belongs to the density value range. Initially a density value range is
specified, and as far as the inference over the annotations are generated, a
new density value range is produced and then, a more precise TF is defined.
– inside(X,Y,Z,R) describes truth values of the voxel X, Y, Z belonging to a
region R. Applying the inference process, a bounding box that best fits the
area of the tissue of interest is derived from an initial location.
– inOrgan(X,Y,Z,I) describes truth values of the voxel X, Y, Z belonging to
the same organ with intensity I. This value is defined by the rule:
baseV oxel(X, Y, Z, I) ∧ seed(X, Y, Z) → inOrgan(X, Y, Z, I). (3)
Given a seed point (seed(X,Y,Z)), known to be part of the tissue of
interest and analyzing its neighborhood, the area around this seed point
is augmented. A point will be part of the tissue if its density value is inside
the density value range of the tissue, and close to the tissue area.
Finally, some facts that need to be defined for each dataset are the following:
– Density value range: a density value range can be specified initially;
however, it can be adapted according to results inferred from the rules.
– Seed point: this is a fix value, received from the user describing a voxel
known to be part of the tissue of interest.
– Bounding box: the rule-based system identifies from an input bounding
box, one that better fits the tissue of interest.
4 Discussion
ANISE just considers the most likely localization of a given tissue. First, an
initial and basic TF is defined for a normalized model. Then, this model is used
for further inferences. Thus, rules are applied independently to the acquisition
method by selecting when a density value for a given point in the space falls
inside an appropriate interval. As previously stated, simple density classification
is not enough to properly determinate matching between voxels of a same
VII
tissue or anatomical organ. Additional inference processes need to be conducted;
they depend on the annotations. In this example, the region of interest that
describes the tissue to be analyzed is presented. A first approach consists of
selecting the most likely location of a region of interest, i.e., a bounding box
covering the organ of interest. Also, PSL predicates are considered as a possible
better approximation of this region with non-zero probability. This is done by
considering the neighborhood around the region of interest and knowing that
dentition, for example, should not be located around eyes or upper areas of
the head; voxels belonging to dentition should be closer around an area, and
distance between dentition voxels should not be longer than certain threshold.
Another inference process to adjust the
probability for points is performed by
anatomical entity
considering knowledge derived
i inmaterial anatomical entity
from ontology relationships, i.e., the i inguinal canal
classification of the term dentition in the i material anatomical entity
Anatomical Set branch. Considering the i anatomical structure
subClass transitive property (see Figure i dentition
3), a seed point is annotated to identify a i material anatomical entity
set element. Then, the voxel neighborhood i mouth floor
i tooth apex
detection algorithm is performed using
PSL predicates. Finally combining all Fig. 3. Scheme from FMA ontology,
inferred facts and probabilities for given identifying the Class and SubClass for
points, likelihood of points that represent dentition.
a particular tissue are estimated; Figure
4 illustrates the whole process. Further,
appropriate TFs for each region are defined and performed. This is done just
using the same TF (Fig. 2(b),(e),(h)) but performing a reasoning task that allows
to detect the voxels that semantically do not correspond to the tooth tissue and
that should not be included in the final volume rendering (see Fig. 2(c),(f),(i)).
Fig. 4. The ANISE Workflow
VIII
5 Conclusions and Future Work
We present ANISE, a framework that exploits knowledge encoded by annotations
of 3D medical images, and enhances the rendering process of the images.
Quality of ANISE renderings have been studied in different images, and we have
observed that they can accurately locate tissues that comprised a medical image.
Annotations allow identifying or validating patterns on images, accurate image
retrieval, and applying the visualization process on regions of interest. Methods
to filter relevant information have been developed at high abstraction level,
allowing extension of the inference process to perform particular algorithms, i.e.,
voxel neighborhood predicates could be improved to allow different methods. In
the future, we plan to enhance the rule-based system to normalize a wider range
of conditions, and include different image modalities (e.g., MR, and PET) as well
as tissues (e.g., blood vessels). Furthermore, we will extend tissue identification
algorithms and rules to: i) detect and annotate anomalies, and ii) identify special
conditions on tissues inside the region of interest. Development of visualization
algorithms to consider not only TF definitions but also different interpretations
of semantic annotations of particular tissues of interest and its corresponding
representation on rendered image is also part of our future work.
References
1. M. Broecheler, L. Mihalkova, and L. Getoor. Probabilistic similarity logic. In
Conference on Uncertainty in Artificial Intelligence, 2010.
2. A. Criminisi, J. Shotton, and S. Bucciarelli. Decision forests with long-range spatial
context for organ localization in ct volumes. In MICCAI workshop on Probabilistic
Models for Medical Image Analysis (MICCAI-PMMIA. Springer), 2009.
3. A. Criminisi, J. Shotton, and E. Konukoglu. Decision forests: A unified framework
for classification, regression, density estimation, manifold learning and semi-
supervised learning. Foundations and Trends in Computer Graphics and Vision,
7(2-3), 2012.
4. http://www-graphics.stanford.edu/data/voldata/CThead.tar.gz.
5. M. Gerl, P. Rautek, T. Isenberg, and E. Gröller. Semantics by analogy for
illustrative volume visualization. Computers & Graphics, 36(3):201–213, 2012.
6. M. M´’oller and S. Mukherjee. Context-driven ontological annotations in dicom
images: Towards semantic pacs. In Proceedings of International Joint Conference
on Biomedical Engineering Systems and Technologies, 2008.
7. B. Preim and D. Bartz. Visualization in Medicine: Theory, Algorithms, and
Applications. The Morgan Kaufmann Series in Computer Graphics., 2007.
8. P. Rautek, S. Bruckner, and E. Gröller. Semantic layers for illustrative volume
rendering. IEEE Trans. Vis. Comput. Graph., 13(6):1336–1343, 2007.
9. C. Rosse and J. Mejino. The foundational model of anatomy ontology. In Anatomy
Ontologies for Bioinformatics: Principles and Practice. The Morgan Kaufmann
Series in Computer Graphics., 2007.
10. http://www.cg.tuwien.ac.at/courses/Visualisierung/1999-2000/skewed head.zip.
11. http://mri.radiology.uiowa.edu/VHDicom/VHMCT1mm/VHMCT1mm Head.tar.gz.