=Paper=
{{Paper
|id=Vol-1747/IT606_ICBO2016
|storemode=property
|title=Measuring the Importance of Annotation Granularity to the Detection of Semantic Similarity Between Phenotype Profiles
|pdfUrl=https://ceur-ws.org/Vol-1747/IT606_ICBO2016.pdf
|volume=Vol-1747
|authors=Prashanti Manda,James P. Balhoff,Todd J. Vision
|dblpUrl=https://dblp.org/rec/conf/icbo/MandaBV16
}}
==Measuring the Importance of Annotation Granularity to the Detection of Semantic Similarity Between Phenotype Profiles ==
https://ceur-ws.org/Vol-1747/IT606_ICBO2016.pdf
1
Measuring the importance of annotation granularity
to the detection of semantic similarity between
phenotype profiles
Prashanti Manda1 , James P. Balhoff2 and Todd J. Vision1
1
Department of Biology, University of North Carolina at Chapel Hill, NC, USA
2
RTI International, NC, USA
Abstract—In phenotype annotations curated from the biolog- The EQ formalism has more recently been adopted by the
ical and medical literature, considerable human effort must be Phenoscape project to curate phenotypes from the literature
invested to select ontological classes that capture the expressivity that are reported to vary among evolutionary lineages [4] with
of the original natural language descriptions, and finer annotation the goal of linking them to gene phenotypes and generating
granularity can also entail higher computational costs for partic- hypotheses about the genetic bases of evolutionary transitions
ular reasoning tasks. Do coarse annotations suffice for certain
[5].
applications? Here, we measure how annotation granularity
affects the statistical behavior of semantic similarity metrics. In the EQ approach, an entity represents a biological ob-
We use a randomized dataset of phenotype profiles drawn ject, e.g. an anatomical structure, an anatomical space, or
from 57,051 taxon-phenotype annotations in the Phenoscape a biological process, while a quality represents a trait or
Knowledgebase. We compared query profiles having variable property that an entity possesses, e.g, shape, color, or size.
proportions of matching phenotypes to subject database profiles Curators often create complex logical expressions called post-
using both pairwise and groupwise Jaccard (edge-based) and compositions by combining ontology terms, relations, and
Resnik (node-based) semantic similarity metrics, and compared spatial properties from multiple ontologies in different ways
statistical performance for three different levels of annotation
granularity: entities alone, entities plus attributes, and entities
to create entities and qualities that adequately represent phe-
plus qualities (with implicit attributes). All four metrics examined notypic descriptions. For example, “big supraorbital bone” is
showed more extreme values than expected by chance when represented as E: supraorbital bone (UBERON 0004747), Q:
approximately half the annotations matched between the query enlarged size (PATO˙0000586). A more complex description
and subject profiles, with a more sudden decline for pairwise such as “parietal fused with supraoccipital bone” is repre-
statistics and a more gradual one for the groupwise statistics. sented by relating the two affected entities, supraorbital bone
Annotation granularity had a negligible effect on the position of (UBERON 0004747) and parietal (UBERON 2001997) using
the threshold at which matches could be discriminated from noise. the quality fused with (PATO 0000642).
These results suggest that coarse annotations of phenotypes, at
the level of entities with or without attributes, may be sufficient to
Annotation of phenotypes at this level of ontological detail
identify phenotype profiles with statistically significant semantic is time consuming and expensive [6]. Annotating evolutionary
similarity. phenotypes at the finest level of granularity often requires
curators to create new ontology terms and request those terms
Keywords—ontology, phenotype, curation, annotation granular- to be added to the ontology. Coarse annotation removes the
ity, semantic similarity need for ontology development by limiting curators to a small
set of attribute level qualities already present in the ontology.
I. I NTRODUCTION Reducing the effort on curatorial tasks such as ontology
development and data preparation improves the annotation rate
To make phenotype descriptions in the biological and medi- from two characters per hour to 14 characters per hour [6].
cal literature amenable to large-scale discovery and compu- Thus coarse annotation can be part of an efficient annotation
tation, a variety of efforts have been launched to convert workflow, and permit larger datasets to be curated for equiv-
such descriptions into logical expressions using ontologies and alent resources. In addition, reasoning over the combinatorial
to integrate them into the larger ecosystem of online, open entity and quality ontology space for EQ annotations poses a
biological information resources [1]. Typically, this involves serious computational challenge.
curation and annotation of phenotypes in the Entity-Quality
Given these competing considerations, what level of anno-
(EQ) formalism [2], which is widely used by model organ-
tation granularity is optimal? The answer may depend on the
ism communities for representation of gene phenotypes [3].
particular application. For Phenoscape, a major goal is to be
manda.prashanti@gmail.com able to find sets of phenotypes that show greater semantic simi-
jbalhoff@rti.org larity than would be expected by chance when comparing sets
tjv@bio.unc.edu of phenotypes from different biological domains (e.g. those
2
observed in evolutionary lineages versus those induced by Since the minimum value of I(A) is zero, at the root of the
genetic manipulations in the laboratory) [5]. When comparing ontology, while the maximum value is − log(1/T ), we can
phenotypes with such different biological origins, we would compute a Normalized Information Content (In ) with range
not expect to see congruence in fine detail for a variety of [0, 1]
reasons. For instance, even if the same or homologous genes I(A)
In (A) =
have contributed to the two profiles, independent changes to − log(1/T )
those genes may underpin the phenotypes, they may be in
The Resnik similarity (sR ) of two ontology classes is defined
lineages for which the genetic networks have diverged, and
as the Normalized Information Content of the least common
there may have been considerable evolutionary modification
subsumer (LCS) of the two classes.
of the phenotype since its first origin. Even if two biological
phenotypes are identical, the way in which the phenotypes sR (A, B) = In (LCS(A, B))
are observed and described by independent researchers may
lead to natural language descriptions, and thus profiles of B. Profile similarity
annotations, that are quite different. With such weak matches,
do finer annotations enable similarities to be detected, or are A set of ontology-based phenotype annotations is called a
finer annotations superfluous or even distracting? phenotype profile. When comparing two profiles, X and Y ,
To explore this issue, we have conducted experiments to where each has at least one, and potentially many annotations,
test the statistical sensitivity of semantic similarity at vary- we could either summarize all the pairwise combinations
ing annotation granularity. Our approach involves simulating of annotations, or we could compute a groupwise similarity
phenotype profiles by sampling from real annotations drawn measure directly as a function of graph overlap.
from the Phenoscape Knowledgebase [5]. We measured sim- 1) Best Pairs: Pairwise approaches summarize the distribu-
ilarity between profiles that shared all, some or none of tion of pairwise Jaccard or Resnik similarity scores between
their annotations, with the remainder drawn randomly from annotations in the two profiles. Here we use the Best Pairs
the population of annotations. We assessed the decline of score. For each annotation in X, the best scoring match in
semantic similarity to the point at which it could no longer Y is determined, and the median of the |X| resultant values
be discriminated from random chance. This was done for four is taken. Similarly, for each annotation in Y , the best scoring
different semantic similarity statistics, and for three levels of match in X is determined, and the median of the |Y | values
annotation granularity. is taken. The Best Pairs score pz (X, Y ) is the mean of these
two medians. The index z can be used to denote whether the
II. M ETHODS pairwise values are Resnik (z = R) or Jaccard (z = J).
A. Semantic similarity metrics pz (X, Y ) = (1/2)[bz (X, Y ) + bz (Y, X)]
The four semantic similarity statistics we have chosen
represent extremes along two different dimensions by which where
n o
semantic similarity metrics vary [7–10]. Edge-based semantic bz (X, Y ) = median sz (Xi , Yj )
similarity metrics use the distance between terms in the on- i∈{1...|X|},j=argmax sz (Xi ,Yj )
j=1...|Y |
tology as a measure of similarity. Node-based measures use
the Information Content of the annotations to the terms being Note that, as defined, pz (X, Y ) = pz (Y, X).
compared and/or their least common subsumer. The similarity 2) Groupwise: Groupwise approaches compare profiles di-
metrics we have chosen are based on Jaccard (edge-based) and rectly based on set operations or graph overlap.
Resnik (node-based) similarity, which are popular in biological The Groupwise Jaccard similarity of profiles X and Y ,
applications (e.g. [11]). For each, we have one version that gJ (X, Y ), is defined as the ratio of the number of classes
summarizes the distribution of pairwise similarities between in the intersection to the number of classes in the union of the
two sets of annotations, and another that calculates a groupwise two profiles
score directly.
1) Jaccard similarity: The Jaccard similarity (sJ ) of two |C(X) ∩ C(Y )|
gJ (X, Y ) =
classes (A, B) in an ontology is defined as the ratio of the |C(X) ∪ C(Y )|
number of classes in the intersection of their subsumers over
the number of classes in their union of their subsumers [12]. where C(X) is the set of classes belonging to X plus their
subsumers.
|S(A) ∩ S(B)| Similarly, the Groupwise Resnik similarity of profiles X
sJ (A, B) =
|S(A) ∪ S(B)| and Y , gR (X, Y ), is defined as the ratio of the normalized
where S(A) is the set of classes that subsume A. information content summed over all nodes in the intersection
2) Resnik similarity: The Information Content of ontology of X, Y to the information content summed over all nodes in
class A, denoted I(A) is defined as the negative logarithm of the union.
the proportion of profiles annotated to that class f (A) out of
P
t∈{C(X)∩C(Y )} In (t)
T profiles in total. gR (X, Y ), = P
t∈{C(X)∪C(Y )} In (t)
f (A)
I(A) = − log where C(X) is defined as above.
T
3
Fig. 1. Profile decay via iterative replacement. Query profiles are selected
from the pool of simulated profiles (lower left). Filled circles represent 1.0
Best Pairs Groupwise
annotations, and annotations within the same profile are enclosed by boxes.
Circles of the same color represent the same annotation. At each iteration, one 0.8
of the remaining original annotations in the query profile is replaced with a 0.6
randomly selected annotation from the pool. The process continues until each Jaccard
of the annotations in the original query profile has been replaced. 0.4
Decayed Profiles 0.2
Similarity score
1 random 2 random 0.0 E
annotation annotations 1.0 EA
Query Profile EQ
0.8
0.6
Resnik
Simulated 0.4
Profiles
Sampling 0.2
without replacement 0.0
2 4 6 8 10 2 4 6 8 10
Annotation Number of annotations replaced
Pool
Fig. 2. Pattern of similarity decay with E, EA, and EQ data as profiles
are decayed via Random Replacement. Solid lines represent the mean best
match similarity of the 5 query profiles to the database after each annotation
C. Source data replacement. Error bars show 2 standard errors of the mean. Dotted lines
represent the 99.9th percentile of the noise distribution.
The Phenoscape Knowledgebase contains a dataset of 661
taxa with 57,051 evolutionary phenotypes, which are phe-
notypes that have been inferred to vary among the taxon’s granularity: entity only (E), entity-attribute (EA), and entity-
immediate descendents [5]. A simulation dataset of subject quality (EQ), we used three different phenotype ontologies,
profiles having the same size distribution of annotations per one for each granularity level, containing phenotype concepts
taxon was created by permutation of the taxon labels. combining terms from Uberon (entities) and PATO (attributes
and qualities). In each evaluation, annotations in the query pro-
files and the simulated database will match at the granularity
D. Simulating profile ‘decay’ level available in the generated phenotype ontology.
To simulate decay of profile similarity, five query profiles
of size ten were randomly selected from the simulated dataset. III. R ESULTS AND D ISCUSSION
For each, there is one profile among the set of subjects for We measured semantic similarity between each of the five
which each annotation has a one-to-one perfect match. For query profiles and their decay series to all 661 profiles in
each of the five profiles, ten progressively decayed profiles the subject database. This was done for each of the four
were obtained by iteratively replacing one of the original semantic similarity metrics (Best Pairs and Groupwise variants
annotations with an annotation randomly selected from among of Jaccard and Resnik metrics) and for each of the three
the 57,051 available (Figure II-D). Thus, for each original granularity levels (E: Entity only, EA: Entity-Attribute, and
profile, there is a profile in which one original annotation has EQ: Entity-Quality). The results are shown in Figure 2). For
been replaced with random annotation, another in which two ease of interpretation, we take the upper 99.9% of the simi-
have been replaced, and so on, through to a fully decayed larity distribution for random profile matches as an arbitrary
profile in which all original annotations have been replaced threshold for comparing the sensitivity of the different series.
with a random one. To characterize the noise distribution for All series cross this threshold when approximately half of
each metric in the absence of semantic similarity, we also the annotations have been replaced, with a sudden decline
generated 5,000 profiles of size ten by drawing annotations in similarity for the Best Pairs statistics and a more gradual
randomly from among the 57,051 available. These profiles decline for the groupwise statistics. While the differences
would not be expected to have more than nominal similarity in sensitivity among the annotation granularity levels are
with any of the simulated subject profiles. subtle, the annotations of intermediate granularity (EA) have
marginally greater sensitivity for all four statistics.
The sharp decline in similarity under the Best Pairs statistics
E. Adjusting annotation granularity at approximately 50% decay can be understood as a result
The evolutionary phenotypes available from Phenoscape of summarizing the pairwise distribution with the median. In
have been annotated with both entities and qualities, and future work, we aim to explore how the sensitivity of pairwise
the intermediate level of attribute is implicit in the quality statistics might be tuned by using different percentiles. Given
annotation due the structure of the PATO quality ontology the relatively flat performance of the Best Pairs statistics when
[4]. In order to measure semantic similarity for three levels of decay was under 50%, we suggest that groupwise statistics are
4
likely to provide greater discrimination between true matches J. Fernandez-Triana, M. Vihinen, P. D. Vize, L. Vogt, C. E. Wall, R. L.
of varying quality and thus better for rank ordering the Walls, M. Westerfeld, R. A. Wharton, C. S. Wirkner, J. B. Woolley,
outcome of semantic similarity searches, e.g. [5]. Our results M. J. Yoder, A. M. Zorn, and P. Mabee, “Finding our way through
phenotypes,” PLoS Biology, vol. 13, no. 1, p. e1002033, 2015.
also illustrate how difficult it can be to statistically discriminate
[2] G. V. Gkoutos, E. C. Green, A.-M. Mallon, J. M. Hancock, and
weakly matching profiles from noise, something which has D. Davidson, “Using ontologies to describe mouse phenotypes,”
received relatively little consideration in many applications of Genome Biology, vol. 6, no. 1, p. R8, 2004.
semantic similarity search to date. [3] C. J. Mungall, G. V. Gkoutos, C. L. Smith, M. A. Haendel, S. E. Lewis,
The relatively minor differences in statistical perfor- and M. Ashburner, “Integrating phenotype ontologies across multiple
mance with varying annotation granularity, with EA showing species,” Genome Biology, vol. 11, no. 1, p. R2, 2010.
marginally greater sensitivity, has implications both for the [4] W. M. Dahdul, J. P. Balhoff, J. Engeman, T. Grande, E. J. Hilton,
process of generating annotations and the implementation of C. Kothari, H. Lapp, J. G. Lundberg, P. E. Midford, T. J. Vision,
M. Westerfield, and P. M. Mabee, “Evolutionary characters, phenotypes
semantic similarity computation. As noted in the Introduction, and ontologies: curating data from the systematic biology literature,”
annotation to EA requires considerably less human curation PLoS One, vol. 5, no. 5, p. e10708, 2010.
effort than EQ, and is almost identical in effort to curation [5] P. Manda, J. P. Balhoff, H. Lapp, P. Mabee, and T. J. Vision, “Using the
to E. Restricting annotation granularity to EA may also ease phenoscape knowledgebase to relate genetic perturbations to phenotypic
the challenge of speeding of curation through machine-aided evolution,” Genesis, vol. 53, no. 8, pp. 561–571, 2015.
natural language processing, e.g. [13]. [6] W. Dahdul, T. A. Dececchi, N. Ibrahim, H. Lapp, and P. Mabee,
Second, the computational expense of measuring semantic “Moving the mountain: analysis of the effort required to transform
comparative anatomy into computable anatomy,” Database, vol. 2015,
similarity can be prohibitive for fine-grained annotations due p. bav040, 2015.
to an explosion in the number of classes required for reasoning [7] C. Pesquita, D. Faria, A. O. Falcao, P. Lord, and F. M. Couto, “Semantic
when annotations draw from multiple ontologies [14]. If the similarity in biomedical ontologies,” PLoS Computational Biology,
inclusion of qualities does not improve sensitivity, that opens vol. 5, no. 7, p. e1000443, 2009.
up the possibility of conducting fast, scalable on-the-fly web- [8] R. M. Othman, S. Deris, and R. M. Illias, “A genetic similarity algorithm
based semantic searches at coarser annotation levels. for searching the gene ontology terms and annotating anonymous
One of the contributions of this work is in introducing a protein sequences,” Journal of Biomedical Informatics, vol. 41, no. 1,
pp. 65–81, 2008.
framework for evaluating the statistical sensitivity of semantic
[9] J. Z. Wang, Z. Du, R. Payattakool, S. Y. Philip, and C.-F. Chen, “A new
similarity metrics. Nonetheless, the results reported here are method to measure the semantic similarity of go terms,” Bioinformatics,
specific to one particular model for the decay of similarity vol. 23, no. 10, pp. 1274–1281, 2007.
between two profiles, in which some portion of annotations [10] X. Wu, E. Pang, K. Lin, and Z.-M. Pei, “Improving the measurement
that match perfectly while others do not match at all. We of semantic similarity between gene ontology terms and gene products:
recognize a need to explore other models, especially ones insights from an edge-and ic-based hybrid method,” PloS One, vol. 8,
where pairs of annotations may match imperfectly. We also no. 5, p. e66745, 2013.
propose that other evaluation criteria should be examined [11] N. L. Washington, M. A. Haendel, C. J. Mungall, M. Ashburner,
M. Westerfield, and S. E. Lewis, “Linking human diseases to animal
to more fully understand the trade-offs involved in building models using ontology-based phenotype annotation,” PLoS Biology,
datasets with a particular level of annotation granularity. vol. 7, no. 11, p. e1000247, 2009.
[12] M. Mistry and P. Pavlidis, “Gene ontology term overlap as a measure
IV. ACKNOWLEDGEMENTS of gene functional similarity,” BMC Bioinformatics, vol. 9, no. 1, p.
327, 2008.
We thank W. Dahdul, T.A. Dececchi, N. Ibrahim and [13] H. Cui, W. Dahdul, A. T. Dececchi, N. Ibrahim, P. Mabee, J. P. Bal-
L. Jackson for curation of the original dataset, along hoff, and H. Gopalakrishnan, “Charaparser+eq: Performance evaluation
with the larger community of ontology contributors and without gold standard,” Proceedings of the Association for Information
data providers (http://phenoscape.org/wiki/Acknowledgments# Science and Technology, vol. 52, pp. 1–10, 2015.
Contributors), and useful feedback from P. Mabee, H. Lapp, [14] P. Manda, C. Mungall, J. P. Balhoff, H. Lapp, and T. Vision, “Investigat-
W. Dahdul, and other members of the Phenoscape team. This ing the importance of anatomical homology for cross-species phenotype
comparisons using semantic similarity,” Biocomputing 2016, pp. 132–
work was funded by the National Science Foundation (DBI- 143, 2016.
1062542).
R EFERENCES
[1] A. R. Deans, S. E. Lewis, E. Huala, S. S. Anzaldo, M. Ashburner,
J. P. Balhoff, D. C. Blackburn, J. A. Blake, J. G. Burleigh, B. Chanet,
L. D. Cooper, M. Courtot, S. Cssz, H. , W. Dahdul, S. Das, A. T.
Dececchi, A. Dettai, R. Diogo, R. E. Druzinsky, M. Dumontier, N. M.
Franz, F. Friedrich, G. V. Gkoutos, M. Haendel, L. J. Harmon, T. F.
Hayamizu, Y. He, H. M. Hines, N. Ibrahim, L. M. Jackson, P. Jaiswal,
C. James-Zorn, S. Khler, G. Lecointre, H. Lapp, C. J. Lawrence,
N. Le Novre, J. G. Lundberg, J. Macklin, A. R. Mast, P. E. Midford,
I. Mik, C. J. Mungall, A. Oellrich, D. Osumi-Sutherland, H. Parkinson,
M. J. Ramrez, S. Richter, P. N. Robinson, A. Ruttenberg, K. S.
Schulz, E. Segerdell, K. C. Seltmann, M. J. Sharkey, A. D. Smith,
B. Smith, C. D. Specht, B. Squires, R. W. Thacker, A. Thessen,