Semantic annotation using biomedical ontologies is required to enable data integration, interoperability, indexing and mining of biomedical data. When used to support semantic indexing the scoring and ranking of annotations become as important as provenance and metadata on the annotations themselves. In the biomedical domain, one broadly used service for annotations is the NCBO Annotator Web service, offered within the BioPortal platform and giving access to more than 350+ ontologies or terminologies. This paper presents a new scoring method for the NCBO Annotator allowing to rank the annotation results and enabling to use such scores for better indexing of the annotated data. By using a natural language processing-based term extraction measure, C-Value, we are able to enhance the original scoring algorithm which uses basic frequencies of the matches and in addition to positively discriminate multi-words term annotations. We show results obtained by comparing three different methods with a reference corpus of PubMed-MeSH manual annotations.
The large quantity of biomedical data available today requires the provision of
efficient tools to search, process, explore and query the data. These data are often
unstructured and are presented in different formats (database, documents, etc.), which prevents
an efficient integration and interoperability. To address these problems, the biomedical
community has turned to ontologies and terminologies to describe their data and turn
them into structured and formalized knowledge [
In this paper, we repport on the use of one of the mostly used biomedical annotation
tool: the NCBO Annotator offered offered within the BioPortal platform [
The NCBO Annotator Web service [
The NCBO Annotator Web service was released in 2009 and used by an increasing numbers of users since then (approx. 400 GB of data per year) and is also embedded in commercial platforms. The Annotator service returns annotations in JSON or XML and uses URIs as defined within BioPortal. Our contribution consists in ranking the annotations by post-processing them, without changing either the annotation themselves or the implementation of the service.
When scoring annotations, a number is assigned to an annotation to indicate its
importance. Higher scores reflect more important or relevant annotations. Typically,
methods will use context information [
In the latest versions of the Annotator, since BioPortal 4.0, end of 2013, the scoring
method has been removed from the implementation, thus transferring the task of
scoring to the users of the service, when all the annotations have been retrieved. In a previous
version of the web service, as released in 2009 [
Cell Carcinoma ... Basal Cell ...Basal Cell..." For each annotating concept, we show: the code, preferred name, matched terms, the matching type, the frequency (F) of this match and finally the annotation scores (S1). The results show that the concept T025 (cell) is the most represented in the text with a score of 96 (twice nested in the expression basal cell and four times nested in the expression Basal cell carcinoma). However, this concept is never present "alone" in the text and one would like to see the concept D002280 (basal cell carcinoma) higher in the ranking as it is one of the most representative concept of the text. Minimally, D002280 should be ranked above D002277 as it is an hyponym (more precise) of carcinoma mentioned in the text. Therefore, the results obtained with the OldScoreNcbo scoring are not very relevant to the needs and expectations of the user although it is true than the text is about the notion of cell.
During our study, we have more extensively experimented the OldScoreNcbo scoring method with different texts and ontologies and varying the hierarchy level. This led us to conclude that this method does not penalize enough the single-word matches within multi-word terms because, as the scoring method considers both full term matches and single word matches, it increases the weight of single word terms, which are also nested in longer terms.1 To address these limitation, we turned ourselves towards ap1 A few months after our study, in Sept. 2014, the Annotator was enriched by a new ’longestonly’ parameter which makes the service ignore shortest matches if a longer match (within Code T025
Preferred name
proaches proposed in the natural language processing community when doing term
extraction from text corpora.
Frequently, biomedical terminologies and ontologies contains several concepts whose
names are nested in names of other concepts. For instance, the word disease, preferred
name of the concept D004194 in MeSH is nested in the term periodontal disease,
preferred name of D010510. During the annotation process, in order to penalize the nested
concepts and to favor non-nested concepts we proposed to use the C-value measure [
C-value is an automatic term recognition method well known in the literature that
combines statistical and linguistic information for the extraction of multi-word and
nested terms. It often gets best precision results when used for automatic term
extraction especially in biomedical studies [
C-value(a) = (1) 8> log2 jaj > > > < >>> log2 jaj > : f (a)
if a is not nested f (a)
1 jP (Ta)j
X f (b) b2Ta ! otherwise
Where a is the candidate string, f (a) the frequency of occurrence of a in the corpus (the text to annotate in our case), jaj the number of words in a, Ta the set of terms that contain a and P (Ta) the number of terms in this set. In a nutshell, C-value either uses frequency of the term if the term is not included in other terms (first line), or decrease this frequency if the term appears in other terms, by using the frequency of those other terms (second line). For example, for a given text containing exactly four times the expression ’breast cancer’ and twice the expression ’cancer’: the C-value of the term breast cancer will be 6.33 which is higher than the C-value of cancer will be 2, while the frequency of terms are respectively 4 and 6.
In order to use the C-value measure to score annotations we had to overcome the problem that C-value operates at the term level where as annotations are related to concepts. We proposed to deal with this by applying the C-value measure on all the matched terms and summing the values of all the terms matched for a given concept in order to obtain the C-Value of that concept. In addition, to keep the advantages of the previous measure, taking into account the type of match, we used the log to weaken the effect of the OldScoreNcbo when calculating the new score. This new scoring method is referred after as ScoreNcboCvalue.
the same ontology) exists e.g., carcinoma matches would be ignored in front of basal cell carcinoma ones. Although, this new parameter fixes partially the problem by removing some matches it does not replace a proper scoring methods that allows to rank all the annotations. In addition, the problem remains when someone wants to get the maximum number of annotations for a given text and thus does not use the new longest-only parameter.
ScoreN cboCvalue(c) = 8 >< log(OldScoreN cbo(c)) >: log(OldScoreN cbo(c))
X C-value(t) if t2Tc
t2Tc otherwise (2)
Where c is an annotating concept, OldScoreN cbo(c) is the score obtained with the previous method for that concept, Tc is the set of matched terms with the concept c and C-value(t) is the C-value score of the term t in the annotated text. As an example, for the same previous text, Table 3 describes the annotations obtained with the ScoreNcboCvalue scoring method. The results show that all annotations generated from matches with the term basal cell carcinoma are now ranked first, followed by the annotations generated from matches with the term cell. Therefore, this new scoring method addresses the first problem identified in previous section by giving better score to the annotations obtained with non-nested matching terms.
However, Table 3 also illustrates than the annotation with T025 (cell) has been pushed after all the annotations generated with the term basal cell carcinoma including the annotations with high level concepts such as D009375 (neoplasms, glandular) or D009370 (neoplasms by histologic type) obtained thanks to the is-a hierarchy expansion. This could be a problem as those terms are more general than the term cell, which had indeed generated direct annotations with preferred name. Therefore, to improve the results of the ScoreNcboCvalue scoring method, we need to decrease the effect of the hierarchy semantic expansion. 3.3
To penalize the score of expanded annotations, we propose to ignore when calculating the C-value of a concept the C-value of hierarchical annotations. A new formula is proposed with ScoreNcboCvalue, where the set Tc refers then to the set of terms annotated with the concept c when ignoring hierarchical annotations (the C-value of hierarchical annotations is consider zero). This new scoring method is designated by ScoreNcboCvalueH.
The scores of annotations generated using the ScoreNcboCvalueH method are presented in Table 4. This scoring method allows us to score and rank efficiently the annotations of the example text. The most appropriate annotation, done with concept D002280, is now ranked first with a very big score compared to the rest of the annotations bellow. In the second pool of annotations, with score between 1 and 2, both the ranking and the score are satisfying as they favor first precise concepts with direct matches such as cell or carcinoma, while keeping after the annotations with more general concepts.
In this section, we evaluate the relevance of the three annotation-scoring methods described previously using a manually annotated corpus: PubMed citations manually annotated with MeSH terms by experts from the US National Library of Medicine. Our hypothesis is that such a MeSH terms should count among the first automatic annotations obtained with the automated method when processing the related title and abstract. In the following, we describe our general evaluation procedure (illustrated in Figure 3): Corpus building: We have first collected 1,250 citations in English from the PubMed2 database, previously annotated with the Medical Subject Heading (MeSH) terminology. For each article, we have a title, an abstract, and the referenced annotations (cf. Figure 4).
Automatic annotation: We have annotated the title and the abstract of each citation of the corpus using the Annotatorwith MeSH. During that step we have generated two sets of annotations for each citation according to the hierarchy level parameter of the Annotator either fixed to 0 (no is-a expansion) or 10 (expansion up to parentlevel-10).
Mesh code mapping: Because PubMed citations annotations do not use the Mesh codes (identifiers) but the term itself in their format (cf. Figure 4), we had to reconcile the term with its code using the NCBO BioPortal search service which allows for a given term to retrieve the different identifiers of a concept, including the ones also returned by the Annotator.
Scoring of annotations: During this step, we have scored the annotations retrieved from the previous step using the three scoring methods described in Section 3. Ranking of annotations: After scoring the annotations, we ranked them in ascending order. We grouped annotations with the same scores and assigned each group a sequential position starting with the group with the highest annotation score (cf.
Figure 5).
Normalization: Because the rank value of a given annotation (or group of annotations) will be different with each scoring methods, we had to normalize the ranking to compare them. 2 http://www.ncbi.nlm.nih.gov/pubmed Comparison with MeSH annotations: At this step we look up the manual annotations retrieved among all the automatic annotations returned by the Annotator and keep only the citations with a recall above 30% (i.e., citations for which at least 30% of manual annotations have been found by the Annotator). For each subset of manual annotations we then compute above two evaluation measures: (i) Average reference annotation rank by citation; (ii) Average reference annotation rank of the whole corpus. (3) (4) Average_citation_rankM (citationi) = 100
X scoreM (c) c2C jCj Where C is the set of the manual annotations of the PubMed citationi found among the automatic annotations of this article with scoring method M .
N
X Average_citation_rankM (citationi) Average_corpus_rankM = i=1 N
Where N is the total number of citations in the corpus. We compare the three scoring methods by calculating the average rank of the corpus for each scoring method. The lower the average rank is, the better, as it means that the reference annotations obtained better scores. Table 5 shows the results obtained with or without hierarchy expansion for each scoring methods.
We first note that the average corpus rank is lower (on results obtained with is-a hierarchy semantic expansion, independently of the scoring method. We also note that without hierarchy expansion ScoreNcboCvalue and ScoreNcboCvalueH obtained the same value, which is normal because the second measure was only affected hierarchy annotations. Finally, we note that with or without hierarchy expansion the ScoreNcboCvalueH method gets the best (lowest) rank which confirms the improvement of the scoring and ranking method. In addition, during our studies, we experimented with smaller recall values and obtained similar results.
This experiment has shown that our new scoring methods are more efficient than the old method used by the Annotator in terms of average corpus rank. Now, it is interesting to know in which cases this conclusion is true i.e., what types of annotations are ranked better. With the use of C-value, created to favour multi-term extraction, our assumption was that annotations made with multi-word terms will be ranked better. Therefore, we have performed basic statistics on the three following sets of annotations: Improved annotations: when the two new methods provide the best rank; Equal annotations: when the three methods give the same normalized rank; Not improved annotations: when the OldScoreNcbo method gives the best rank.
For each set, we estimate the percentage of annotations done directly with a term made of one, two, three, four or five words or indirectly with is-a semantic expansion. Table 6 shows the results obtained for each annotation sets. We notice that in both cases, more than 60% of annotations that are equal or not improved by our new methods are annotations done with a single word term match. In addition, between 70% and 90% of improved annotations are done with a multi-word term match. This confirms us the improvements brought by our new methods was mainly on annotations done with multiword terms. Also, it is important to notice that single word matches are not explicitly penalized by our method, but just pushed after multi-word matches. This means that 1 word 2 words 3 words 4 words 5 words 1 word 2 words 3 words 4 words 5 words 1 word 2 words 3 words 4 words 5 words 0 59 36 3 0 0 0 1 0 0 0 81 16 10 49 36 3 0 12 51 26 1 word 2 words 3 words 4 words 5 words 1 word 2 words 3 words 4 words 5 words 1 word
in the case of ontologies with mostly single world terms, our method will still behave
properly.
In this paper, we have presented our approach to improve the results of the NCBO
Annotator by ranking annotations according to their relevance. We have proposed two
new methods and compared them one another as well as with the original annotation
scoring provided in the first version of the Annotator (not available anymore since
BioPortal 4.0). The introduction of the C-value measure has led to significant
improvements in the scoring and ranking allowing to discriminate positively annotations made
with multi-word terms and penalizing hierarchical annotations. We have evaluated the
performance of these scoring methods using PubMed-MeSH annotations as a reference
corpus. The evaluation demonstrated that the two new methods are more efficient than
the old method of the Annotator in terms of average corpus rank. One limit of our
approach is due to the fact that the Annotator service cannot handle structured data for
which more context information (e.g., title, abstract, preconditions, etc.) will improve
the scoring of the annotations as explained in [
We are currently working to offer our new scoring methods to the community of users as an add-on when calling the NCBO Annotator. Indeed, an important advantage is that the scores can be computed only by processing the Annotator results and with no requirement to change the service implementation. As of today, the Java implementation of the methods is available as a JAR file offering the scoring functions taking as input the exact Annotator outputs and generating exactly the same XML or JSON outputs but completed and ranked with the scores. This JAR library is available on request. Our long term perspective, within the Semantic Indexing of French Biomedical Data Resources (SIFR) project (http://www.lirmm.fr/sifr) is to offer a service endpoint implementing several improvements (negation, disambiguation, new semantic expansion, new outputs formats, etc.) of the NCBO Annotator done with pre and post processing while still calling the Annotator service. 6
This work was supported in part by the French National Research Agency under JCJC program, grant ANR-12-JS02-01001, as well as by University of Montpellier, CNRS and the Computational Biology Institute (IBC) of Montpellier. We thank the National Center for Biomedical Ontology (NCBO) for latest information about the Annotator.