One of the long-standing challenges in natural language processing is uniquely identifying entities in text, which when performed accurately and with formal ontologies, supports efforts such as semantic search and question-answering. With the recent proliferation of comprehensive, formalized sources of knowledge (e.g., DBpedia, Freebase, OBO Foundry ontologies) and advancements in supportive Semantic Web technologies and services, leveraging such resources to address the entity disambiguation problem in the industry setting as “off the shelf” within natural language processing pipelines becomes a more viable proposition. In this paper, we evaluate this viability by building and evaluating an entity disambiguation pipeline founded on publicly available ontology services, namely those provided by the NCBO BioPortal. We chose BioPortal due to its current use as an ontology repository and provider of ontological services for the biomedical informatics community. To consider its usage outside the biomedical domain, and given our immediate project goal for facilitating semantic search over Earth science datasets for the DataONE project, we focus on the disambiguation of geographic entities. For this work, we leverage NCBO's Term service in conjunction with NCBO's entity disambiguation service, the Annotator, to demonstrate an enhancement of the Annotator service, through application of a vector space model representation of ontological entities and relationships to drive scoring improvements. This work ultimately provides a methodology and pipeline for improving publicly available ontology service-based entity disambiguation, demonstrated through an enhanced version of the NCBO Annotator service for geographic named entity disambiguation.
One of the long-standing challenges in natural language processing is uniquely identifying entities in text, which when performed accurately and with formal ontologies, supports efforts such as semantic search and question-answering. Semantic search would greatly facilitate our project, the Data Observation Network for Earth (DataONE), as it is aimed at limiting the excessive time and effort spent to discover, acquire, interpret, and use related data for biological, ecological, environmental and Earth science data.1 The DataONE metadata catalog is composed of content uploaded by participating research institutions, that includes a keywords field populated by data managers, where these keywords and keyword-“worthy” terms in the scientific abstracts are not yet linked to domain knowledge in a formal way to aid discovery, besides what exist as disparate controlled vocabularies. Clearly here, the use of publicly available formal ontologies to disambiguate terms of relevance for search provides advanced capabilities for precise search and a “free extension” to their existing vocabularies that does not require manual development.
With the recent proliferation of comprehensive, formalized sources of knowledge (e.g., DBpedia,2 Freebase,3 OBO Foundry ontologies4) and advancements in supportive Semantic Web technologies and services, leveraging these resources to address the entity disambiguation problem in the industry setting as “off the shelf” within natural language processing pipelines becomes a more viable proposition. In this paper, we evaluate this viability by building and evaluating a novel entity disambiguation pipeline founded on publicly available ontology services, namely those provided by the NCBO BioPortal. We chose BioPortal due to its central role as a ontology repository and provider of ontological services for a given community, that being biomedical informatics. To consider its usage outside the biomedical domain, and given our immediate project goal for facilitating semantic search over Earth science datasets for the DataONE project, we focus on the disambiguation of geographic entities, using the Gazetteer Ontology (GAZ).
In this work, we use NCBO’s Term service in unison with NCBO’s own entity disambiguation service, the NCBO Annotator, to demonstrate an enhancement over the Annotator service, using a vector space representation of ontological entities and relationships to drive scoring improvements. Fittingly then, we evaluate our results against the NCBO Annotator, and apply the TopN scoring method, since both systems are suited to provide inputs to a semi-automated semantic-mapping workbench environment. Ultimately, this work evaluates whether our techniques improve on TopN mapping accuracy of the Annotator service, including if introducing all domain-level relationships into the vector space provides additional discriminatory power over just those relationships considered “broader”, while at the same time provides insights into the process of using and augmenting publicly available ontology-service driven web services for entity disambiguation.
Our overall approach extracts named entity labels from natural language text, and where applicable, maps each to a resource of an ontology that best represents and disambiguates its meaning. The set of entities that can be disambiguated within our pipeline is flexible, to disambiguate what in the formal ontology community is referred to as particulars or concepts (or types), and what in the Web Ontology Language (OWL) considers individuals or classes,5 that the Semantic Web community at large via the Resource Description Framework (RDF) refers to simply as resources.6 Thus we describe our pipeline at the abstraction of “resource”, and where appropriate for its current application we describe how it functions for geographic named entities disambiguation. In the following sections we describe related work, background on the topic-based vector space model algorithm we apply, and our methodology and pipeline that utilizes these algorithms and services. In Section 4 we evaluate results using the TopN scoring metric, comparing against the existing annotation service using an initial, hand-curated gold standard. In Section 5 we present qualitative findings that resulted from application of our the pipeline, and in sections 6 and 7 we discuss future work and conclusions.
In this section we formally present the definition and implementation of an eTVSM
model [
b ak; : : : ; å å k=0 rn2P(r;k)
+ ak ;~r = 1 ~
t
~
ktk
Each resource in the graph representation of an ontology is assigned an index from 1
to n, which means that each resource vector has size n and there are n resource vectors.
The vector for resource n is calculated by assigning an exponentially declining weight
to connected resources and summing the total weights for each resource. For example,
resource i is assigned a weight of 1, all resources one node away from i adds a weight of
0.5, resources two nodes away from i add a weight of 0.25, etc. The sum of the weights
are normalized. In this way we assign a resource vector to each candidate resource. The
constant a is an exponential decay which is used to determine how important resources
are distant from the candidate. [
Next, an interpretation vector is constructed from an interpretation, i, which is a unique mapping of a named entity label to a resource, which we multiply by an ambiguity weight: In order to reduce the effects of ambiguous mappings (i.e., which map labels to multiple resources), we weigh each interpretation vector by an ambiguity weight g(i), defined as: ~i =~r g(i) g(i) =
1 j j : j 2 I(k); k 2 K(i)j (1) (2) (3) ~td = å wd;~ii i2I (4) The ambiguity weight g(i) forces labels which are mapped to many resources to have less weight toward disambiguating other named entity labels. Here, K(i) is the set of all labels for interpretation i, and I(K) is the set of all interpretations derived from label k. We represent the document as the collection of all resource mappings by summing the weighted interpretation vectors: Each interpretation vector is weighted by multiplying the frequency at which the named entity label for interpretation i is found in the extracted named entities from document d. The cosine similarity metric is used to compare individual interpretation vectors for the document against the document vector, ~td . Incorrect interpretations will not significantly affect the document vector and result in a low cosine similarity score, whereas interpretations that are more similar to other interpretations will result in a higher cosine similarity.
We apply a three-phase pipeline for executing our entity disambiguation approach, including NER, resource mapping, and eTVSM scoring. We describe each phase as a black-box system, with inputs and outputs which link the phases together. Our overall pipeline takes as input an unstructured text document in the form of a science publication abstract from DataONE outputs a ranked list of candidate resources.
The NER phase of our pipeline extracts named entities embedded in unstructured
text using the Natural Language Tool Kit (NLTK) [
The resource-mapping phase of our pipeline processes a list of named entity labels and returns a set of named entity label to resource mappings and an ontology portion (i.e., a set of statements) describing each matched resource. In the context of geospatial named entity recognition, there are multiple resources which are referred to by the same preferred label annotation. For example, in the United States, the city of Springfield can refer to greater than 40 different cities located in multiple states. Also some states, such as Wisconsin, have multiple cities named Springfield. Therefore, the purpose of the resource-mapping phase is to map each named entity label to those resources which define a unique instance of that particular location with that name. For every named entity label extracted from the NER phase, we query the NCBO Annotator Service for a list of candidates, that is, resources which could potentially define that particular named entity within an ontology.
Next, we obtain statements about the resources, including annotations as well as object property and subclass statements, using the NCBO Term Service, by supplying a resource URI and ontology identifier as parameters to the service.7 The object property and subclass-based statements are filtered according to a preselected set of properties, where the objects of these statement are recursively submitted to the Term Service. In this way, we perform a breadth-first search along a set of properties until the final resource in the path is reached. This phase concludes when the statements are collected for each candidate resource.
The eTVSM scoring phase scores each entity label-to-resource mapping in a given
document by first generating a TVSM of the obtained ontology graph, and then encoding
the resource mappings into an eTVSM. We use the cosine similarity metric to determine
how strongly one candidate resource is related to all candidate resources for a given
document [
In this section we introduce a small hand-annotated dataset which we use as a gold standard for comparing our result against the existing NCBO Annotator Service. We score output from the NCBO Annotator Service and from our pipeline using the TopN scoring metric. We demonstrate how our approach contributes an enhancement to the NCBO Annotator Service for geographic named entity disambiguation using the eTVSM algorithm. The process for which we created our gold standard is described below.
First, we selected scientific paper abstracts by processing each abstract through our pipeline and selecting those which have at least one named entity label that has greater than 10 candidate concept mappings. This decision was due to a lack of preexisting gold standard data and our desire to disambiguate highly ambiguous named entity labels. For each named entity label we hand-annotated it with a concept in the Gazetteer Ontology which correctly identifies it. The result is 24 unique named entities labels, which were extracted using the NER phase of our pipeline. For these 24 named entity labels, there are 18 unique candidate concepts contained in the GAZ ontology, which we discovered by querying the annotator service with our named entity labels. The remaining six named entity labels were either too generic for us to assign a unique concept, or the correct concepts are not contained in the most current version of GAZ; in order to get a meaningful comparison of our pipeline against the NCBO Annotator Service we excluded these six labels for our evaluation. The 18 named entity labels were hand-mapped to concepts in GAZ, composing our gold standard dataset which we use in our evaluation.
In the setting of a semantic-mapping workbench, when a prospective curator inspects the list of candidate concepts for a set of named entities extracted from a document, the most accurate or correct concept will ideally appear at or near the top of a ranked list of candidate concepts. In fact, the Annotator website presents candidate concepts in this ranked manner. To be fair, there are just two levels of ranking produced by the Annotator 7
We describe the atoms of the ontology portion extraction in terms of statements, to maintain our abstraction at the level of resources. In the case of named entity disambiguation these statements are a mix of assertions and axioms, and in the case of conceptual entities these statements are primarily axioms.
TopN =
N
å ack j
j=1
N
å ai
i=1 c
(5)
service, determined by whether a match was made on the preferred or alternate term,
while our approach uses a much more granular similarity score metric. Still, we evaluate
our augmented version of the annotator alongside the publicly available Annotator
Service in order to demonstrate improvement upon the existing service. Therefore we
evaluate the two approaches using the position of the correct concept returned in a ranked list
of concept-mapping scores. To do this, we use the TopN scoring metric [
For the evaluation of our pipeline, we construct our vector space using two methods. The first construction uses located in, while the second uses all (13) relationships contained in the GAZ ontology. This comparison provides some evidence that including relationships beyond those considered generalizations improves scoring. Table 189 shows the comparison between the NCBO Annotator Service and our pipeline as constructed using the two methods. These results demonstrate that the application of the eTVSM-scoring phase outperforms the Annotator Service for disambiguating geographic named entities. The results demonstrate that a quantitative approach for disambiguation which measures and ranks the strength of each concept mapping outperforms an approach which relies on simple lexical matching.
In this section we present some qualitative insights of applying our pipeline to a sample scientific publication abstract. Figure 110 shows a graph output of our pipeline as applied to a sample scientific abstract. In this run we included all geographic relationships from GAZ during the concept-mapping phase.
We confirmed that our ontology-based resource mapping approach successfully down ranks irrelevant resources and thus outperforms purely lexical-based resource map8
A version of this paper with figures included is available at http://tw.rpi.edu/web/doc/ImprovingOntologyServiceDrivenEntityDisambiguation/. https://www.flickr.com/photos/127739444@N02/15066654769/ https://www.flickr.com/photos/127739444@N02/15230553716/ pings. For instance, where only lexical matching algorithms are used (e..g., the current Annotator Service), the entity label ‘Oregon’ receives equal ranking for “State of Oregon (GAZ:00002515)” as resources “Oregon (GAZ:22225751)” and “Oregon (GAZ:00084619)” (cities in Michigan and Illinois). Our approach leverages fine-grained geographic relationships and considers relationships with other mapped named entities mentioned in the same abstract (e.g., Deer Creek, Josephine County), so that the similarity score of the mismatches for ‘Oregon’ is significantly lower than the correct one.
We also identified improvements to the scoring of correct resources after applying additional relationships in the resource-mapping phase of our pipeline, beyond those considered generalizations (e.g., located in). For example, for the abstract of the fourth study of our gold standard, the named entity label ‘Cumberland River’ was mapped to three distinct candidates. When only the located in relationship is applied, the two incorrect resources received a score while the correct resource (GAZ:00150754) did not (see left side of Table 211); however, when applying all geographic relationships, the correct resource accurately received the highest score. This is due to relationships to other resources, via the inclusion of additional kinds of geographic relationships, that have been mapped to other named entity labels extracted from the abstract (shown in Figure 212).
Finally, we learned that our pipeline improves the quality of ontologies available through Bioportal, by facilitating curators in the process of identifying and reporting existing gaps. For example, in an abstract that mentions the Deer Creek Field Station and Educational Center of the state of Oregon, the named entity label ‘Deer Creek’ returned 29 unique resources labeled as ‘Deer Creek’, none of which were the correct one. We informed the GAZ team, who quickly created the resource and appropriate statements, increasing coverage of the ontology. Figure 1 illustrates the results of the pipeline after the newly added resource “Deer Creek (GAZ:00633440)” was included, which became the highest ranking candidate resource for ‘Deer Creek’ due to relationships to “Josephine County” and “State of Oregon”.
In this section we discuss recent work that applies ontologies to the named entity
recognition (NER) problem, including that which the current work uses and builds upon:
the NCBO Annotator and enhanced topic-based vector space modeling (eTVSM).
Researchers at the BBC experimented with eTVSMs [
The NCBO BioPortal project supports efforts to linking unstructured text to
ontologies through publicly accessible services for leveraging community based ontologies [
Aside from the BBC and NCBO efforts, there exists extensive previous research
in the area of entity disambiguation leveraging ontology or more generally, linked data
sources. Alexopoulos et al. [
Our work differs from these in that we focus on the practicality of using NCBO Bioportal and its APIs as an ”off the shelf” resource for applying eTVSM for semantic disambiguation within an NLP pipeline. BioPortal is of particular interest because the ontologies registered with it include many that are developed by expert curators. The benefit of our approach is that the more explicit relationships and to what resources they relate are used for disambiguation and subsequently for fine-grained semantic search capabilities. What results from our work is an enhanced version of the NCBO Annotator for geographic entity disambiguation. Due to the Annotator’s wide usage, it provides immediate utility to the community upon release.
In future work, we will expand our gold standard to help determine if they further validate our results. Since it is a time-intensive process, we will seek external resources for performing the work, such as Mechanical Turk.19 We will also leverage such resources for tagging corpora for geographic named entities, which can be used for statistically training the NLP tokenizer that we employ in the pipeline.
Our methodology and pipeline lays a foundation for widening its use to conceptual entities and other types of named entities. Therefore, in future work we will evaluate how well, in practice, that our results are generalizable for disambiguating concepts and named entities for other domains, such as biomedical. On the side of named entities, one requirement is to request the NCBO to add additional ontologies that are specific to individuals, similar to the crowdsourced content available via Dbpedia.
For concept-based disambiguation, we lose the immediate benefit of the NLTK named entity recognizer, but which is mitigated when corpora tagging is carried out for the concept domain of interest, via Mechanical Turk or use of some other resources (e.g. PubMed). While there exists a wide range of biomedical ontologies available that cover similar sets of concepts, we will incorporate and test publicly available mappings between NCBO-registered ontologies, though performing the mapping task itself falls out of our scope. For the ontology extraction task of the resource-mapping phase, the mechanism for obtaining axioms at the class level instead of assertions at the instance level remains the same via the NCBO Term service.
In cases where concept mappings are not available, selecting the ontology to use that provides the best coverage and overall representation becomes more critical, as the eTVSM approach requires the selection of one ontology. This selection should be an automated process, as within the context of an annotation software tool for semi-automated mapping, it reduces burden on the annotator, enabling them to focus on finding the most accurate concept match in a ranked list of candidates. Therefore, in future work we will leverage a domain classifier for selecting the most suitable ontology for disambiguation.
To further support annotation software tools that leverage our pipeline, we will make modifications to capture the statements in RDF and/or OWL for ease of rendering in graph form; currently we are applying the XML-based results from the NCBO services in a non-RDF graph representation for processing into the vector models. We anticipate that, generally, the graph representations (as shown in Figure 1), when presented, will provide a curator context and visual justification of the ranking scores. Finally, at the time of this writing the NCBO ontology service for the version of BioPortal being used is deprecated, therefore we are working to port our code to leverage the latest version prior to making it publicly available.
To help address the challenge of using publicly available ontology services for entity disambiguation, in this paper we 1) provide an enhanced version of the NCBO Annotator service for geographic named entity through novel application of vector space modelbased disambiguation in concert with existing NCBO Term and Annotator services; 2)
https://www.mturk.com/ demonstrate that using the available fine-grained relationships in an expert-curated ontology improves disambiguation; and 3) provide insights into the process of using publicly available ontology-service driven web services and expert-curated domain ontologies for entity disambiguation and organically improving upon those services.
In support of future semantic mapping workbench applications, this approach provides a ranked list of results using quantitative scoring methods to disambiguate named entities. In Section 4 we evaluated the performance of our pipeline against the Annotator Service using the TopN scoring metric and demonstrated how in the context of a manual curation workbench, our pipeline provides benefit by reducing the time a curator would spend looking for the concept that correctly matched an extracted named entity label. To further demonstrate its value as an enhanced version of the NCBO BioPortal Annotator Service, in Section 5 we presented some insights resulting from the pipeline being applied to Earth and environmental science abstracts, leveraging domain-level relationships available in GAZ to power the disambiguation process.
Our approach also helps aid curators to create gold standard datasets as training data for performing entity disambiguation using statistical machine learning methods. For curators who manage metadata like those within the DataONE project, the output from our pipeline could be added as metadata, improving metadata quality by showing how named entities in the text are related, which can we used to enhance search capabilities. Our approach provides benefits over using methods that do not rely on ontologies for the disambiguation task, or when ontologies with minimal semantics (e.g., broader relationship in SKOS) are used, as subsequent search interface capabilities will have better precision.