This paper focuses on the use of corpus-based machine learning (ML) methods for fine-grained semantic annotation of text. The state of the art in semantic annotation in Life Science as in other technical and scientific domains, takes advantage of recent breakthroughs in the development of natural language processing (NLP) platforms. The resources required to run such platforms include named entity dictionaries, terminologies, grammars and ontologies. The demand for domain-specific, comprehensive and low cost resources led to the intensive use of ML methods. The precise specification of the ML task goal and target knowledge, and the adequate normalization of the training corpus representation can notably increase the quality of the acquired knowledge. We argue in this paper that integrated ML-NLP architectures facilitate such specifications. We illustrate our demonstration with four representative NLP tasks that are part of the BioAlvis semantic annotation platform. Their impact on the quality of the semantic annotation is qualified through the evaluation of an IR application in Bacteriology.
Despite the growing number of available structured databases dedicated to
biomedicine, a large part of the domain knowledge is only available in documents in
natural language. Besides, several services centralize publications in Health and Life
Sciences. The main one, Entrez PubMed (NCBI), references over 16 millions of
papers [
Life and Health sciences are recognized as critical knowledge-intensive domains
for the Semantic Web [
Automatic semantic annotation supplies a meaningful structure to free texts expressed in natural language with the purpose of allowing machine processing. In the Semantic Web framework, the semantic annotation consists in an interpretation of the text supported by an ontology, i.e. the assignment of concepts and relations of an ontology to fragments of text. The extent of the annotated text fragments is fairly variable depending on the target application. IE and IR target specific bits of information contained in short fragments of text, i.e. terms, words and multi-word units. Fig. 1 shows an example of the semantic annotation of a sentence from a scientific article on Molecular Genetics. The word GerE denotes a protein and the word sigK denotes a gene. The negative interaction concept is supported by the inhibit verb. GerE (resp. sigK) and the verb inhibit are instantiations of the arguments of the ontology relation agent (resp. target) between the protein and the concept negative interaction.
As an illustration of the production and exploitation of a semantic annotation in the context of IR, we present the BioAlvis variant of the Alvis framework focused on bacteriology that performs as follows. The annotation pipeline enriches documents with fine-grained semantic annotations acquired through the successive application of NLP tools. The result is passed to the indexing component and exploited by the semantic search engine. The IR service normalizes the user queries in the same way as the documents: words are lemmatized, terms and named entities (NE) are replaced by their canonical forms and the concepts are replaced by their paths from the ontology root. This strategy differs from usual query expansion that consists in replacing each query term with the set of synonyms and sub-concepts. The Alvis method indeed drastically reduces the complexity queries and makes its interpretation legible for the user. The user can also directly benefit from the annotation of concepts by performing ontology-based facet refinement through a rich Web user interface.
Software platforms for text corpus annotation integrate a common range of
linguistic processes into pipelines, typically: tokenization, word and sentence
segmentation, named entity and term tagging, part-of-speech tagging, syntactic
parsing and semantic concept and relation annotation. Each process relies on
linguistic resources relevant to the target domain, which requires important
acquisition efforts. Most platforms do not specifically include automatic knowledge
acquisition facilities (e.g. Luxid®, MedScan®, AKS2®, InGenuity®) or in a limited
way (e.g. Luxid I2E® for NER), although corpus-based Machine Learning provides
an attractive alternative to the manual acquisition of such resources. Technically, a
single annotation pipeline can process documents for application purposes as well as
for preparing training corpora with the intent of acquiring new linguistic resources.
However, in most implementations, this virtuous feedback does not translate into
close ML and NLP software integration. The input of the ML is usually computed by
a subpart of the NLP pipeline but the output is not directly usable by subsequent NLP
components. This is the case in Gate / Amilcare [
We claim that semantic annotation can greatly benefit from a full integration of the ML components that feed the knowledge bases. Beyond format homogeneity, close integration compels the architecture designer to specify the respective roles of each NLP component involved in the semantic annotation process, to identify precisely all types of knowledge along with their interdependencies, and the target knowledge to be acquired by each ML component.
Breaking down the semantic annotation task into well-identified NLP elementary
steps has a positive effect on the production quality of the associated ML component.
Relevant regularities are more easily identified by the ML system and human
annotation of training examples is easier and of higher quality when it concerns a
singled out knowledge type. For example, in formal knowledge representation
frameworks, the tagging of semantic types and the tagging of properties are two
distinct steps. In the phrase, “mouse synaptophysin gene”, the annotation of
synaptophysin as an object of type gene is handled separately from the annotation of
its property belongs to the species mouse. The knowledge acquisition goals for the
recognition of gene names and their related species must be achieved by at least two
distinct ML tasks applied to two different training corpora. The increased
homogeneity of corpora that results from normalization reduces the number of
examples to annotate manually. Unfortunately, many knowledge acquisition
approaches to NER do not follow this principle [
Moreover, the clarification of dependencies among the different types of knowledge provides a basis for increasing knowledge modularity and reducing annotation inconsistencies. Operationally, the dependencies between knowledge types impose a constrained order of linguistic/semantic/acquisition processing steps that should be made explicit. In a structured modular view of the linguistic knowledge base, higher level knowledge should encapsulate lower level knowledge. Then, in order to learn a given target knowledge K, the training example representation should be based on the knowledge on which K depends and no any lower level knowledge. For instance, for the sake of modularity, relation recognition rules should not be learned from shallow clues such as punctuation marks. Previous NLP steps should have interpreted the punctuation marks into relevant information such as sentence ends (sentence segmentation) or abbreviations (named-entity normalization).
Hereafter, we present the results obtained by applying these principles to the development and the integration of knowledge acquisition facilities into the Alvis platform. We focus on the acquisition of critical resources that are required by semantic annotation, with respect to the variety of learnable linguistic knowledge, i.e. named entities, terms, concepts and relations. We demonstrate their learnability (section 3) and the benefit of fine-grained semantic annotation in terms of quality and density of annotations for a given domain and application: IR in Biology (section 4).
The Alvis annotation/acquisition pipeline (A3P in the following) has been
developed within the Alvis project [
The XML annotation format relies on a layered representation where each layer
gathers the annotations from a single type of knowledge, in a similar way as described
in [
A3P bootstraps by providing an annotated corpus for the acquisition of the knowledge for the next components in the pipeline sequence. As shown in Fig. 2, the linguistic analysis modules are fed by knowledge bases (drums). Their acquisition is achieved by an array of corpus-based acquisition tools involving ML methods.
The next subsections describe four representative acquisition tasks of BioAlvis, their target knowledge, the example representation and the principles of the integration of the learned knowledge into the knowledge bases. Most of the learning results described in the following were obtained from a representative training corpus in bacteriology of 2,397 scientific paper references from MedLine referred to as the Bacillus corpus designed in 2001. It is the result of the following query to PubMed: “Bacillus subtillis AND (transcription OR promoter OR sigma factor)”.
The term named entity usually designates proper names associated to an
ontological category or semantic type (e.g. place, person). The proper names are rigid
designators that denote a referential entity in an unambiguous way [
BioAlvis NER component, RenBio focuses on protein/gene and species recognition. It achieves NE tagging by GenBank-based dictionary mapping and by the application of disambiguation and variation rules. The disambiguation rules specify what contexts are required for each type of named entity. In parallel, variant dictionaries and variation rules in the form of hand-crafted regular expressions deal with common typographic alterations. Rules for disambiguation and recognition of new entities are automatically acquired by supervised machine learning from a reference training corpus. For example, the simple rule,
A word, followed by the word protein, 4 letters long, starting and ending with an upper-case letter, is a protein name. applied to the text, “The GerE protein inhibits transcription” assigns GerE to the protein category, even if GerE is not in the protein dictionary.
The linguistic features of the training examples are computed by segmentation, lemmatization and typographic analysis (e.g. length, case, presence of symbols and digits, co-occurring words) of the training corpus performed by the annotation pipeline. The annotation of positive examples is done by first mapping the NE dictionary and then its manual correction by domain experts. Negative examples are automatically generated under the closed-world assumption.
The RenBio rules for gene and protein name annotation in BioAlvis were learnt
from the Bacillus corpus. The dictionary mapping on the training corpus tagged 7,185
occurrences. 10 biologists analyzed, corrected and completed the tagging. They found
12% false positives due to ambiguities and 12% false negatives due to new names.
We applied the C4.5 algorithm of induction of decision trees (J48 WEKA library
version [
We claim that the example representation features and accurate specifications of the learning goal permitted higher quality of the training examples, thus improving the conditions for the learning algorithms. The good results were not due to any breakthrough in ML since we apply a regular well-known algorithm.
On the one hand, the automatic linguistic pre-processing by BioAlvis of the examples has contributed to drastically reduce the dimension of the example description space and to remove potential sources of errors. Moreover in order to discriminate between NE and non-NE by their context, we picked the most relevant trigger words by feature selection. For instance, words like gene, operon or transcription are more likely present around gene names than any other word.
On the other hand, the learning goal was specified according to the role of the NER
in the semantic annotation process. This strongly determines the guidelines for the
manual annotation of the training examples by experts. Our strict annotation
guidelines address several phenomena that could hinder the quality of the annotation.
The principles are as follows: NE annotation should be restricted to single entities for
learnability and knowledge modeling reasons, it should exclude terms that denote
general semantic categories and properties and the entity span should exclude the
description of entity qualifiers (e.g. in “recA gene”, only “recA” is annotated). The
detailed description of the guidelines can be found in [
Our experiments demonstrate that the combination of the appropriate
normalization of the data with the consistent annotation of training examples by
experts improve machine learning performance in terms of precision, recall and size
of training sample (see [
The BioAlvis term analysis component identifies the phrases that represent
semantic units. These are single or multi-word nominal or verbal expressions that
refer to specific domain concepts (e.g. plant pest). The term analysis module achieves
term recognition and normalization, which consists in tagging the term with its
canonical form. Semantic ambiguities are processed afterwards by the semantic
typing module (section 3.5), while inflections are handled beforehand by the
lemmatization module. Similarly to NE, off-the-shelf lists of terms are not sufficient
to annotate documents because terms may be ambiguous and subject to variation.
Moreover, in scientific and technical domains, terminologies are generally incomplete
with respect to the specific application needs [
BioAlvis includes two acquisition modules for enriching the terminology with new
terms and variants from a training corpus. The term acquisition component is the
YaTeA term extractor [
Variation spectrum is much larger for terms than for NE. In addition to minor
graphical variations, it includes morpho-syntactic alterations that can deeply modify
the form of the term (e.g. plant pest, pests on plants and pests that attack plants).
Term normalization involves complex linguistic and domain knowledge encoded in
variation rules. BioAlvis integrates FASTR [
Applied to the Bacillus corpus, YaTeA extracted 6,699 candidate terms occurring at least twice, among which 3,560 were validated by a group of 3 experts in terminology and biology, yielding 52% precision. The recall evaluated on a gold standard subset of Bacillus was 67%. FASTR then extracted 2,335 variants. The validation by two experts was done in a few days and resulted in 676 synonym sets and 1,569 hyponyms. Additionally, from 715 MeSH terms found to occur in the Bacillus corpus, FASTR identified 1,899 new variants among which 397 hyponyms and 117 synonyms.
Such methods, when integrated in a pipeline, appear to be very competitive compared to manual acquisition. The approach offers exhaustiveness regarding to a corpus that is a clear advantage for knowledge-based application.
Semantic typing relates concepts from the ontology to semantic units in the text
after their identification by the term and NE recognition components. The ontology
concepts are organized into hierarchies. Semantic typing annotates the semantic units
with the whole concept path to the hierarchy root without any a priori assumption on
the concept level relevance. In A3P, the ontology-lexicon relation is explicit: the
concepts of the ontology are represented at the lexical level by the canonical forms of
terms and named entities. In case a given semantic unit is polysemic, disambiguation
rules select the right concept in the ontology with respect to its syntactic context in
the document; BioLG [
The acquisition of concept hierarchies and disambiguation rules is supported by the
ML hierarchical conceptual clustering tool Asium [
A large part of the potential learning errors is avoided upstream by choosing an
appropriate representation of the training examples. Terminology and NE
normalization significantly improve the quality of the learned classes by increasing
the homogeneity of the training data. It also decreases the number of parsing errors
and reduces the computation time, since the parser avoids computing dependencies
inside the terms as detailed in [
Although the concepts are validated along their construction by Asium, they cannot be integrated as such in the ontology. Their structure does not necessarily represent the model needed for the application and may require validation and revision. The alignment between learned ontology and existing ones also remains a critical problem. The modeling strategy we adopted for the development of BacteriOntology is based on an ontology skeleton crafted by hand by biologist experts and computer scientists from MIG-INRA laboratory. The hierarchical model results from the integration of several existing resources: (1) the highest levels of the ontologies and thesauri GO and MeSH; (2) relevant domain-specific information resources (Riley and Subtilist function classifications and NCBI species taxonomy); (3) concepts denoted by the 300 most frequent terms (section 3.2.2) acquired from our corpus. Asium results were then used to extend this core ontology and populate its classes. The current version of the resulting BacteriOntology defines 5,888 concepts, structured into 6 generality levels (excluding the extremely deep species hierarchy). The quality of BacteriOntology acquired with Asium support was globally evaluated through IR (section 4).
Domain specific relations are usually more difficult to identify in the text than
concepts because they are less directly supported by contiguous text fragments.
BioAlvis annotation of relations focuses on gene interaction and relies on relation
extraction rules. For a given relation, the rules check the type of the semantic units in
the ontology in order to spot candidate relation arguments, and the type of the
syntactic dependencies between them. For instance, in the text: GerE inhibits the
expression of sigK, the gene interaction between the protein agent GerE and the target
gene sigK is identified in the simplest case, by the rule expressed in first-order logic:
gene_interaction (X,Z) :- type(X,Protein), subject(X,Y), type(Y,Interaction_action),
obj(Z,Y), type(Z, Gene).
where Protein, Interaction_Action and Gene are ontology concepts, and obj and
subject are syntactic dependencies. Many complex gene interaction cases are handled
with the same method including those involving regulon membership and promoter
binding (detailed method in [
The LLL dataset on genic interaction [
We have designed the BioAlvis version of Alvis for the evaluation of ML-based
semantic annotation benefit and the delivery of knowledge-based application to
biologists (e.g. IR). This section reports on the experimental evaluation of the
BioAlvis semantic annotation for semantic search and its comparison to other
indexing and search models. We characterize Alvis search as semantic in the sense
that it automatically interprets the meaning of the query with respect to the
terminology and the ontology: Alvis searches for more specific and variant terms and
it assists query refinement by ontology and terminology navigation (see [
The reported experiments concern the adaptation of enterobacteria to changes in their environment. Enterobacteriaceae is a large family of bacteria of the intestine, including many human pathogens, like the well-known Salmonella that causes inflammation of the intestine (Gastroenteritis). Their virulence is due to their capacity to survive and grow in hostile environment conditions imposed by their hosts (acidity, high temperature or oxidative stress induced by iron starvation and superoxide radicals). Part of these conditions is due to the response of the host organisms to pathogen infection. The deep understanding of the bacteria response mechanisms at a molecular level to these stress factors is a key point toward the design of more efficient drugs. The goal of the search is to find descriptions of pathogen reactions and was translated into the following query: enterobacteria stress genome component.
In order to test BioAlvis, the Bacteriology document collection was built by first querying PubMed with all bacterial genera names from the GenBank taxonomy. Then we used a Bayesian filter to exclude documents that were not bacteriology stricto sensu. The result is a medium-size corpus containing 322,982 references of 70 words on average. This corpus was processed by BioAlvis in 60 hours on a cluster of 20 processors. The resulting semantic annotation was indexed and supplied to the Alvis search engine. Table 2 summarizes the main figures of the acquired linguistic resources (as described in section 3) and tagging features.
The annotation is dense due to the type of documents and the corpus-based strategy of the lexicon acquisition. For instance, the BioCreative II corpus contains on average 4.6 gene or protein names per document while there are 6 in our corpus.
Google and Google Scholar index very large collections. Google references around 24 billions of web pages. The size of Google Scholar is estimated at more than one billion references. Both systems perform simple stemming on documents and queries. Our hypothesis is that in specific domains, they will (1) retrieve more incorrect results compared to semantic search, because they do not disambiguate words; (2) miss relevant documents by not exploiting synonymy and related terms.
In Entrez PubMed, each indexed reference is manually assigned a set of terms representative of the document topic from the MeSH thesaurus. The manual annotation avoids ambiguities in document indexing but is quite expensive to maintain since it requires highly-trained experts who read the full-text articles. Entrez PubMed searches query terms in the full-text without any linguistic analysis as well as in the MeSH term index by expanding the query with synonyms and more specific terms according to the MeSH thesaurus. In all cases, the resolution of query ambiguity is postponed to query refinement by the user.
To illustrate the strategy of BioAlvis, we detail how the example query enterobacteria stress genome component is processed: the words are lemmatized; the recognized semantic units are normalized and assigned to the BacteriOntology concepts that belong to taxonomies: enterobacteria, stress and genome component. BioAlvis expands the search to documents where sub-concepts of the query term occur. For instance, the taxonomic group of enterobacteria contains Escherichia coli and Salmonella enterica among hundreds of other bacteria species. In the same way stress defines 17 different types of stress such as heat-shock and phosphate starvation. Again, genome component represents 62 different sub-concepts (e.g. operon and promoter). Each of these concepts references its variants and synonyms. For instance, heat-shock is synonym of temperature upshift, thermal upshock, and temperature upshock according to our terminology. Additionally, query lemmatization allows BioAlvis to search regardless of word inflections and derivations (stress / stressing / stresses). The interface displays the detail of the interpretation so that the result is understandable.
As we could benefit from Bacteriology expert analysis, we opted for a qualitative
evaluation of our system. Beyond rough figures, a comparative study of the answer
sets has characterized the missing and irrelevant documents retrieved by each service.
More complete results can be found in [
The very large answer set of Google (245,000) was expected because of the document collection size and the generality of the query. Google and Google Scholar search for the stemmed query words in the documents. As no semantics is used, all documents with sub-concepts of the query words were missed. We tested the query with a replacement of genome component by (gene OR promoter) that are two productive sub-concepts and found that 50 % more documents were retrieved. As Google and Google Scholar make use of stemming, they find 8 times more documents than with exact matches. For instance, documents with enterobacterial were found thanks to stemming.
Entrez Pubmed expands queries in a similar way as BioAlvis by following MeSH relations top-down. The term enterobacteria is expanded into tens of subconcepts as well as genome component. The query yields 1,031 relevant answers, but documents about specific stresses, such as phosphate starvation (97) were missed because stress is not defined in MeSH despite of its importance in Biology and it is then searched without any specialization. Five more documents could have been found if Entrez PubMed had lemmatized the documents. When MeSH term index is disabled, no document is retrieved as no paper full-text contains all the query words.
BioAlvis retrieved fewer documents than Google Scholar for two reasons: (1) its document collection is smaller (2) Google Scholar indexes scientific papers full text whereas BioAlvis only indexes abstracts and titles. It does not question the semantic annotation approach but the document collection preparation. BioAlvis missed also relevant documents because of the lack of some relevant sub-concepts of stress in the ontology like acid shock. This can be addressed by completing the ontology from a larger training corpus.
Regarding relevance of the documents, the accuracy of the answer sets varies a lot among the services. Google and Google Scholar results contain a vast amount of false positives. This is mainly due to the fact that the answer set contains many documents that mention only a subset of the query terms. The rank of these documents is very low but they are however counted in the answer set. The amount of false positives in Google Scholar is less important because the indexed corpus is smaller and more focused. Beyond the main problem of spurious co-occurrence of the query words mentioned above, the indexing of irrelevant subparts of the document caused many errors. For instance, citations of the document as occurring in Citeseer or SpringerLink sites are indexed with the document itself. BioAlvis retrieves false positives to a much lesser extent. Most of the irrelevant documents were papers about organisms other than Enterobacteriacea, many of them including a mention of a homology with the extensively studied enterobacterium Escherichia coli. This observation stresses the importance of filtering semantic annotations for IR purposes, so that semantic annotation focuses on the main topics of the paper.
While formal languages for ontology representation have made great advances, there
are few formal or operational proposals designed to tie ontologies to linguistic
knowledge [
In this paper, we pointed out the challenge of integrating ontology knowledge and
linguistic knowledge into a consistent model. In order to alleviate the lack of
specialized knowledge to feed NLP tools, knowledge acquisition and ML methods are
applied to training corpora. This raises the problem of integrating the processes of
knowledge resource acquisition and the exploitation of these resources. We proposed
an operational approach based on the clear specification of the learning task and the
normalization of the example representation. Following these principles, we
developed large resources in Biology for each linguistic step and demonstrated their
efficiency through the semantic annotation of a representative Web corpus and its use
in an IR application [
Acknowledgment. This research has been partially supported by the RNTL ExtraPloDocs and the FP6-STREP Alvis projects. Special thanks are due to Philippe Bessières. Without his expert participation in Biology, BioAlvis development would not have been possible. The terminology part of this work has greatly benefited from Annick Lacombe’s expertise.