Amid growing public concern about the spread of infectious diseases such as avian influenza and SARS, there is an increasing need for collecting timely and reliable information about disease outbreaks from natural language data such as online news articles. In this paper we introduce BioCaster, a text mining-based system for infectious disease detection and tracking currently being developed, and discuss the development of a domain ontology and schema for the annotation of terms. In particular we focus on the comparison between two approaches, 1) a traditional task-oriented approach with a simple schema that does not strictly follow ontological principles, and 2) a formal approach which is ontologically well-founded but adds extra requirements to the annotation schema. We report on several critical problems that were highlighted by an entity annotation experiment, attributable to the purely task-oriented ontology design. A second experiment based on a formally constructed ontology produced improved annotation results despite the apparent complexity of the annotation schema.
As shown by the recent outbreak of Severe Acute
Respiratory Syndrome (SARS) and emerging cases
of avian influenza, infectious diseases have the
potential to spread rapidly through person-to-person
transmission within densely populated areas and
across country borders through international air
travel. The first line of defense against rapidly
spreading diseases is surveillance, led by the World
Health Organization (WHO) and national health
authorities. Catching an outbreak earlier has clear
implications for both morbidity and mortality as well
as the feasibility of containment [
Several major challenges exist in locating Webbased information in a timely manner using traditional search methods: (1) the massively increasing volume of dynamically changing unstructured news data available on the Web makes it extremely difficult to obtain a clear picture of an outbreak in a timely manner, (2) the large-scale republication of reports from centralized news agencies requires redundancy to be identified and removed, (3) the initial reports of an outbreak are contained in only a few news articles which will usually be overlooked by traditional search engines which use keyword indexing, (4) the first reports of an infectious disease will often be reported in local news media which are only available in the local language. Experience has shown that this requires computer systems to have at least a partial understanding of the domain through ontologies, term lists and databases as well as specialized multilingual resources.
To address the information needs in the domain of
infectious disease outbreaks, standard Information
Extraction technology has been adapted for
retrospective archive search [
In our early development of BioCaster it became
clear that we needed a rigorous schema for markable
entities. Since the system relies on high quality
human annotated training data for constructing
named entity recognizers (NERs), any inconsistency
introduced into the annotation schema by ontological
inconsistencies should be harmful for annotation
performance, both human and machine. Surprisingly
while there have been several studies on the mapping
problem between terms and coding systems such as
the UMLS Metathesaurus [
Epidemiologists are concerned with the circumstances in which diseases occur in a population and the factors that influence their incidence, spread, recognition and control. Our initial discussions with domain experts at the National Institute of Infectious Diseases revealed several common scenarios for gathering information from Web news including cases involving the spread of a communicable disease across international borders and the contamination of blood products. From these initial discussions we collected examples of early outbreak news reports and compiled a list of significant entity classes which included DISEASE1, CASE, LOCATION SYMPTOM, TIME, DRUG, etc.
Subsequent follow up discussions and examination of the literature revealed that we can categorize these concepts according to the information needs of the scientists as shown in Table 1.
Genetic epidemiology adds another dimension to the information needs as the genetic makeup of the host plays a key role in determining susceptibility or resistance to pathogens. We therefore chose to add in a further level of detail about the host which includes genes and their products, identified with a §. Finally we had 19 categories of concepts which we want to identify in news texts (Table 2).
At this stage we were aware that some of the
important concepts in Table 2 are
contextuallydependent and intrinsically different from others.
For example, CASE and TRANSMISSION represent
roles (discussed in [
domain entity classes. participate in, while most others, such as PERSON, BACTERIA, and NON_HUMAN, represent types.
We had two options for constructing the ontology and annotation schema, according to how to deal with concepts of a different nature. The first approach is rather task-oriented. Here we do not make any distinction between context-dependent concepts and others. This results in a somewhat simpler ontology: all categories of concepts are represented as classes which follow a disjoint entity class principal that has been the underlying premise of NERs. The corresponding annotation schema will also be simpler, since instances of context-dependent classes are annotated in the same way as those of other classes, e.g. <NAME cl="PERSON">Kofi Annan</NAME> <NAME cl="CASE">a 12 year-old girl</NAME> infected with H5N1 (The details of this schema will be given in the next section.) In this task-oriented approach, we can annotate exactly what the event frame needs to identify. For example, we can exclude from annotation non-named, non-case mentions, which we are not interested in. A defect of this approach is that it is not ontologically well-founded.
The alternative approach is a more formal one where we make a clear distinction between contextdependent concepts and others, based on wellfounded ontological principles. The result is likely to be a more complex ontology in which contextdependent concepts have a different status from other concepts. The corresponding annotation schema will also be more complex as well, since roles are annotated in a different way from those of entity classes. In order to achieve ontological consistency we also need to annotate more mentions than the former approach, including those that will not instantiate event frames.
From the two approaches above, out of expediency we chose the former for the first annotation experiment. The reason being that it seemed easier for annotators and that we could find almost no precedent works in named entity annotation which dealt with formal analysis of entities and role concepts.
Based on the list of categories of concepts in Table 2, we constructed the ontology shown in Figure 1. Note that CASE and TRANSMISSION, which represent Transmission Host The delivery or dispersal method Persons carrying a disease Environment
Location and climate * Not included in the current schema § Genetic level entities
Example properties
Infectivity, pathogenicity, virulence, incubation
period, communicability
Dermal, oral, respiratory
Age, gender, occupation,
Large population centre, enclosed building, mass
transport system, rural village
Classes Examples Description
ANATOMY liver, pancreas, nervous system, eLa cel, Body parts including tissues and cells
BACTERIA Escherichia coli O157, tubercle bacillus Eubacteria
CASE a 35-year-old woman, the third case Confirmed cases of diseases
NT_CHEMICAL beryllium, organophosphate pesticide Chemicals intended for non-therapeutic purposes *1
T_CHEMICAL Relenza, immunosuppressive drug, oseltamivir Chemicals intended for the treatment of diseases*1
CONTROL stamping out, screening, vaccination Control measures to lower the risk of transmission of a
disease
DISEASE H5N1 avian influenza, SARS, cholera A deviation in the normal functioning of the host caused
by a persistent agent (pathogen) or some environmental
factor
DNA Sp1 site, triple-A, c-jun gene Includes the names of DNAs, groups, families, molecules,
domains and regions*2
LOCATION Viet Nam, Jakarta, Sumatra Island, Asia A politically or geographically defined location*3
NON_HUMAN civet cats, poultry, flies Multi-cell organism other than humans, i.e. "animals"
ORGANIZATION the Ministry of Health, WHO, Pasteur Institute Corporate, governmental, or other organizational entity*3
PERSON Jean Chretien, Murray McQuigge A named person or family
PRODUCT botulism antitoxin, Influenza vaccine Biological product, (e.g. vaccines, immune sera)
PROTEIN STAT, RNA polymerase II alpha subunit Includes the names of proteins, groups, families,
molecules, complexes and substructures*2
RNA IL-2R alpha transcripts, TNF mRNA Includes the names of RNAs, groups, families, molecules,
domains and regions*2
SYMPTOM cough, fever, dehydration, convulsion Alterations in the appearance of a case due to a disease
TIME Tue Jan 3, winter, March, since October, 2003 Temporal expressions that can be anchored on a
timeline*4
TRANSMISSION HIV-tainted blood products, BSE-infected cows Source of infection
VIRUS Ebola virus, HIV Viruses such as HIV, HTLV, EBV *2
Descriptions marked with *1 , *2, *3, *4 are based on those in MeSH [
In order to restrict the markable mentions to exactly those that we aimed to identify with the text mining system, we defined CASE as the class of confirmed cases which are unnamed, and PERSON as the class of named persons who are not cases. We considered this would narrow down the number of markable mentions since unnamed mentions for non-cases need not be annotated. We also instructed annotators to markup only the single most appropriate class, prohibited multiple classes. An example of annotated text is shown below:
The <NAME cl="ORGANIZATION">Ministry of Health</NAME> in <NAME cl="LOCATION"> Indonesia</NAME> has today confirmed <NAME cl="CASE">a fatal human case</NAME> of <NAME cl="DISEASE">H5N1 avian influenza</NAME>. <NAME cl="CASE">A 27year-old woman</NAME> from <NAME cl="LOCATION">Jakarta</NAME> developed symptoms on <NAME cl="TIME">17 September</NAME>. She contracted the virus from close contact with infected <NAME cl="TRANSMISSION">birds</NAME>.
In the annotation schema used in the example above, the attribute cl takes the entity class label as its value.
For example "<NAME cl="PERSON">Kofi Annan</NAME>" means that the entity mentioned by "Kofi Annan" is related to the class PERSON.
The reason for using this rather vague expression is to cover two relations between mentioned entities and the ontology we want to describe. The first is "is an instance of", and the other one is "is a subclass of".
Some of the markable texts mention a particular and others mention a universal. For example, names of persons, locations and organizations are usually used to refer to a particular, whereas names of chemical substance, viruses and proteins are often used to refer to universals. This is one of the factors which makes ontology-based annotation a complicated process. It should be noted though that we intend to work towards a clear distinction between the two relations in future work.
During the first annotation experiment, we had many problem reports form annotators, and found a significant number of inconsistencies in the annotation results. Most of the problems could be traced back to poor design of the domain ontology and the annotation schema. Follow up analysis on the corpus yielded the following symptoms of error: • • •
Gaps in the annotation schema shown by the existence of mentions to entities which it is desirable to annotate but the annotation schema does not cover.
Ambiguity between context-dependent concepts and context-independent ones Idiosyncratic annotations which are forced on annotators due to the disjoint entity class principal.
At the initial stage of our analysis we considered that distinguishing CASE (as confirmed cases of a disease which are unnamed humans) from PERSON (named persons who are not cases of a disease) was rather natural, since CASE entities are in general anonymous. However, in the news articles there were some examples where cases were mentioned by name as follows: E1
Tests carried out in a UK laboratory confirmed that M.A and F died from the H5N1 strain2 In addition, we found that there were more frequent mentions of putative cases than we had expected. 2 In this example we only show initials of the victims' names. These mentions were often annotated as CASE by annotators although we restricted the scope of this class only to confirmed cases.
a Taiwanese is suspected to have died of SARS Follow up discussions with public health experts revealed that mentions of putative cases are important, especially in the early stages of disease outbreaks, and we concluded that they should be identified by the system. However, the existing framework made them difficult to capture.
One of the classes which confused annotators most was TRANSMISSION (source of infection). Below are typical examples of problematic cases.
E3 E4 E5 E6
Victims contract the virus from close contact with infected birds There is no known cure for Ebola, which is transmitted via infected body fluids An Irish woman infected with Hepatitis C by a contaminated blood product 18 hospitalized after consuming chapattis Annotators had a problem in annotating ‘birds' in E3 since those can be classified as both TRANSMISSION and NON_HUMAN (animals). ‘Body fluid’ in E4 is also ambiguous between TRANSMISSION and ANATOMY (body parts), and also ‘blood product’ in E5 is ambiguous between TRANSMISSION and PRODUCT (biological product). Most of the TRANSMISSION instances found in the text were those which could be categorized as NON_HUMAN, and the cases which belonged only to TRANSMISSION, such as ‘chapattis’ in E6, were very few.
E7 E8 <NAME cl="PERSON">Hudd</NAME> has written several books on music hall and Variety...
Doctors later diagnosed <NAME cl="CASE">Hudd</NAME> with a chest infection...
In the example above, it is clearly undesirable that the same entity is related to PERSON in E7 and CASE in E8. Although the annotator was aware of the choices the principal of disjoint classes forced a choice.
We trained a support vector machine [
Although we chose the task-oriented approach for its simplicity and ease of implementation the results from automatic NER and subsequent corpus analysis revealed that problems arose because we made no clear distinction between context-dependent and context-independent classes. We decided to take an alternative, formal and linguistically-sound approach, and distinguish context-dependent concepts from others in both the ontology and the annotation schema.
The first step was to use the classification method
proposed by Guarino and Welty ([
rigidity identity (supplying) identity (carrying) dependency classification
ANATOMY +R +O + I - D Type
BACTERIA +R +O + I - D Type
CASE ~R - O + I +D Material Role
NT_CHEMICAL ~R - O + I +D Material Role
T_CHEMICAL ~R - O + I +D Material Role
CONTROL ~R *1 - O*2 + I +D Material Role
DISEASE +R +O*3 + I +D Type
DNA +R +O + I - D Type
LOCATION +R +O + I - D Type
NON_HUMAN +R +O + I - D Type
ORGANIZATION +R +O + I - D Type
PERSON +R +O + I - D Type
PRODUCT +R +O + I +D Type
PROTEIN +R +O + I - D Type
RNA +R +O + I - D Type
SYMPTOM +R +O + I +D Type
TIME +R +O + I - D Type
VIRUS +R +O + I - D Type
TRANSMISSION ~R - O - I +D Formal Role
*1 We consider that this class is anti-rigid, since it is possible that an action which is an instance of CONTROL in the current world is not an
instance of CONTROL in some other accessible world. The same action may be conducted for different purposes in different worlds.
*2 This class includes events. In DOLCE top level categories (Gangemi et al.[
(E : "actually exist at time t")
subsumed by a property supplying that IC.
A property φ supplies an IC (+O) iff i) it is rigid;
ii) there is a necessary or sufficient IC for it; and iii)
the same IC is not carried by all the properties
subsuming φ.
<Dependency> ([
For some of the roles in Table 3, we modified their status in the annotation schema.
CASE and PERSON were problematic since we distinguished them according to the form of expression (unnamed/named), in addition to the case/non-case distinction. In order to cover the mentions which could not be annotated in the first experiment, we extended the scope of the PERSON class to include person instances in general, and eliminate the unnamed/named and case/non-case distinctions. We modified the annotation schema so that CASE is not the value of cl attribute, but is the case attribute which applies to the referred instance of PERSON. This attribute takes the value true when the mentioned instance is a confirmed case of disease, false when the instance is not a case, and putative when the instance is a suspected case. Named case mentions and suspected case mentions are annotated as follows:
that <NAME cl="PERSON" case="true">M.A</NAME>...
E10 <NAME cl="PERSON" case="putative">a Taiwanese</NAME> is suspected to have died of SARS The meaning of case attribute-value pairs can be described in logical description and natural language as follows: <...cl="PERSON" case="true">John</...>: case(j) "It is true that the person j mentioned by "John" is an instance of the CASE class" <...cl="PERSON" case="false">John</...>: ￢case(j) "It is false that the person j mentioned by "John" is an instance of the CASE class" <...cl="PERSON" case="putative">John</..>: ◇case(j) "It is possible that the person j mentioned by "John" is an instance of the CASE class" As shown above, the values of the case attribute correspond to logical operators such as ￢ and ◇. The values of case attributes specify the modes of linkage between the referred concept and the CASE class. The formal basis we had in mind when formulating the case attribute are as follows: 1) every instance of a non-rigid class must be an instance of some rigid class, 2) the relations between a non-rigid class and its instance are often modified by modal/temporal operators. The first point drove us to create the case attribute which apply to instances of some rigid class, here, PERSON. The second point is the motivation for us to set values to include negative and modal operators. This schema can be extended if we allow a wider value range for the case attribute to include other modal/temporal operators, although currently we restrict the values to the three above.
It is worth noting that there is a trade-off between this revised schema and the former schema which is that we have increased the number of the markable entities, since we need to annotate unnamed, noncase mentions which are not directly related to the purpose of the system.
We defined the transmission attribute which applies to mentions of ANATOMY, PRODUCT, PERSON and NON_HUMAN classes. As shown in the following examples, 'birds' are always related to NON_HUMAN, and take a 'true' value only when they are mentioned as a source of infection. It can also take a 'putative' value to cover mentions to possible sources of infection.
E11 Victims contract the virus from close contact with infected <NAME cl="NON_HUMAN transmission="true">birds</NAME>
Concept classification revealed that T_CHEMICAL
and NT_CHEMICAL have "the situation dependency
obtained from extending types" discussed in [
We then defined therapeutic attribute which applies to mentions of CHEMICAL and takes the value true when the entity is intended for therapeutic use and false otherwise.
As a result of the modifications above, our revised ontology is shown in Figure 2. We also added new classes CONDITION (status of patients: 'hospitalized' 'died 'in critical condition', etc) and OUTBREAK (collective disease incident: 'outbreak', 'pandemic', etc). Information about CONDITION is important for experts to know the rate of hospitalization and death and determine the alert level. Mentions of OUTBREAK include expressions which are specific to disease outbreak news, increasing the specificity of our detection system. We located PERSON and NON_HUMAN under metazoa, and added a number attribute (which takes one or many as its value) to be applied to PERSON instances.
With insights from the revised ontology we also changed the annotation method by dividing the process into two distinct stages as shown in Figure 3: 1) annotation of mentions to non-role (rigid) concepts and 2) annotation of role (non-rigid) concepts. therapeutic attribute case attribute number attribute transmission attribute therapeutic attribute
We asked three PhD students to annotate a further 300 news articles. This time we used the revised annotation method 1 and 2 shown in Figure 3.
As a result of distinguishing between Role concepts (case, transmission, therapeutic) from others in the annotation schema, problem reports on these classes were reduced, and the annotation results were also improved. Contrary to our expectations, the complexity of the new annotation schema and the increased number of markable mentions seemed to have no negative influence on the annotator’s speed.
The improvement can be seen empirically in the NER results. We re-annotated the corpus used in the first experiment using the revised annotation schema. This time the F-score for all classes rose to 79.96 (+3 compared to the previous result). Especially, significant increases of the F score were observed in the classes for PERSON (66.28; +11.33 compared to the previous result), case mentions among PERSON (65.63; +12.46), and NON_HUMAN (73.21; +5.21).
Some of the problems reported in this second experiment were related to context dependency (antirigidity, situation dependency) discussed in Section 6.2.
The most difficult class seemed to be CONTROL (control measures to lower the risk of diseases). As shown in Table 3, we consider this class is also nonrigid, and it includes mentions which refer to subclasses of the CONTROL class regardless of situation ("quarantine" "vaccination"), and others which can be a control measure depending on the situation ("warning" "blockade"). This characteristic seems to cause the difficulty.
So far we have resolved the complexity of nonrigid concepts by defining attributes which apply to instances of rigid classes (e.g. the case attribute for the class PERSON). This strategy, however does not seem to be effective for CONTROL since it is not easy to identify a rigid superclass for CONTROL which can be realistically annotated in the text. For example, EVENT can be considered as a rigid class subsuming CONTROL, but currently it is not realistic to manually annotate every mention of an event. Currently we are seeking for a way to deal with this problem.
The study in this paper was motivated by our need
for a high quality annotation schema to support
detection of novel entities in the infectious disease
outbreak domain. We discussed two experiments
based on alternative approaches for constructing an
ontology-based annotation schema. The amount of
data in our study is relatively small but empirical
results indicate support for our view that there is a
positive effect in adopting well founded ontological
principals over an ad-hoc task-based approach.
Although this study is not a formal evaluation of
ontologies, it is still an evaluation from the viewpoint
of ontology application to the task of natural
language annotation. The classification method of
Guarino and Welty ([
An alternative possibility exists which we have not addressed in this paper which is to reformulate the tradition NER task to allow for overlapping (nested) and multi-class entities. This however introduces significant additional complications in both the recognizer models and in the annotation schema so we have adopted a less radical formulation in this work.
As the next step in this study, we are now extending our simple taxonomy to a multi-lingual ontology; enriching the current taxonomic structure with domain-sensitive relations. The resulting ontology will be freely available for re-use. At the initial stage we are focusing on English, Japanese, Vietnamese, Thai, Chinese (standard) and Korean. We hope to add other Asia-Pacific languages in the future.
We gratefully acknowledge partial funding support from the Japan Society for the Promotion of Science (grant no. 18049071). We also thank the anonymous reviewers for helpful comments.