As shown by the recent outbreak of Severe Acute Respiratory Syndrome (SARS) and emerging cases of avian influenza, infectious diseases have the potential to spread rapidly through person-to-person transmission within densely populated areas and across country borders through international air travel. The first line of defense against rapidly spreading diseases is surveillance, led by the World Health Organization (WHO) and national health authorities. Catching an outbreak earlier has clear implications for both morbidity and mortality as well as the feasibility of containment [1]. However a lack of surveillance system infrastructure in Southeast Asia, which is currently the focus of an avian H5N1 epidemic is seen as hindering control efforts. In addition to traditional surrogate methods such as reporting notifiable diseases and over-the-counter (OTC) sales monitoring, public health experts are increasingly considering news and other reports available on the World Wide Web (Web) as a costeffective means of helping to find and track early cluster cases, enabling a timely and appropriate response. Such rumour-based information may be of particular value for assessing possible outbreaks in areas where formal reporting procedures are absent or not well established.
Several major challenges exist in locating Webbased information in a timely manner using traditional search methods:
(1) the massively increasing volume of dynamically changing unstructured news data available on the Web makes it extremely difficult to obtain a clear picture of an outbreak in a timely manner, (2) the large-scale republication of reports from centralized news agencies requires redundancy to be identified and removed, (3) the initial reports of an outbreak are contained in only a few news articles which will usually be overlooked by traditional search engines which use keyword indexing, (4) the first reports of an infectious disease will often be reported in local news media which are only available in the local language. Experience has shown that this requires computer systems to have at least a partial understanding of the domain through ontologies, term lists and databases as well as specialized multilingual resources. To address the information needs in the domain of infectious disease outbreaks, standard Information Extraction technology has been adapted for retrospective archive search [2] but only a few systems are currently actively deployed with the most prominent being the Global Public Health Intelligence Network (GPHIN) [3], a successful but semi-closed system used by the WHO. We are now developing BioCaster, a text mining system based on an openly available multilingual ontology for proactive notification about priority disease outbreaks. A key component of the BioCaster system is the use of automated learning methods to identify novel entities and events using features derived from annotated examples in a multilingual collection of news articles. The initial target languages are English, Japanese, Vietnamese and Thai.
In our early development of BioCaster it became clear that we needed a rigorous schema for markable entities. Since the system relies on high quality human annotated training data for constructing named entity recognizers (NERs), any inconsistency introduced into the annotation schema by ontological inconsistencies should be harmful for annotation performance, both human and machine. Surprisingly while there have been several studies on the mapping problem between terms and coding systems such as the UMLS Metathesaurus [4] as well as biomedical annotation experiments [5] [6] [7] there have been to the best of our knowledge no studies conducted into the method by which new domain models suitable for biomedical text mining should be organized. We report here on our initial experience which showed that the task-oriented annotation schema based on a poorly-considered domain ontology can indeed be harmful to accuracy. Re-organizing this schema using well founded ontological principles produced better results, despite the added complexity.
are concerned with the circumstances in which diseases occur in a population and the factors that influence their incidence, spread, recognition and control. Our initial discussions with domain experts at the National Institute of Infectious Diseases revealed several common scenarios for gathering information from Web news including cases involving the spread of a communicable disease across international borders and the contamination of blood products. From these initial discussions we collected examples of early outbreak news reports and compiled a list of significant entity classes which included DISEASE 1 , CASE, LOCATION SYMPTOM, TIME, DRUG, etc.
Subsequent follow up discussions and examination of the literature revealed that we can categorize these concepts according to the information needs of the scientists as shown in Table 1.
Genetic epidemiology adds another dimension to the information needs as the genetic makeup of the host plays a key role in determining susceptibility or resistance to pathogens. We therefore chose to add in a further level of detail about the host which includes genes and their products, identified with a §. Finally we had 19 categories of concepts which we want to identify in news texts (Table 2).
At this stage we were aware that some of the important concepts in Table 2 are contextuallydependent and intrinsically different from others. For example, CASE and TRANSMISSION represent roles (discussed in [8] [9] [10] [11] among others) which are dependent on the existence of events they participate in, while most others, such as PERSON, BACTERIA, and NON_HUMAN, represent types.
We had two options for constructing the ontology and annotation schema, according to how to deal with concepts of a different nature. The first approach is rather task-oriented. Here we do not make any distinction between context-dependent concepts and others. This results in a somewhat simpler ontology: all categories of concepts are represented as classes which follow a disjoint entity class principal that has been the underlying premise of NERs. The corresponding annotation schema will also be simpler, since instances of context-dependent classes are annotated in the same way as those of other classes, e.g. <NAME cl="PERSON">Kofi Annan</NAME> <NAME cl="CASE">a 12 year-old girl</NAME> infected with H5N1
(The details of this schema will be given in the next section.) In this task-oriented approach, we can annotate exactly what the event frame needs to identify.
For example, we can exclude from annotation non-named, non-case mentions, which we are not interested in. A defect of this approach is that it is not ontologically well-founded.
The alternative approach is a more formal one where we make a clear distinction between contextdependent concepts and others, based on wellfounded ontological principles. The result is likely to be a more complex ontology in which contextdependent concepts have a different status from other concepts. The corresponding annotation schema will also be more complex as well, since roles are annotated in a different way from those of entity classes. In order to achieve ontological consistency we also need to annotate more mentions than the former approach, including those that will not instantiate event frames. From the two approaches above, out of expediency we chose the former for the first annotation experiment. The reason being that it seemed easier for annotators and that we could find almost no precedent works in named entity annotation which dealt with formal analysis of entities and role concepts.
Based on the list of categories of concepts in Table 2, we constructed the ontology shown in Figure 1 [12], GENIA ontology [13], MUC-7 [14], and HUB-4 [15], respectively.
In the annotation schema used in the example above, the attribute cl takes the entity class label as its value. For example "<NAME cl="PERSON">Kofi Annan</NAME>" means that the entity mentioned by "Kofi Annan" is related to the class PERSON. The reason for using this rather vague expression is to cover two relations between mentioned entities and the ontology we want to describe. The first is "is an instance of", and the other one is "is a subclass of". Some of the markable texts mention a particular and others mention a universal. For example, names of persons, locations and organizations are usually used to refer to a particular, whereas names of chemical substance, viruses and proteins are often used to refer to universals. This is one of the factors which makes ontology-based annotation a complicated process. It should be noted though that we intend to work towards a clear distinction between the two relations in future work.
During the first annotation experiment, we had many problem reports form annotators, and found a significant number of inconsistencies in the annotation results. Most of the problems could be traced back to poor design of the domain ontology and the annotation schema. Follow up analysis on the corpus yielded the following symptoms of error:
roles, have the same status as other classes since we adopted the task-oriented approach as discussed in the last section. We developed annotation guidelines to annotate non-overlapping mentions related to the classes in news articles and hired two PhD informatics students as annotators. After 1-week of training consisting of guideline review, case study discussions and test cases, we started the annotation process with 200 news articles taken from domain sources, including WHO epidemic reports, IRIN, and Reuter news.
Gaps in the annotation schema shown by the existence of mentions to entities which it is desirable to annotate but the annotation schema does not cover.
In order to restrict the markable mentions to exactly those that we aimed to identify with the text mining system, we defined CASE as the class of confirmed cases which are unnamed, and PERSON as the class of named persons who are not cases. We considered this would narrow down the number of markable mentions since unnamed mentions for non-cases need not be annotated. We also instructed annotators to markup only the single most appropriate class, prohibited multiple classes. An example of annotated text is shown below: Ambiguity between context-dependent concepts and context-independent ones Idiosyncratic annotations which are forced on annotators due to the disjoint entity class principal.
At the initial stage of our analysis we considered that distinguishing CASE (as confirmed cases of a disease which are unnamed humans) from PERSON (named persons who are not cases of a disease) was rather natural, since CASE entities are in general anonymous. However, in the news articles there were some examples where cases were mentioned by name as follows:
The <NAME cl="ORGANIZATION">Ministry of Health</NAME> in <NAME cl="LOCATION"> Indonesia</NAME> has today confirmed <NAME cl="CASE">a fatal human case</NAME> of <NAME cl="DISEASE">H5N1 avian influenza</NAME>. <NAME cl="CASE">A 27year-old woman</NAME> from <NAME cl="LOCATION">Jakarta</NAME> developed symptoms on <NAME cl="TIME">17 September</NAME>. She contracted the virus from close contact with infected <NAME cl="TRANSMISSION">birds</NAME>. E1 Tests carried out in a UK laboratory confirmed that M.A and F died from the H5N1 strain2
In addition, we found that there were more frequent mentions of putative cases than we had expected.
These mentions were often annotated as CASE by annotators although we restricted the scope of this class only to confirmed cases.
E2 a Taiwanese is suspected to have died of SARS Follow up discussions with public health experts revealed that mentions of putative cases are important, especially in the early stages of disease outbreaks, and we concluded that they should be identified by the system. However, the existing framework made them difficult to capture.
One of the classes which confused annotators most was TRANSMISSION (source of infection). Below are typical examples of problematic cases.
E3 Victims contract the virus from close contact with infected birds E4 There is no known cure for Ebola, which is transmitted via infected body fluids E5 An Irish woman infected with Hepatitis C by a contaminated blood product E6 18 hospitalized after consuming chapattis Annotators had a problem in annotating 'birds' in E3 since those can be classified as both TRANSMISSION and NON_HUMAN (animals). 'Body fluid' in E4 is also ambiguous between TRANSMISSION and ANATOMY (body parts), and also 'blood product' in E5 is ambiguous between TRANSMISSION and PRODUCT (biological product). Most of the TRANSMISSION instances found in the text were those which could be categorized as NON_HUMAN, and the cases which belonged only to TRANSMISSION, such as 'chapattis' in E6, were very few.
E7 <NAME cl="PERSON">Hudd</NAME> has written several books on music hall and Variety... E8 Doctors later diagnosed <NAME cl="CASE">Hudd</NAME> with a chest infection... In the example above, it is clearly undesirable that the same entity is related to PERSON in E7 and CASE in E8. Although the annotator was aware of the choices the principal of disjoint classes forced a choice.
We trained a support vector machine [13] (for details, see Takeuchi and Collier [14]) for named entity recognition based on the annotated corpus of 200 news articles. 10-fold cross validation experiments were performed using TinySVM3 . A -2/+1 features window was used that included surface word, orthography, biomedical prefixes/suffixes, lemma, head noun and previous class predications. The Fscore for the all classes in Table 2 was 76.96. Among the problematic classes were found to be PERSON, CASE and NON_HUMAN (many instances of which had ambiguity with TRANSMISSION) which had F-scores below our expectation: PERSON (54.95), CASE (53.17), NON_HUMAN (68.0).
Although we chose the task-oriented approach for its simplicity and ease of implementation the results from automatic NER and subsequent corpus analysis revealed that problems arose because we made no clear distinction between context-dependent and context-independent classes. We decided to take an alternative, formal and linguistically-sound approach, and distinguish context-dependent concepts from others in both the ontology and the annotation schema.
The first step was to use the classification method proposed by Guarino and Welty ([9] and [10]) which is based on meta-properties (rigidity, identity, dependency), in order to classify categories of concepts in Table 2 We consider that this class is anti-rigid, since it is possible that an action which is an instance of CONTROL in the current world is not an instance of CONTROL in some other accessible world. The same action may be conducted for different purposes in different worlds. *2 This class includes events. In DOLCE top level categories (Gangemi et al. [19]), Events are under the class of Perdurant/Occurrence. It seems to be controversial what the identity condition for events should be. Davidson [20] proposes a condition such that "events are identical if and only if they have exactly the same causes and effects". In any case it should be reasonable to assume that this class itself does not supply ICs but inherits them from the upper level classes. *3 What we consider ICs for this class is as follows: Two instances of diseases are identical iff the two are experienced by the same host at the same time, are caused by the same agent (e.g. H5N1 virus for "H5N1 avian influenza") and have the same set of characteristic alterations/symptoms (e.g. inflammation of the lung for "pneumonia").
necessary IC: E(x, t)∧φ(x, t)∧E(x, t')∧φ(y, t')∧ x=y →Γ(x, y, t, t') sufficient IC: E(x, t)∧φ(x, t)∧E(x, t')∧φ(y, t')∧ Γ(x, y, t, t') →x=y (E : "actually exist at time t")
Any property φ carries an IC (+I) iff it is subsumed by a property supplying that IC.
A property φ supplies an IC (+O) iff i) it is rigid; ii) there is a necessary or sufficient IC for it; and iii) the same IC is not carried by all the properties subsuming φ.
<Dependency> ([10], p.7) externally dependent property φ (+D): ∀x□(φ(x) →∃y ω(y) ∧￢P(y, x) ∧￢C(y, x)) (P: "is a part of") (C: "is a constituent of")
Classification results are shown in Table 3. Most concepts such as ANATOMY, NON_HUMAN, and PERSON are classified as Type, whereas the concepts which were problematic in the first experiment were classified as Role: TRANSMISSION (Formal Role) and CASE (Material Role).
According to the further classification of non-rigid concepts by Kaneiwa and Mizoguchi [18], these cases are classified as timedependent concepts.
For some of the roles in Table 3, we modified their status in the annotation schema.
CASE and PERSON were problematic since we distinguished them according to the form of expression (unnamed/named), in addition to the case/non-case distinction. In order to cover the mentions which could not be annotated in the first experiment, we extended the scope of the PERSON class to include person instances in general, and eliminate the unnamed/named and case/non-case distinctions. We modified the annotation schema so that CASE is not the value of cl attribute, but is the case attribute which applies to the referred instance of PERSON. This attribute takes the value true when the mentioned instance is a confirmed case of disease, false when the instance is not a case, and putative when the instance is a suspected case. Named case mentions and suspected case mentions are annotated as follows:
E9 Tests carried out in a UK laboratory confirmed that <NAME cl="PERSON" case="true">M.A</NAME>... E10 <NAME cl="PERSON" case="putative">a Taiwanese</NAME> is suspected to have died of SARS
The meaning of case attribute-value pairs can be described in logical description and natural language as follows: <...cl="PERSON" case="true">John</...>: case(j) "It is true that the person j mentioned by "John" is an instance of the CASE class" <...cl="PERSON" case="false">John</...>: ￢case(j) "It is false that the person j mentioned by "John" is an instance of the CASE class" <...cl="PERSON" case="putative">John</..>: ◇case(j) "It is possible that the person j mentioned by "John" is an instance of the CASE class"
As shown above, the values of the case attribute correspond to logical operators such as ￢ and ◇.
The values of case attributes specify the modes of linkage between the referred concept and the CASE class. The formal basis we had in mind when formulating the case attribute are as follows: 1) every instance of a non-rigid class must be an instance of some rigid class, 2) the relations between a non-rigid class and its instance are often modified by modal/temporal operators. The first point drove us to create the case attribute which apply to instances of some rigid class, here, PERSON. The second point is the motivation for us to set values to include negative and modal operators. This schema can be extended if we allow a wider value range for the case attribute to include other modal/temporal operators, although currently we restrict the values to the three above.
It is worth noting that there is a trade-off between this revised schema and the former schema which is that we have increased the number of the markable entities, since we need to annotate unnamed, noncase mentions which are not directly related to the purpose of the system.
We defined the transmission attribute which applies to mentions of ANATOMY, PRODUCT, PERSON and NON_HUMAN classes. As shown in the following examples, 'birds' are always related to NON_HUMAN, and take a 'true' value only when they are mentioned as a source of infection. It can also take a 'putative' value to cover mentions to possible sources of infection.
E11 Victims contract the virus from close contact with infected <NAME cl="NON_HUMAN transmission="true">birds</NAME> T_CHEMICAL /NT_CHEMICAL Concept classification revealed that T_CHEMICAL and NT_CHEMICAL have "the situation dependency obtained from extending types" discussed in [18] and have the same status as 'weapon' and 'table '. T_CHEMICAL includes chemicals mentioned as drugs in any context and those regarded as drugs in some context. Here we removed the two classes and made the parent node CHEMICAL as a class for annotation.
We then defined therapeutic attribute which applies to mentions of CHEMICAL and takes the value true when the entity is intended for therapeutic use and false otherwise.
As a result of the modifications above, our revised ontology is shown in Figure 2. We also added new classes CONDITION (status of patients: 'hospitalized' 'died 'in critical condition', etc) and OUTBREAK (collective disease incident: 'outbreak', 'pandemic', etc). Information about CONDITION is important for experts to know the rate of hospitalization and death and determine the alert level. Mentions of OUTBREAK include expressions which are specific to disease outbreak news, increasing the specificity of our detection system. We located PERSON and NON_HUMAN under metazoa, and added a number attribute (which takes one or many as its value) to be applied to PERSON instances.
With insights from the revised ontology we also changed the annotation method by dividing the process into two distinct stages as shown in Figure 3: 1) annotation of mentions to non-role (rigid) concepts and 2) annotation of role (non-rigid) concepts.
We asked three PhD students to annotate a further 300 news articles. This time we used the revised annotation method 1 and 2 shown in Figure 3.
As a result of distinguishing between Role concepts (case, transmission, therapeutic) from others in the annotation schema, problem reports on these classes were reduced, and the annotation results were also improved.
Contrary to our expectations, the complexity of the new annotation schema and the increased number of markable mentions seemed to have no negative influence on the annotator's speed.
The improvement can be seen empirically in the NER results. We re-annotated the corpus used in the first experiment using the revised annotation schema. This time the F-score for all classes rose to 79.96 (+3 compared to the previous result).
Especially, significant increases of the F score were observed in the classes for PERSON (66.28; +11.33 compared to the previous result), case mentions among PERSON (65.63; +12.46), and NON_HUMAN (73.21; +5.21).
Some of the problems reported in this second experiment were related to context dependency (antirigidity, situation dependency) discussed in Section 6.2.
The most difficult class seemed to be CONTROL (control measures to lower the risk of diseases). As shown in Table 3, we consider this class is also nonrigid, and it includes mentions which refer to subclasses of the CONTROL class regardless of situation ("quarantine" "vaccination"), and others which can be a control measure depending on the situation ("warning" "blockade"). This characteristic seems to cause the difficulty.
So far we have resolved the complexity of nonrigid concepts by defining attributes which apply to instances of rigid classes (e.g. the case attribute for the class PERSON). This strategy, however does not seem to be effective for CONTROL since it is not easy to identify a rigid superclass for CONTROL which can be realistically annotated in the text. For example, EVENT can be considered as a rigid class subsuming CONTROL, but currently it is not realistic to manually annotate every mention of an event. Currently we are seeking for a way to deal with this problem.
The study in this paper was motivated by our need for a high quality annotation schema to support detection of novel entities in the infectious disease outbreak domain. We discussed two experiments based on alternative approaches for constructing an ontology-based annotation schema. The amount of data in our study is relatively small but empirical results indicate support for our view that there is a positive effect in adopting well founded ontological principals over an ad-hoc task-based approach. Although this study is not a formal evaluation of ontologies, it is still an evaluation from the viewpoint of ontology application to the task of natural language annotation. The classification method of Guarino and Welty ([9], [10]) which was originally proposed to achieve consistency in the configurational structure of ontologies, was adapted and found to be useful for improving annotation performance.
An alternative possibility exists which we have not addressed in this paper which is to reformulate the tradition NER task to allow for overlapping (nested) and multi-class entities. This however introduces significant additional complications in both the recognizer models and in the annotation schema so we have adopted a less radical formulation in this work.
As the next step in this study, we are now extending our simple taxonomy to a multi-lingual ontology; enriching the current taxonomic structure domain-sensitive relations. The resulting ontology will be freely available for re-use. At the initial stage we are focusing on English, Japanese, Vietnamese, Thai, Chinese (standard) and Korean. We hope to add other Asia-Pacific languages in the future.
We gratefully acknowledge partial funding support from the Japan Society for the Promotion of Science (grant no. 18049071). We also thank the anonymous reviewers for helpful comments.