Linguistic scope-based and biological event-based specula- tion and negation annotations in the Genia Event and Bio- Scope corpora Veronika Vincze∗1 , György Szarvas2 , György Móra1 , Tomoko Ohta3 and Richárd Farkas∗1,4 1 University of Szeged, Department of Informatics, Szeged, Hungary 2 Technische Unversität Darmstadt, UKP Lab, Darmstadt, Germany 3 University of Tokyo, Tsujii Laboratory, Tokyo, Japan 4 Hungarian Academy of Sciences, Research Group on Artificial Intelligence, Szeged, Hungary Email: vinczev@inf.u-szeged.hu; szarvas@tk.informatik.tu-darmstadt.de; gymora@inf.u-szeged.hu; okap@is.s.u-tokyo.ac.jp; rfarkas@inf.u-szeged.hu; ∗ Corresponding author Abstract Background: The treatment of negation and hedging in natural language processing has received much interest recently, especially in the biomedical domain. However, open access corpora annotated for negation and/or speculation are hardly available for training and testing applications, and even if they are, they sometimes follow different design principles. In this paper, the annotation principles of the two largest corpora containing annotation for negation and speculation – BioScope and Genia Event – are compared. BioScope marks linguistic cues and their scopes for negation and speculation while in Genia biological events are marked for uncertainty and/or negation. Results: Differences among the annotations of the two corpora are thematically categorized and the frequency of each category is estimated. We found that the largest amount of differences is due to the issue that scopes – which cover text spans – deal with the key events and each argument (including events within events) of these events is under the scope as well. In contrast, Genia deals with the modality of events within events independently. Conclusions: We think that the useful information for the biologist can be acquired from the key events, thus if we aim to detect ”new knowledge”, an automatic scope-detector trained on BioScope can contribute to biomedical information extraction. However, for detecting the negation and speculation status of events (within events) syntax-based rules investigating the dependency path between the modality cue and the event cue may be em- ployed. Background plications seek to extract factual information from text. In order to distinguish assertions from un- In natural language processing (NLP) – and in par- reliable/uncertain information and negated state- ticular, in information extraction (IE) – many ap- 81 ments, linguistic devices of negation or hedges have lations are annotated for negation (1100 sen- to be identified. Applications should handle detected tences in size). modified parts in a different manner. A typical ex- ample is protein-protein interaction extraction from • The BioScope corpus [8], which includes three biological texts, where the aim is to mine text evi- types of texts from the biomedical domain – dence for biological entities that are in a particular namely, radiological reports, biological full pa- relation with each other. Here, while an uncertain pers and abstracts from the Genia corpus – an- relation might be of some interest for an end-user notated for both negation and hedge keywords as well, such information must not be confused with and their linguistic scopes (20924 sentences). factual textual evidence (reliable information). • The system developed by Medlock & Briscoe There are several available negation and hedge [6] made use of a corpus consisting of six papers detection systems (usually for the clinical and bio- from genomics literature in which 1537 sen- logical domains). The first systems were fully hand- tences were annotated for speculation. These crafted [1–3] without any empirical evaluation on a texts – with re-annotation – are also included dedicated corpus. Recently, there have been sev- in BioScope. eral corpora published with manual annotation and several rule-based systems have been developed and • Shatkay et al. [16] describe a database where evaluated on them [4, 5]. 10000 biomedical sentences are annotated for Recent approaches exploit machine learning polarity and three levels of certainty. models. Medlock & Briscoe [6] used single words as input features in order to classify sentences In the corpora Genia Event and BioInfer, biolog- from biological articles (FlyBase) as speculative or ical concepts (relations and events) have been anno- non-speculative based on semi-automatically col- tated for negation and – in the case of Genia Event lected training examples. Szarvas [7] extended their – for hedging as well, but linguistic cues (i.e. which methodology to use n-gram features and a semi- keyword modifies the semantics of the statement) supervised selection of the keyword features. Using have not been annotated for them. In the last two BioScope [8] for training and evaluation, Morante corpora, speculative annotation can be found on the et al. [9] developed in-sentence scope detectors for sentence level. negation and speculation following a supervised se- In contrast to those, BioScope was not fine-tuned quence labeling approach, while Özgür and Radev for information extraction tasks but it contains lin- [10] constructed a rule-based system that exploits guistic annotation for hedge and negative cues and syntactic patterns. BioScope is also the source their in-sentence scope as well. Its chief objective of training and evaluation datasets of the CoNLL- is to investigate these language phenomena in a 2010 Shared Task [11]. Several related works have general, task-independent and linguistically-oriented also been published within the framework of The way. Automatically recognized in-sentence scopes BioNLP’09 Shared Task on Event Extraction [12], (i.e. the negated or hedged text spans) are impor- where a separate subtask was dedicated to predict- tant for many natural language processing applica- ing whether the recognized biological events are un- tions. For instance der negation or speculation [4]. In this paper we focus on corpora annotated for • in clinical document classification tasks [17, negation and speculation. There are several avail- 18], the goal is to assign labels to medical doc- able corpora outside the biomedical domain (e.g. uments according to factual statements about FactBank [13], Wikipedia weasels [11]) as well. How- the patient in question. Here the removal (or ever, we deal here with biological information extrac- separate handling) of hedged or negated text tion and to our best knowledge, the following related spans has a great contribution in the training corpora have been constructed for this domain: and prediction phases as well. • The Genia Event corpus [14] which annotates • In information retrieval the query mentions biological events with negation and two types under hedging can be ranked lower, of uncertainty (9372 sentences). • in machine translation the extension of nega- • The BioInfer corpus [15] where biological re- tion or speculation scopes has to be precisely 82 known in order to translate meaning ade- and 50 sentences where at least one of the arguments quately. of an event was under a BioScope negation scope and marked as existing by Genia (50+50 sentences were Although the BioScope corpus consists of clini- selected for speculation analogously). By manual in- cal and biological documents, its annotation guide- spection of this sample we thematically categorized lines do not contain any domain-specific instruction. these differences. Councill et al. [19] employed BioScope as training corpus for detecting negated scopes for opinion min- ing from product reviews, which proves its task- and Results domain-independence. Annotation principles In the following sections, the hedge and nega- tion annotation principles of BioScope and Genia BioScope annotation Event are compared, resolution strategies for the dif- When annotating keywords and their scopes in the ferences are offered and we discuss how BioScope BioScope corpus [8], the corpus builders followed a can contribute to identifying ”new knowledge” in min-max strategy. When marking the keywords, a biomedical papers. minimalist strategy was followed: the minimal unit that expresses hedging or negation is marked as a keyword. Special attention is paid to the case of complex keywords, that is, words that express un- Methods certainty or negation together, but not on their own In this paper we quantitatively compare the nega- (either the semantic interpretation or the hedging tion and speculation annotations of the BioScope strength of its subcomponents are significantly dif- and Genia Event corpora. We investigated sentences ferent from those of the whole phrase). that occur in both corpora, i.e. the intersection of The scopes of negative and speculative keywords the two corpora containing 958 abstracts and 8942 are extended to the largest syntactic unit possible. sentences (abstracts that were not segmented in the Thus, annotated scopes always have the maximal same way on the sentence level in the two corpora length. In the next example, however is not affected were neglected) was used. This corpus contains 1287 by the hedge cue but it should be included within the negation and 1980 speculation BioScope scopes scope, otherwise the keyword and its target phrase (376 nested scopes) while 2123 non-exist and 1475 would be separated (scopes are marked by brackets probable Genia events (200 events have both la- and keywords are bold): bels). As for negation, events with at least one clue [Atelectasis in the right mid zone is, how- occurring within a negative scope in BioScope and ever, possible]. being annotated as non-exist in Genia Event were considered as cases of agreement. As regards to That is why the corpus builders preferred to include speculation, events with at least one clue within a every possible element within the scope rather than speculative scope in BioScope and being marked as exclude elements that should probably be included. probable in Genia Event were accepted as cases of As for annotating, the most important thing to con- agreement. Mismatches included events with differ- sider is that hedging or negation is determined not ent labels in the two corpora (e.g. an event labeled just by the presence of an apparent cue: it is rather as negative in Genia Event and speculative in Bio- an issue of the keyword, the context and the syntac- Scope) on the one hand, and events annotated only tic structure of the sentence taken together. in one of the corpora on the other hand. The scope of a keyword can be determined on the In order to understand the differences between basis of constituency grammar. The scope of verbs, the annotation principles and to investigate the pos- auxiliaries, adjectives and adverbs usually extends sible contribution of the BioScope annotation to Ge- to the right of the keyword. In the case of verbal el- nia event modality detectors, we randomly sampled ements, i.e. verbs and auxiliaries, it ends at the end 200 sentences from the intersection of the two cor- of the clause (if the verbal element is within a rela- pora. This sampling consists of 50 sentences where tive clause or a coordinated clause) or the sentence, events are marked to be negated by Genia and none hence all complements and adjuncts are included, of its arguments was included in a negation scope in accordance with the principle of maximal scope 83 size. In the case of elliptic sentences, the scope of In the corpus, no explicit marking of either the the negative keyword may be deleted as in: keywords or the scope of negation and hedging can be found. This decrease was seen in patients who responded to the therapy as well as in those who did [not]. Number of disagreements Table 1a shows the number of cases of agreement In these cases, the scope contains only the keyword. and disagreement between the two corpora (agree- ment rate: 48%). The numbers in column TP (true positive) denote instances which are considered in Genia Event modality annotation the same way in both corpora. The numbers in col- The Genia Event corpus was primarily designed for umn BPGN refer to cases where in BioScope any (biological) event annotation [14] and the database clue of a Genia event is under a negative / specu- contains annotation for uncertainty and negation lative scope, however, in Genia Event, it is not. As on the level of events. The annotation scheme fo- opposed to this, in column GPBN, the numbers show cuses on events, and arguments of events can occa- cases where Genia contains some speculative / neg- sionally be found across clause boundaries, typically ative annotation for any argument of the event but due to anaphora or coreference (out of 35419 Ge- BioScope does not. nia events used in our experiment, 1127 referred to an external event and 2076 clues are arguments of an event expressed in another sentence (mostly clue- Categorization of differences types theme (1447 instances, 70%) and cause (619 In this section, mismatches in annotation between instances, 29.8%)). the Genia Event and the BioScope corpora are pre- As for uncertainty, events can have three labels sented. Systematic differences are categorized on the in the corpus: certain, probable and doubtful. basis of a possible solution aiming to resolve the Events are marked as doubtful if they are under in- mismatch, and subtypes of these categories are il- vestigation or they form part of a hypothesis, etc. lustrated with examples along with their estimated An example (event arguments are underlined in our frequencies based on a random sample of 200 anno- examples) for a doubtful event is provided here: tation differences (see Table 1b). We then investigated if HCMV binding Event-centered vs. linguistic annotation An essential also resulted in the translation and se- difference in annotation principles between the two cretion of cytokines. corpora is that Genia Event follows the principles of Event-centered annotation [14] while BioScope an- Events are considered probable if their existence can- notation does not put special emphasis on events not be stated for certain. An example of a probable as it aims a task-independent modeling of specula- event is shown here: tion and negation. Event-centered annotation means that annotators are required to identify as many bi- Together, this evidence strongly impli- ological events as possible within the sentence then cates BSAP in the regulation of the label each separately for negation and speculation. CD19 gene. Events are usually expressed by verbs, however, (de- The attribute certain is chosen by default if none of verbal) adjectives and nouns can also refer to events. the two others hold: an event the existence of which Consider the following example: cannot be questioned in any way. Calcineurin acts in synergy with PMA to As for negation, events are marked with the la- inactivate I kappa B/MAD3, an inhibitor bels exist or non-exist. An example for a negated of NF-kappa B. event is shown below: This sentence describes two events, the inactivation Analysis of Tax mutants showed that of I kappa B/MAD3 by Calcineurin and the inhibi- two mutants, IEXC29S and IEXL320G, tion of NF-kappa B by I kappa B/MAD3. were unable to significantly transactivate From a linguistic point of view, an event is un- the c-sis/PDGF-B promoter. derstood as a predicate together with its arguments 84 BPGN GPBN event within event 68% – TP BPGN GPBN syntax 7% 3% negation 1554 1484 569 lexical semantics 20% 72% probable 1295 3761 180 morphological negation – 14% annotation error 5% 11% TOTAL 100% 100% Table 1: Numbers of agreement and disagreement between BioScope and Genia Event and the frequency of mismatch categories. and the role of the predicate can be fulfilled by a Syntactic issues Some of the mismatches in annota- verb, a noun, or an adjective in the text. In contrast tion can be traced back to syntax. For instance, the to this, BioScope is not event-oriented in the above treatment of subjects remains problematic since in sense. Instead, verbs play a central role, i.e. a verb BioScope it is only the complements that are usually and its arguments form one event in BioScope as included within the scope of a keyword (that is, sub- well. Accordingly, the above sentence refers to one jects are not with the exception of passive construc- event in BioScope and inhibitor is not considered as tions and raising verbs) in contrast to Genia where a predicate. events are argument-centered (i.e. complements and As a consequence, there are much more events in subject are considered) as in: Genia than in BioScope. The multiplicity of events Both c-Rel and RelA induced in Genia Event and the maximum scope principle ex- jagged1 gene expression, whereas ploited in BioScope taken together often yields that a mutant defective for transactivation a Genia event falls within the scope of a BioScope did [not]. keyword, however, it should not be seen as a specu- lated or negated event on its own. Here we provide In this example, no argument of the event denoted an illustrative example: by induced is under the BioScope scope, which yields a case of disagreement. In summary, our data [suggest With regards to the problem concerning the that changes in the composition of treatment of subjects, the dependency parse of the transcription factor AP-1 is a key sentence/clause might help the correct identification molecular mechanism for increasing of the modality of the events. We can apply the IL-2 transcription and may underlie the following rule: if a verb that functions as the trig- phenomenon of costimulation by EC]. ger word for an event is negated or hedged, all its children in the dependency tree (including the sub- According to the BioScope analysis of the sentence, ject as well) are to be included in the scope of the the scope of suggest extends to the end of the sen- modifier. In this way, instances of mismatch when tence. It entails that in Genia it is only the events is it is only the subject that is within the scope of the a key molecular mechanism and underlie the phe- modifier (e.g. in the case of elliptic sentences) can nomenon that are marked as probable, neverthe- be eliminated from the GPBN set. less, the events changes, increasing, transcription Semantic issues There are some cases where the dif- and costimulation are also included in the BioScope ference in annotations originates from conceptual speculative scope. Thus, within this sentence, there discrepancies. These differences can hardly be re- are six Genia events out of which two are labeled solved without harmonizing the annotation princi- as probable, however, in BioScope, all six are within ples behind the corpora and re-annotating the data, a speculative scope, resulting in two cases of agree- however, the most typical cases are presented here. ment and four cases of disagreement. Concerning Events labeled as doubtful in Genia Event are the whole corpora, the large number of BPGN cases rarely annotated as speculative in BioScope. In Ge- (see Tables 1a and 1b) can be explained in a similar nia Event, the investigation, examination, study, etc. way. of a phenomenon does not necessarily mean that the 85 phenomenon exists. However, in BioScope this as- why it is marked as a negative keyword in Bio- pect is neglected and phenomena being under inves- Scope. However, in Genia, ’lack of something’ is tigation, examination, etc. are only marked as in- understood as negation of status, not negation of stances of speculation if they are within the scope of an event. Hence here the class type of the event a speculative keyword (e.g. whether ). As only 17% is negative regulation but the event itself is as- of doubtful Genia event clues is under speculation sertive (out of 4347 negative regulations in Genia scope, we focus just on the probable class during 4164 are assertive, some of which are annotated as our comparison. negative in BioScope due to semantically negative There are some examples of mismatch where a keywords). generalization or a widely accepted claim is stated. Another case of conceptual discrepancy is mor- Grammatically, these sentences usually occur in the phological negation, i.e. on the morphological level, passive voice without explicitly marking the agent the clueword contains a negative prefix such as in- (i.e. the one whom the claim originates from). Such or un-. Here is a typical example: sentences are instances of weaseling [20], and are an- notated as probable events in Genia, however, in In monocytic cells, IL-1beta treatment BioScope they are not as they express a different led to a production of ROIs which is type of uncertainty: it is the exact source of the independent of the 5-LOX enzyme but opinion that is missing rather than the factuality of requires the NADPH oxidase activity. the event (it is known that some hold this opinion but it is unknown who they are). It is a kind of un- The event denoted by led is not triggered by the certainty expressed at the discourse level as opposed presence of the 5-LOX enzyme, thus, there is no to uncertainty on the semantic level. An example regulation event here and this is expressed in Genia for a weasel sentence is shown below: by marking the regulation event with the attribute non-exist while in BioScope its meaning is consid- Receptors for leukocyte chemoattrac- ered to be lexicalized and not necessarily negative. tants, including chemokines, are tradi- Mismatches originating from morphological tionally considered to be responsible for negation mostly include the adjective independent. the activation of special leukocyte func- We argue that although this word contains a nega- tions such as chemotaxis, degranulation, tive prefix at the level of morphology, its meaning and the release of superoxide anions. is lexicalized and not necessarily negative: it rather describes a state or a lack of relation between its Weasel sentences and cue phrases can be automat- arguments. In this way, it could be treated simi- ically detected by employing machine learnt mod- larly to lack, that is, not the event itself but its state els. For instance, the CoNLL-2010 Shared Task should be negated. On the other hand, cluewords dataset [11] includes a corpus dedicated to weasel including morphological negation can be easily iden- detection in Wikipedia articles. We suppose that the tified by automatic methods (segmenting the word phenomenon of weasel is domain-independent hence into a negative prefix and an existing (adjectival) the model trained on Wikipedia could be adequately morpheme) and these can be automatically tagged applied for (biological) scientific publications as well. as negative cues. Sometimes an event is marked as negation in Bio- The interpretation of some speculative keywords Scope but not in Genia: too seems to vary in BioScope and Genia Event. The most striking example is the case of events modified [Lack of full activation of NF-AT] could by other words or phrases expressing ability (e.g. be be correlated to a dramatically reduced able to, ability etc.), which are annotated for proba- capacity to induce calcium flux and bility in Genia but not in BioScope. An example is could be complemented with a calcium offered here: ionophore. NF-kappa B activation correlated with As lack is understood as ’the state of not hav- the ability of CD40 to induce Ab ing something’, it denotes negation, i.e. the non- secretion and the up-regulation of existence of the following NP complement, that is ICAM-1 and LFA-1. 86 A highly interesting subclass of words expressing Similar considerations implied the design of ability is when the derivational suffix conveys the the ”Meta-Knowledge Annotation Scheme for Bio- ’ability’ meaning as in inducible or inhibitable. Take Events” [21]. It introduces dedicated labeling di- the following sentence: mensions of events about Despite stimulation with LPS, disrup- • New Knowledge (yes/no), the motivation of tion of the NF-kappaB signaling path- which is that events ”. . . could correspond to way in precursor B cells led to the loss new knowledge, but only if they represent ob- of inducible Oct-2 DNA binding activ- servations from the current study, rather than ity in vitro and the suppression of Oct- observations cited from elsewhere. In a simi- 2-directed transcription in vivo. lar way, an analysis drawn from experimental results in the current study could be treated as new knowledge, but generally only if it The event described by inducible can be paraphrased represents a straightforward interpretation of as Oct-2 DNA binding activity can be induced in results, rather than something more specula- vitro, which is an ’ability’ usage of the auxiliary can, tive.” thus, it is annotated for probability in Genia but not in BioScope. • Knowledge type (investigation / observation The lexical semantic-related differences originate / analysis / general) whose ”. . . purpose is to from conceptual discrepancies of the two corpora. form the basis of distinguishing between the These mismatches can hardly be resolved without most critical types of rhetorical/pragmatic in- harmonizing the annotation principles behind the tent, according to the needs of biologists.” corpora and re-annotating the data. As one of the chief design goals of BioScope annotation was to Krallinger [22] also argues that from a biologist be task-independent and the modality annotation of point of view only the events supported by experi- Genia is fine-tuned to biological event extraction, bi- mental evidence are interesting. ological information extractors may incorporate the In conclusion, as the BioScope corpus is designed modality principles of Genia while BioScope annota- to be task-independent its scopes could not be ap- tions may be followed when the target domain differs plied directly for the deep and detailed (sub)event from the biomedical one. annotation of Genia, however, it can recognize the Lastly we note that few differences (about 5.7%) negation and hedge state of chief statements (new in annotation can be obviously traced back to anno- knowledge). Note that there are in-sentence scope tation errors. detectors published (which achieve 58% strict F- measure) [11] for this task. Discussion BioScope for event modality detection Detailed event annotations We discussed in the previous section that the scopes Table 1a and 1b reveal that the biggest subset of of BioScope are not useful directly to the detec- the differences (60%) came from the issue that Genia tion of assertion and certainty state of Genia events, handles events within events as individual informa- however, we believe that using cue phrases in event tion sources while BioScope deals with constituent- modality detection can yield significant contribution. based text spans. An interesting question for consid- For instance, Kilicoglu and Bergler [4] constructed eration is whether the expected output of an infor- lexicons for speculation and negation keywords and mation extraction system consists of facts solely on introduced rules for recognizing the modality state of the basis of this textual evidence, where the trigger an event by utilizing the dependency path between for the event does not belong to the main statement the event clue phrase and the speculation/negation of the sentence/document. Note that the informa- cue. tion content of these events within events is usually Kilicoglu and Bergler employed hand-crafted lex- introduced and discussed in detail in other parts of icons for cue recognition, however, keywords are am- the document or in other publications or belongs to biguous, i.e. they express speculation and nega- the trivial domain knowledge. tion just in certain contexts. Hence a cue phrase 87 detection system is needed which classifies tokens mains is concerned, the annotation system in Bio- based on their local context then the dependency Scope seems to be more easily adaptable to non- paths between these predicted speculation/negation biomedical applications because of the high level of evidences and event triggers should be analyzed. domain specificity in the Genia Event annotation The BioScope corpus can be employed as a training system. dataset for general speculation/negation cue clas- As regards to the frequency of mismatch cate- sifiers. The state-of-the-art modifier cue detectors gories, we found that the largest amount of differ- achieve strict phrase-level F-measures over 80% [11]. ences is due to the issue that scopes aim to iden- Dependency-based rules defined for each (sub)type tify the negation/certainty status of the key event in of keywords can be also added to the system in order the sentence and each argument of these key events to determine the negative/speculative status of the (including arguments that are events themselves) is event. As future work, we plan to develop an event under scope as well in BioScope. In contrast, Ge- modality detector which uses BioScope as a training nia deals with the modality of events within events database for identifying speculation/negation cues independently. We think that the useful informa- and is enhanced by hand-crafted dependency-based tion for the biologist can be acquired from the key rules for determining the modality of the event. events, thus if we aim to detect ”new knowledge”, an automatic scope-detector trained on BioScope can contribute to biomedical information extraction. The usability of different annotation schemes On the other hand, BioScope cue phrases could be also employed to identify the assertion and certainty As discussed earlier, the annotation scheme of Bio- status of events. To reach this goal, we plan to Scope relies on linguistic principles while Genia develop a procedure which makes use of automat- Event is based on a more detailed annotation system ically recognized negation/speculation cues and em- specifically tailored to biological event annotation, ploys syntax-based rules (investigating the depen- where several complex relations are encoded be- dency path between the modality cue and the event tween participants of the events – often across clause cue) to classify the status of the event. boundaries. In this way, the annotation scheme of Genia Event is highly domain-specific and the corpus can be fruitfully utilized in biomedical information extraction, resulting in a deep and precise analysis of Authors contributions biological events though it might require a lot of ad- Veronika Vincze and Tomoko Ohta categorized the ditional work to adapt the system to other domains. mismatches between the two corpora. György Móra On the other hand, as the BioScope annotation implemented the software tools for gathering and vi- scheme is linguistic-based, scope- and cue-marking sualizing the differences. Richárd Farkas and György rules extracted from the corpus data can be more Szarvas carried out the statistical analysis of mis- easily exploited when developing negation/hedge de- matches. tectors in other domains as well. Acknowledgements This work was supported in part by the NKTH Conclusions grants (project codenames MASZEKER and TEX- In this paper, we discussed the differences between TREND) of the Hungarian government. The au- the linguistic-based and event-oriented annotation of thors would like to thank the annotators of the two negation and speculation in biological documents. corpora for their devoted efforts. We defined categories for the differences between the linguistic scope-based BioScope and the event- oriented Genia Event corpora. They have an in- tersection of documents (biological abstracts) which was randomly sampled, frequencies of mismatch cat- egories were estimated and resolution strategies were also offered for them. As far as information extraction in different do- 88 References Volume for Shared Task 2009:1–9, [http://www.aclweb. 1. Light M, Qiu XY, Srinivasan P: The language of Bio- org/anthology/W09-1401]. science: Facts, speculations, and statements in 13. Saurı́ R, Pustejovsky J: FactBank: a corpus an- between. In Proc. of Biolink 2004, Linking Biologi- notated with event factuality. Language Resources cal Literature, Ontologies and Databases (HLT-NAACL and Evaluation 2009, 43:227–268, [http://dx.doi.org/10. Workshop:) 2004:17–24. 1007/s10579-009-9089-9]. [10.1007/s10579-009-9089-9]. 2. Friedman C, Alderson PO, Austin JHM, Cimino JJ, Johnson SB: A General Natural-language Text Pro- 14. Kim JD, Ohta T, Tsujii J: Corpus annotation for cessor for Clinical Radiology. Journal of the Ameri- mining biomedical events from literature. BMC can Medical Informatics Association 1994, 1(2):161–174, Bioinformatics 2008, 9:10, [http://www.biomedcentral. [http://jamia.bmj.com/content/1/2/161.abstract]. com/1471-2105/9/10]. 3. Chapman WW, Chu D, Dowling JN: ConText: an al- 15. Pyysalo S, Ginter F, Heimonen J, Björne J, Boberg J, gorithm for identifying contextual features from Järvinen J, Salakoski T: BioInfer: a corpus for infor- clinical text. In Proceedings of the ACL Workshop on mation extraction in the biomedical domain. BMC BioNLP 2007 2007:81–88. Bioinformatics 2007, 8. 4. Kilicoglu H, Bergler S: Syntactic Dependency Based 16. Shatkay H, Pan F, Rzhetsky A, Wilbur WJ: Multi- Heuristics for Biological Event Extraction. In Pro- dimensional classification of biomedical text: To- ceedings of the BioNLP Workshop Companion Volume ward automated, practical provision of high- for Shared Task 2009:119–127, [http://www.aclweb.org/ utility text to diverse users. Bioinformatics 2008, anthology/W09-1418]. 24(18):2086–2093, [http://bioinformatics.oxfordjournals. org/cgi/content/abstract/24/18/2086]. 5. Aramaki E, Miura Y, Tonoike M, Ohkuma T, Mashuichi H, Ohe K: TEXT2TABLE: Medical Text Summa- 17. Pestian JP, Brew C, Matykiewicz P, Hovermale D, John- rization System Based on Named Entity Recogni- son N, Cohen KB, Duch W: A shared task in- tion and Modality Identification. In Proceedings of volving multi-label classification of clinical free the BioNLP 2009 Workshop 2009:185–192, [http://www. text. In Proceedings of the ACL Workshop on BioNLP aclweb.org/anthology/W09-1324]. 2007 2007:97–104, [http://www.aclweb.org/anthology/ 6. Medlock B, Briscoe T: Weakly Supervised Learn- W/W07/W07-1013]. ing for Hedge Classification in Scientific Liter- 18. Uzuner O: Recognizing obesity and comor- ature. In Proceedings of the ACL 2007:992–999, [http: bidities in sparse data. Journal of Amer- //www.aclweb.org/anthology/P/P07/P07-1125]. ican Medical Informatics Association 2009, 7. Szarvas Gy: Hedge Classification in Biomedical 16(4):561–70, [http://www.biomedsearch.com/nih/ Texts with a Weakly Supervised Selection of Key- Recognizing-obesity-comorbidities-in-sparse/19390096. words. In Proceedings of ACL-08 2008:281–289, [http: html]. //www.aclweb.org/anthology/P/P08/P08-1033]. 19. Councill I, McDonald R, Velikovich L: What’s great 8. Vincze V, Szarvas Gy, Farkas R, Móra Gy, Csirik J: and what’s not: learning to classify the scope The BioScope Corpus: Biomedical Texts Anno- of negation for improved sentiment analysis. In tated for Uncertainty, Negation and their Scopes. Proceedings of the Workshop on Negation and Specu- BMC Bioinformatics 2008, 9(Suppl 11):S9, [http://www. lation in Natural Language Processing, Uppsala, Swe- biomedcentral.com/1471-2105/9/S11/S9]. den 2010:51–59, [http://www.aclweb.org/anthology/W/ W10/W10-3110]. 9. Morante R, van Asch V, van den Bosch A: Joint memory-based learning of syntactic and semantic 20. Ganter V, Strube M: Finding Hedges by Chas- dependencies in multiple languages. In Proceedings ing Weasels: Hedge Detection Using Wikipedia of CoNLL 2009:25–30. Tags and Shallow Linguistic Features. In Proceed- 10. Özgür A, Radev DR: Detecting Speculations and ings of the ACL-IJCNLP 2009 Conference Short Pa- their Scopes in Scientific Text. In Proceedings of the pers 2009:173–176, [http://www.aclweb.org/anthology/ 2009 Conference on Empirical Methods in Natural Lan- P/P09/P09-2044]. guage Processing 2009:1398–1407, [http://www.aclweb. 21. Nawaz R, Thompson P, Ananiadou S: Evaluating org/anthology/D/D09/D09-1145]. a meta-knowledge annotation scheme for bio- 11. Farkas R, Vincze V, Móra Gy, Csirik J, Szarvas Gy: events. In Proceedings of the Workshop on Negation and The CoNLL-2010 Shared Task: Learning to De- Speculation in Natural Language Processing, Uppsala, tect Hedges and their Scope in Natural Lan- Sweden 2010:69–77, [http://www.aclweb.org/anthology/ guage Text. In Proceedings of the Fourteenth Con- W/W10/W10-3112]. ference on Computational Natural Language Learning 22. Krallinger M: Importance of negations and ex- (CoNLL-2010): Shared Task 2010:1–12, [http://www. perimental qualifiers in biomedical literature. In aclweb.org/anthology/W/W05/W05-0201]. Proceedings of the Workshop on Negation and Specu- 12. Kim JD, Ohta T, Pyysalo S, Kano Y, Tsujii J: Overview lation in Natural Language Processing, Uppsala, Swe- of BioNLP’09 Shared Task on Event Extraction. In den 2010:46–49, [http://www.aclweb.org/anthology/W/ Proceedings of the BioNLP 2009 Workshop Companion W10/W10-3108]. 89