This paper describes the pilot task Processing modality and negation, which was organized in the framework of the Question Answering for Machine Reading Evaluation Lab at CLEF 2012. This task was de ned as an annotation exercise consisting on determining whether an event mentioned in a text is presented as negated, modalised (i.e. affected by an expression of modality), or both. Three teams participated in the task submitting a total of 6 runs. The highest score obtained by a system was 0.6368 macroaveraged F1 measure.
is a grammatical category that allows to change the truth value of a
proposition. Research on modality and negation in Natural Language Processing (NLP)
has progressed thanks to the availability of data sets where extra-propositional
aspects of meaning are annotated, such as the certainty corpus (
The paper is organized as follows. Section 2 de nes the phenomena that are the focus of the task. Section 3 describes the task. Section 4 presents the participating teams and their results, and Section 5 puts forward some conclusions.
In the context of this task, negation is understood as a grammatical phenomenon used to state that some event, situation, or state of a airs does not hold. Negation can be expressed by a variety of grammatical categories, as shown in the examples below.2 { Nouns:
In the face of an international inability to put the sort of price on carbon use that would drive its emission down, an increasing number of policy wonks, and the politicians they advise, are taking a more serious look at these other factors as possible ways of controlling climate change. { Verbs:
A large gate at the front prevents people from sleeping rough in the disused courtyard. { Prepositions:
They simply asserted claims about Mr Obama without providing the court (or anyone else) with any convincing reason to believe those claims. { Adverbs:
The witnesses whom Ms Taitz called to testify (you can read them here, in the transcript) were never tendered as experts. { Determiners:
They usually have no experience of the company's products or markets. { Pronouns:
None of these measures has come close to solving the problem. { Pre xes:
The new pact has left some important problems unsolved. 2 In the examples below cues are marked in bold and the negated and/or modalised event is underlined. { Conjunctions:
Neither the decision nor the changes themselves were based on anyones political beliefs or ideology.
In the examples above negation is expressed explicitly, but it can also be expressed implicitly, for example, by means of certain linguistic contexts: { Conditional constructions:
If matter and antimatter were truly symmetrical, then they would have come into existence in equal amounts during the Big Bang.
In this sentence the conditional construction determines that the events expressed by were and come into existence are implicitly negated. { Combination of certain types of verbs and verb tenses:
The process to determine the Democratic nominee was supposed to have ended four years ago.
In this case, the use of the verb suppose and the past perfect tense indicates that the event 'end' has not happened.
A description of negative contexts is presented in (
From a theoretical perspective, modality can be de ned as a philosophical concept, as a subject of the study of logic, or as a grammatical category. There are many de nitions and classi cations of modal phenomena. For this task we understand modality in a broad sense. Modality will not only refer to epistemic modality (typically expressed by modal verbs), but also to concepts such as hedging, uncertainty, factuality, evidentiality, and subjectivity. These concepts are related to the expression of the attitude of the speaker towards her statements in terms of degree of certainty, reliability, subjectivity, sources of information, and perspective.
Epistemic modality, as described by Lyons (10, p.793), is concerned with
matters of knowledge and belief, \the speakers opinion or attitude towards the
proposition that the sentence expresses or the situation that the proposition
describes". Palmer de nes two types of propositional modality: epistemic, used by
speakers \to express their judgement about the factual status of the
proposition, and evidential, used to indicate the evidence that they have for its factual
status (13, p.8{9). The term hedging is originally due to Lako , who describes
hedges as \words whose job is to make things more or less fuzzy" (8, p. 195).
Evidentiality is related to the expression of the information source of a statement
(1, p.1). Certainty is a type of subjective information that can be conceived of
as a variety of epistemic modality (
In the context of this task, we will consider that an event is modalised when it is not presented as certain or factual. Many linguistic devices can be used to present an event as modalised.
{ Modal verbs:
Mr Sakurai fears many of Minamisomas evacuees may never come back. These alternatives could also improve the content and prospects of other climate action.
Global greenhouse-gas emissions must fall by half to limit climate change. { Epistemic adjectives:
Providing most of that energy from wind, sunshine, plants and rivers, along with a bit of nuclear, is possible. { Epistemic adverbs:
It will probably never again generate the majority of America's energy. { Epistemic nouns:
Insiders reckon the possibility of being let o the hook by a new Administration. { Propositional attitude verbs and adjectives:
We do not believe these attacks breached the servers that support our Domain Name System network.
We hope to unveil it before the month is up. The ECB was considering writing down the value of its Greek bonds to the price it paid for them { Generics and habituals:
American universities are usually happy to accept such good students.
There is a big di erence between drawing a map and following it. { Future tense:
They will start to decide that its not worth the money. { Conditional constructions:
If you are highly motivated to minimise your taxes, you can hunt for every possible deduction for which youre eligible.
The investment required to decarbonise power would average about 30 billion ( $ 42 billion) a year over 40 years. { Expression of purpose/goal:
Europe has set a goal of reducing emissions by 80-95% by 2050. The investment required to decarbonise power would average about 30 billion ( $ 42 billion) a year over 40 years. { Expression of need:
China has less urgent need to bolster growth. { Expression of obligation:
All that gassy baggage must go.
Rich countries should cut the most. { Expression of desire: They want it raised to 30%. { Epistemic judgment verbs:
Suggesting that such a large number of Americans are doing a job that is no longer necessary was perhaps not the wisest move politically. We can assume that this has probably been known about since the beginning of this century. { Epistemic evidential verbs:
Turkey seems to favour a rival Russian-backed project. { Epistemic deductive verbs:
Some deduce from the overall picture that as China and other authoritarian states get more educated and richer, their people will agitate for greater political freedom, culminating in a shift to a more democratic form of government.
More information about modality can be found in the study by Portner (
The exercise Processing modality and negation is de ned as an annotation task in which systems have to determine whether an event mentioned in a text is presented as negated, modalised (i.e. a ected by an expression of modality), or both. This information can be relevant for machine reading systems, since negated and modalised events should be treated di erently than factual events in the inference making process. The term event is understood in a broad sense to refer to events and states.
The input for a system is a text where all events expressed by verbs are identi ed automatically. The identi cation of verbs is done automatically with the Stanford POS Tagger (v. 3.0, 2010-05-10) (24; 23). Verbs that are not identi ed by the tagger are not marked. Only the main verb of a verbal form is marked. The output should be a label per event. The possible values of the label are four: MOD, NEG, NEGMOD, NONE.
{ An event is assigned the tag NONE when it is presented as certain and it happened (e.g., Half of Europe's electricity comes from fossil fuels). { An event is assigned the tag NEG when it is presented as certain and did not happen (e.g., Half of Europe's electricity does not come from fossil fuels). { An event is assigned the tag MOD when it is not presented as certain and is not negated (e.g., Half of Europe's electricity might come from fossil fuels). { An event is assigned the tag NEGMOD when it is not presented as certain and is negated (e.g., Half of Europe's electricity might not come from fossil fuels).
Figure 1 shows an example of an input sentence and the output expected from a system.
INPUT Some <event id=1>deduce</event> from the overall picture that as China and other authoritarian states <event id=2>get</event> more educated and richer, their people will <event id=3>agitate<event> for greater political freedom, <event id=4>culminating</event> in a shift to a more democratic form of government. OUTPUT e1=NONE e2=MOD e3=MOD e4=MOD
The organization provided one example document with gold annotations and 8 test documents in English for evaluation, two for each of the topics of the QA4MRE task: AIDS, climate change, music and society, and Alzheimer's disease. The test documents are articles from the journal The Economist.3 The gold annotations of the test documents were released when the evaluation period nished. No training documents were provided for this edition. The total number of events and the class distribution is presented in Table 1. The event labels are distributed similarly across topics, with the exception of MOD, which is more frequent in the climate documents.
Topic
Aids Alzheimer Climate Music
All # Events NONE % MOD % NEG % NEGMOD % 247 139 56,27 95 38,46 5 2,02 8 3,23 180 99 55,00 65 36,11 9 5,00 7 3,88 590 264 44,74 269 45,59 33 5,60 24 4,06 247 139 56,27 95 38,46 5 2,02 8 3,24 1244 655 52,65 474 38,10 64 5,14 41 3,29
Table 1. Number of events and class distribution.
In order to solve the task, participants could use any existing resources such as corpora, lexicons or NLP tools such as factuality pro lers, scope resolvers, hedge and negation detectors, etc. The only requirement was that the task be solved automatically. 3 The Economist kindly made available the texts for non-commercial research purposes.
The output of systems was evaluated against a manually annotated gold standard. The test documents were annotated by two annotators. As indicated above, the labels to be annotated were four: NONE, MOD, NEG, NEGMOD. The annotators used the decision tree presented in Figure 2 in order to decide which label was to be assigned to an event.
3 groups participated submitting 6 runs. Table 2 shows the list of participating teams and the reference to their reports.
The task was evaluated at the event level in terms of F1. The highest score obtained by a system was 0.6368 macroaveraged F1measure. Table 3 indicates the number of runs submitted per team and the highest macroaveraged F1obtained. A baseline was calculated by assigning the majority class (MOD) to all events, which scored 0.1380. All submitted runs score above baseline.
Team # of runs Highest F1 CLaC 2 0.6368 desancis 3 0.5339 JUCSENLP 1 0.3378 baseline (all MOD) - 0.1380
Table 3. Number of runs and highest scores per team.
The CLaC team extended NEGATOR, a heuristics-based system initially developed to process negation. The system is composed of three modules: the rst component detects and annotates negation and modality cues; the second component detects and annotates the syntactic scope of the detected negation and modality cues based on information from the dependency graph. The third component determines which of the annotated events are within the detected scopes of negation, modality or within the intersection of both in order to assign the event labels.
The desancis team developed also a rule-based system de ned in JAPE (Java Annotation Patterns Engine). The system is composed of three modules: the VG module tags verbal groups with lexical category, mode, tense, aspect, voice, and modality; the MODNEG module tags particles that may be related to modality and/or negation; the LABELER module tags the event under analysis by means of rules that use contextual information about modality and negation.
The JUCSENLP team developed a system that relies on a list of modality and negation cues. The label assigned to an event depends on whether the event is preceded in the sentence by the modality and negation cues contained in the list. More information about the systems can be found in the individual description papers referenced in Table 2.
The scores per run are provided in Table 4 in terms of overall macroaveraged F1, overall accuracy and F1 per label. Considering that modality and negation tagging is a recently emerged task and that the corpus provided by the organization is small, we consider that the results obtained by systems are encouraging.
Run CLaC 1 CLaC 2 desancis 1 desancis 2 desancis 3 JUCSENLP 1
In this paper we described the task Processing modality and negation organized in the framework of the QA4MRE Lab at CLEF. The task was de ned as an annotation exercise in which systems had to determine whether an event mentioned in a text is negated and/or modalised. Participants were provided with 8 test documents, 2 per each of the following topics: AIDS, music, climate change, Alzheimer's disease. No training documents were provided, since the task was not intended as a machine learning task. The three teams that participated submitted heuristics based systems that outperformed a majority class baseline. Taking into consideration that this is a new task and that the task is di cult, we consider that the results obtained by participants with heuristics based systems are encouraging.
This study was made possible through nancial support from the University of Antwerp (GOA project BIOGRAPH). We are grateful to the organizers of the QA4MRE lab at CLEF 2012 for their support and for hosting the pilot task. We also thank Janneke van de Loo for helping with the data annotation.