English. We have produced a corpus annotated for modality which amounts to approximately 20,000 words in English, French, and Italian. The annotation scheme is based on the notion of epistemic construction and virtually languageindependent. The annotation is rigorously evaluated by means of a newly developed strategy based on the alignment of the entire epistemic constructions as identified and marked up two annotators. The corpus and the agreement scoring tools are publicly available.
Modality is a pervasive phenomenon crucial to
language understanding, analysis, and automatic
processing
As far as the detection of polarised judgments
goes, there have been substantial annotation
efforts in recent years, exemplified by recurring and
increasing sentiment analysis tasks within the
context of the Semeval evaluation campaign.1
Attention has also been given to more specific
factuality tasks such as the CoNLL-2010 Shared Task on
identifying hedges
Indeed, a major bottleneck in the creation of
modality-annotated resources is the very notion of
modality itself, as encapsulating this phenomenon
in one exhaustive but workable definition is far
from trivial
first multilingual corpus annotated for modality. The annotation scheme is virtually languageindependent, and the annotation is evaluated according to a specifically designed methodology 1http://alt.qcri.org/semeval2017/ index.php?id=tasks. Note that in 2017 within the sentiment analysis track there was also a task on truth detection, which goes to show how closely related the two phenomena indeed are. which is portable to other tasks where annotators are left with substantial freedom in the selection of the tokens to be marked up. The corpus and the tools for scoring agreement are publicly available (http://modal.msh-vdl.fr/,https:// bitbucket.org/lennyklb/modality/). the constructions to be annotated without controlling for any pre-selection, incurs the risk of a wide range of choices, and substantially low agreement. We discuss this in the Evaluation section. In the remainder of this section we explain the scheme and the procedure we used to annotate the corpus.
The MODAL Corpus is the first corpus of dialogues in multiple languages annotated for phenomena of (epistemic) modality.
MODAL consists of three equivalent resources
of English, French and Italian dialogues. These
were drawn from the Santa Barbara Corpus of
Spoken American English
In the construction of MODAL, we were guided by two main principles: maximum expressivity, and cross-lingual validity. We therefore took an approach to annotation that would simultaneously ensure both.
Specifically, we did not want to annotate a predetermined list of epistemic constructions and assign functions to them. Indeed, this would make the scheme very much language-dependent, as specific tokens/constructions would need to be identified for each language. Additionally, it would restrict the annotation to this pre-selection, which could not be exhaustive.
As an alternative approach, we provided a theoretical meaningful, and operationalisable definition of epistemic modality. On this ground, thus only at a later stage, the annotators identified the linguistic constructions that realise epistemic modality in the three different languages. Thus, rather than going from constructions to functions, we go from functions to constructions.
While this approach has the advantage of being valid cross-linguistically, and maximising expressivity, it also potentially has a major problem. Letting the annotators choose freely the tokens and
We employed a two-fold procedure: epistemic constructions are first identified, and then annotated with their features.
Identification of epistemic constructions In
order to annotate epistemic modality in dialogues,
we subscribed to a communitarian
Consequently, we annotated not only constructions in which a marker is realized by a more grammaticalized element, such as a modal verb (Example 1), but also constructions in which a marker is realized lexically (Example 2) or prosodically (Example 3):
A penguin might lay two eggs and at that point [. . . ]
Besides, we annotated both monological epistemic constructions in which a marker expresses the evaluation of the truth-value of a scope by a single speaker (Example 4), and dialogical epistemic constructions in which two or more markers are used to negotiate the evaluation of the truth-value of one and the same scope among the participants in a conversation (Example 5): apparently it was very very muddy it was abnormally warm and it was just a big mudbath out there [. . . ] ALIC: I don’t think Darren put anything on it .
NICO: Mhm .
ALIC: Right .
ALIC: Okay . (1) (2) (3) (4) (5)
represented the epistemic constructions identified in the corpus as triadic constructions consisting of a marker, a scope and a relation between the marker and the scope, as shown in Figure 1.
We formalised the marker, the scope and the relation between them as three elements each endowed with its own formal and functional properconstruction
modal relation [Probablymarker] [it is the postmanscope] ties. Each element is then annotated with syntactic, semantic, and pragmatic features according to the developed annotation scheme.
Building on
Operationalising the annotation task From a
theoretical perspective, the scheme is grounded in
the Construction Grammar framework
The originality of the general approach and of the
annotation procedure led us to develop an
origi2Further information regarding the distribution of
categories and examples is available at the project’s
website (http://modal.msh-vdl.fr/ and in (Pietrandrea,
forthcoming)).
nal technique for testing the inter annotator
agreement, essentially based on the percentage of
overlap between the spans of text identified as markers
or scopes by the annotators
Annotators can identify any textual element as part of a modalised construction, and each annotator works on their own file. This means that in order to assess agreement, we first need to try to align the constructions marked up in the two files. We do so via anchors. Anchors can be aligned iff: they are of the same type (marker or scope) they overlap in content by at least a given proportion of lexical material, which we base on character offset. For example, for a required overlap of 50% and a token length of an anchor A of ten tokens, the content of the candidate anchor from the other file needs to have at least five subsequent words in common with A.
This process results in a collection of pairs of aligned anchors. For example, considering annotator a and annotator b, we would have an aligned pair of marker ta and marker tb.
The final step is to iterate through the relations that judge a introduced and align them with relations that judge b introduced. In order to explain the procedure of further alignment to relations, we take judge a as reference, but in terms of scores it doesn’t make any difference which direction we go, since precisionab = recallba so that eventually f scoreab = f scoreba. Relations consist of a marker and one or multiple scope portions. Aligning relations is done by pairing up markers and scopes into relations introduced by judge a and check if the aligned counterparts of these markers and scopes by judge b are part of a relation as well. When this is the case, we deem the two constructions as “the same”.
Next, we have to assess agreement on the
features assigned to relations and markers. While
agreement over alignment is measured using
precision/recall/f-score as we have to deal with
potentially different spans, for the relations’ and
markers’ features, we can then use Cohen’s Kappa
Because of freedom in the annotation of the extension of anchors, as mentioned above we evaluated alignment at different percentages of overlap. The scores for the alignment of scopes for all three languages is shown in Table 3. While for Italian we observe that even when evaluating alignment of full strings (i.e. requiring 100% overlap), the agreement stays high, this is not the case for French and English. Indeed, if complete overlap of scopes is required to deem the annotations equivalent, F-scores drop quite a bit. We do not include a table for the scores on the markers as they do not change substantially with varying degrees of overlap. This is due to the fact that markers are often just single words, or very short anyway. Fscores range from 0.91 at 10% and 0.90 at 100% for English, from 0.86 at 10% and 0.85 at 100% for French, and stay stable at 0.94 for Italian. For this reason, we can be lenient with markers’ alignment, which we set at 10% when evaluating relations. Interestingly, though, we can observe that when evaluating agreement on the features, Kappa increases when stricter alignment is required (Table 4). This is likely due to the fact that on fully agreed upon strings, the assigned features are also agreed upon.
For the Italian annotation of the relation’s features, at overlap 100%, we observe K = 0:86 for the FUNCTION feature, K = 0:82 for TYPE, and K = 0:72 for POLARITY.3
To provide a more detailed view into the
disagreements of the type feature, for instance
(whose final agreed upon distribution was reported
in Figure 2 above), in Figure 4 we show the
confusion matrix for Italian. We can observe that the
largest number of confusions arise from mixing up
the categories indirect inferential and
no evidence. Indeed, the precise delimitation
between these two categories is a long- and
hotdebated issue in the literature on epistemic
modality
Overall, we can see that our annotation, albeit granting the annotators a lot of freedom, is substantially reliable.4
3Very similar scores are observed at different degrees of overlap.
4Please note that for all agreement results, for all languages, the reader is referred to the project’s website, where dir aud dir vis ind inf ind rep
mem no ev quot 1 Modality can be reliably annotated in multiple languages by taking a bottom-up, functional approach paired with a solid annotation scheme, trees to guide the annotators’ decisions, and a rigorous evaluation strategy. With this approach, we have produced the first multilingual corpus annotated for modality, which can be potentially used to train modality detection models as well as to further study modality itself. By making all of the data publicly available, and by sharing our annotation experience, we also hope to provide a blueprint for creating modality-annotated resources in yet more languages.
The research described in this paper was supported by the Maison des Sciences de l’Homme du Val de Loire (grant MoDAL 2015), the IRCOM Consortium, the “Laboratoire Lige´rien de Linguistique – UMR7270”, and the Groningen Meaning Bank. The authors are also grateful to the anonymous reviewers for their comments. examples of disagreement are also included (http:// modal.msh-vdl.fr/).