This paper provides an overview of challenges that occurred during the creation of the annotation scenario for the attitude annotation in diplomatic speeches of the UNSC. The scenario followed the attitude part of the Appraisal theory. The various challenges in annotating the speeches such as the extent of arguments, identification of attitude in verbal forms, and complex structures, were classified, and, in part, resolved. This paper would be useful for anyone considering this type of attitude analysis when working with diplomatic and political texts.
types of discourse. The prominent characteristics of these
texts are the understated tone [
Another problematic side of this annotation type is In our previous publications, we have discussed the
that the researchers often omit the step of creating a notion of attitude in diplomatic discourse and described
formalized annotation scenario, which is why motivation our view on the most suitable annotation schemes for
and consistency of their choices are hard to follow [
This study aims to present a discussion on the issues provided the outcomes of the first batch of annotation
observed while creating the discourse-specific annota- [
Diplomatic speeches form a very particular and pe- our calculations and commentary on the Cohen’s kappa culiar group of texts that are very diferent from other which varies between 0.44 and 0.32 depending on the selected category.
The objectives of this research are to define and exemplify the challenging parts of the annotation scenario and ofer solutions for the observed issues to make the annotation process reliable and stable, resulting in a satisfactory inter-annotator agreement. The clarity and reproducibility of the annotation process would also help further automatize the processing of attitudes.
The structure of the paper is outlined in three main ITAT (Information technologies – Applications and Theory), Workshop on Automata, Formal and Natural Languages 2024 (WAFNL 2024) * Corresponding author. $ anisimova@ufal.mf.cuni.cz (M. Anisimova); sarka.zikanova@mf.cuni.cz (Š. Zikánová) https://ufal.mf.cuni.cz/mariia-anisimova (M. Anisimova); https://ufal.mf.cuni.cz/sarka-zikanova (Š. Zikánová)
0000-0002-2478-2815 (M. Anisimova); 0000-0002-7805-9649 (Š. Zikánová)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) sections, namely the brief description of the annotation scenario, the detailed description of the issues observed, a brief outline of the future work, and the main conclusions.
Sections 5 and 6 correspond to the Limitations and Ethical considerations of the presented research. more convenient to start with, while gradually adapting the evaluation criteria to the task.
2. Description of the annotation nario creation has first shown a set of challenges, which
scenario vary in nature, and dificulty and have proven to be either
general to this type of text analysis [
The subsystem of attitude according to the Appraisal sence of an attitude based on the token’s polarity, as if theory provides a framework for the analysis of evalua- judging each token individually by deciding if it may tive expressions by categorizing them as being an afect signify the presence of an attitudinal expression. Such (an emotional reaction), a judgement (an expression of an approach, however, does not allow for capturing all ethical evaluation), or an appreciation (an evaluation of of the available attitudinal meanings. Let’s take a look at aesthetics) and defining their polarity and explicitness. Example 1. and analyze it by first highlighting all of the Each category is then subdivided into a separate tree of positive and negative connotations available to estimate choices making the system a complex and informative if the available sentiment in separate tokens correlates structure. with the identification of expressions of attitude.
The dataset that is to be annotated, is also described Example 1: Even items subject to control would go to Iraq in detail in the scenario. It consists of a corpus of 100 once there is [confidence: POS] that they would not be used diplomatic speeches, selected according to a set of crite- to rebuild Iraq’s [[weapons: NEG] of [mass [destruction: ria that allows for proportional representation of diver- NEG]: NEG]] or [[improve: POS] its military capabilities]. sity according to the topic of a speech, the year of the Such complex lexical structures may contain multiple meeting, and the country the speaker represents. The layers of semantic connotations, which may lead to the language of the data is English, and the speeches were annotator’s confusion due to the superposition of the laeither originally presented in English, or were the ofi- bels. The annotators may be drawn to find the attitudinal cial UN translations. The information about the original meaning of every available token conveying attitudinal language of the speech, as well as the speaker’s afilia- polarity which often leads to viewing an annotated fragtion and sex, the topic of the session, and its year are ment in isolation of the broader text meaning. stored in the metadata of each text. The corpus consists Another approach to attitude identification is based of 105592 tokens and 7296 types, and the average length on the subjective evaluation of a textual fragment under of a sentence is 32 tokens. consideration based on the contextual boundaries of the
Our initial goal was for the guidelines to be of a qual- syntactic structure that bounds it, as well as the context ity that would allow for the full disambiguation of the of the whole text (which in the case of our study is a task, which means the annotators would select the same speech). text span and assign completely identical labels to it in Let us take another look at the same sentence, where terms of attitude type and subtype, sentiment polarity, the possible locations of attitude are defined by its suband explicitness. We therefore apply the strictest ap- jective interpretation and context of the whole structure: proach to annotations assessment at the time as it is
Even items subject to control would go to Iraq [once Let’s analyze an excerpt from one of the diplomatic there is confidence that they would not be used to rebuild speeches provided in Example 4: Iraq’s weapons of mass destruction or improve its military Example 4: Experts in our countries must immediately capabilities: JUDGEMENT-INVOKED]. begin to analyse it and then draw the appropriate conclu
Here, the speaker provides an implicit evaluation of the sions. future possible behaviour of Iraq, expressing their hope Here, there is a definite presence of positive judgement and inclination towards Iraq not rebuilding its weapons coded by the directive ’must’. experts must begin to analof mass destruction. This scenario is presented as a sub- yse would be our first choice to annotate. The problem jective evaluation of the possible future behaviour, which is, such a fragment, if non-explicit, does not convey any means that the attitude in question should be perceived attitude on its own, therefore inclusion of the whole conas an implicit evaluation of the desired appropriate action text necessary for understanding of the chosen label is of Iraq. advised: beginning of analysis is only good if it leads to
Even though this way of finding attitudes leads us drawing appropriate conclusions. to a somewhat more vague definition of the borders of the annotated expression, it also provides us with more 2.3.1. On annotating articles contextual meaning as opposed to the cumulative polarity of separate tokens.
Let us exemplify this approach again with the Example 2:
Example 2: [A number of these have been under discussion for some time in this Council: AFFECT-INVOKED], and [if we were able to agree on this package, it would be an important step forward in that regard as well: JUDGEMENTINVOKED].
In this example, the speaker implies their emotional state of unsatisfaction by the amount of time that an issue is already under discussion while providing their implicit evaluation of the appropriate course of future actions.
Our preferred approach is to judge the span of the annotated fragment by the context and the explicitness of the evaluation, which would lead to a clearer logic and accountability of the annotation process.
Attitude examples observed in the corpus were proven
to make use of both explicit and implicit (inscribed and
invoked) ways of expressing an attitude. The diplomatic
discourse is known for its indirectness [
The inscribed attitudes only require the explicitly subjective tokens to be annotated, as shown in Example 3:
Example 3:France [fully supports: AFFECT-INSCRIBED] the search for a political solution.
This decision would lead to a suficient increase in the expected average length of the annotated fragment for the invoked annotated attitudes in comparison to the inscribed attitudes.
Our main solution for annotating invoked attitudes is to capture as much meaning as possible by annotating as few tokens as needed.
determiners. As most of the annotated entities constitute either an evaluative adjective or a collocation of an evaluative adjective and an evaluative noun, it is necessary to decide whether or not articles defining the evaluative phrase should be included in the annotation or not.
Our solution to this question is related to the overall viewpoint on the attitude framework and depends on t
If an attitude is expressed explicitly (inscribed tag) as in ’a capable management’, ’we are happy about the resolution’, ’the valuable lesson’, etc.) and the perceived meaning of the adjective+noun collocation is enough to understand it in the full, annotation of any additional tokens should be omitted. Usually, it is enough to only mark one token or collocation that is used for expressing an attitude. There is no need to include any additional tokens in the annotated fragment, for example, if an attitude is found within a phrase [they have thought of a wonderful solution], we would advise only marking the evaluative adjective and omit to annotate the article that precedes it and the noun that follows. We also advise against article annotation when a noun that is in collocation with the attitude-bearing adjective has no attitudinal meaning itself (the unfortunate circumstances).
However, if an attitude is a part of the superlative form (the best) or is expressed implicitly and the broader context is necessary for understanding the expression, any needed number of tokens within one syntactic clause could be considered (as in the veto today will not prevent that).
Another situation that may cause confusion and subsequent mistakes in annotation includes the cases, where the two annotated fragments overlap. An overlap happens if one attitude occurs within the scope of the annotated fragment of another attitude. This may happen in many ways, however, if there are only two attitudi- has led to a parallel annotation experiment as well as nal elements, the overlap may look one of the following to the creation of a background for the newly-published ways: corpus of evaluation in diplomatic speeches of the UNSC 1) The first option is for the second label to overlap (CoDipA 1.0). with the first one as in (a(b)); We are specifically interested in investigating the atti2) The second option is for the first element to overlap tude development processes throughout the selected time with the second one as in ((a)b); frame and in comparing the findings on an inter-conflict 3) The third option is the inclusion of an element (b) scale. Another expected development is the practical in the middle of the first element (a) following a scheme application of the acquired data on a fine-tuned large lan(a(b)a) as in Example 5: guage model (such as Bert or GPT 4) to establish whether
Example 5: There is no doubt that we all want there is a potential for expanding the analysis to a bigger [Afect-inclination] to resolve this problem [Judgement- scale. tenacity]. The results of both quantitative and qualitative eval
Here, the whole sentence is an invoked judgement with uation of the annotation outcomes are expected to be an inclusion of an inscribed afect (in bold) corresponding published as well. to the expression of the speaker’s desire.
To technically deliver this solution it is necessary to
allow for overlapping annotation in doccano [
The issue of interrupted annotation refers to a situation, where a span of attitudinal text includes a sequence of non-attitudinal tokens. Annotating unnecessary tokens may lead to a decrease in the quality and reliability of the future corpus. As per our observations, such an annotation scenario occurs solely with the explicitly formulated attitudes, therefore deciding on this type of label is advised as a first step.
Here is an example of an interrupted attitude (Example 6), an excerpt from another diplomatic speech. An invoked judgment of the previous lack of actions of the Security Council is interrupted by a referral to one of the resolutions, which does not add any additional attitudinal meaning to the annotated fragment and should therefore be excluded from it.
Example 6: Unfortunately, in the past hundred days, the very [limited suspension of the sanctions: JUDGEMENTINVOKED-PART1] established by the Security Council resolution 943 (1994) [has also not been entirely fulfilled: JUDGEMENT-INVOKED-PART2].
Our solution to resolving it is purely technical and includes the creation of an additional label [none] that is to be assigned to the tokens excluded from the annotated fragment.
challenging parts of annotating attitudes according to the
Appraisal theory scheme [
The first problem that an annotator would encounter if they would seek attitude in diplomatic speeches, would be attitude identification and deciding on the textual borders of an attitudinal fragment. We have outlined these processes and provided examples from our corpus.
In the next sections, we have ofered an overview of the technically challenging parts of the attitude annotation process, such as overlapping and interrupted types of attitudes, as well as their solution.
This paper presents an up-to-date analysis of the ongoing research based on the annotation project, including the development of a discourse-specific annotation scenario. The findings from the presented annotation scenario should not yet be considered as final, they may be updated in the final version of the scenario before its publication.
01/207PROV and the Czech Science Foundation (project no. 24-11132S, Disagreement in Corpus Annotation and Variation in Human Understanding of Text), and partially supported by SVV project number 260 698; a part of the used data comes from the project no. LM2018101 by the Czech Ministry of Education, Youth and Sports (Digital Research Infrastructure for Language Technologies, Arts and Humanities).