=Paper=
{{Paper
|id=Vol-3792/paper10
|storemode=property
|title= Problematic Cases of Attitude Annotation in Diplomatic Speeches
|pdfUrl=https://ceur-ws.org/Vol-3792/paper10.pdf
|volume=Vol-3792
|authors=Mariia Anisimova,Šárka Zikánová
|dblpUrl=https://dblp.org/rec/conf/itat/AnisimovaZ24
}}
== Problematic Cases of Attitude Annotation in Diplomatic Speeches
==
https://ceur-ws.org/Vol-3792/paper10.pdf
Problematic cases of attitude annotation in diplomatic
speeches
Mariia Anisimova1,* , Šárka Zikánová1
1
Charles University
Faculty of Mathematics and Physics
Institute of Formal and Applied Linguistics, 6 Malostranské náměstí 25, 118 00 Praha, Czech Republic
Abstract
This paper provides an overview of challenges that occurred during the creation of the annotation scenario for the attitude
annotation in diplomatic speeches of the UNSC. The scenario followed the attitude part of the Appraisal theory. The various
challenges in annotating the speeches such as the extent of arguments, identification of attitude in verbal forms, and complex
structures, were classified, and, in part, resolved. This paper would be useful for anyone considering this type of attitude
analysis when working with diplomatic and political texts.
Keywords
Appraisal theory, attitude analysis, manual annotation, diplomatic discourse, corpus linguistics
1. Introduction types of discourse. The prominent characteristics of these
texts are the understated tone [5] and indirectness. These
The research on annotating attitudes [1] and its various pragmatic features prove to be very important to how
categories has long been an ongoing process [2], [3]. As diplomats express opinions, which are most frequently
described before, annotating attitudes is a complex pro- not of their own but of the political body they represent.
cedure not only due to elaborated annotation schemes It is also because of them, that the diplomatic attitudes
but also due to the lack of definitive criteria for the iden- form a separate group of attitude-bearing expressions
tification and categorization of attitudes and the other and require a comprehensive approach in the process of
appraisal labels. annotation.
Another problematic side of this annotation type is In our previous publications, we have discussed the
that the researchers often omit the step of creating a notion of attitude in diplomatic discourse and described
formalized annotation scenario, which is why motivation our view on the most suitable annotation schemes for
and consistency of their choices are hard to follow [2]. its evaluation [6], explained the annotation process and
This, in turn, causes issues with testing the consistency environment, as well as the criteria for selecting the data
of the annotations, and test-retest reliability [4]. for our corpus of diplomatic speeches. We have then
This study aims to present a discussion on the issues provided the outcomes of the first batch of annotation
observed while creating the discourse-specific annota- [7], which was then utilized for redefining the annotation
tion scenario suited for annotating types of attitudes and scenario based on problematic and unclear cases of anno-
their subcategories as defined by [1]. We classify the tation. In the most recent addition to our project [8], we
issues and particularities observed during the creation have also presented the Corpus of Diplomatic Attitudes
of the scenario as well as the annotation process and (CoDipA 1.0), which includes a description of the selected
compare them to the issues observed by [2]. We then speeches from the United Nations Security Council anno-
offer our perspective on the choice of a possible solution tated with the attitude part of the Appraisal theory and
and discuss further experiments. expands on corpus and inter-annotator statistics, such as
Diplomatic speeches form a very particular and pe- our calculations and commentary on the Cohen’s kappa
culiar group of texts that are very different from other which varies between 0.44 and 0.32 depending on the
selected category.
ITAT (Information technologies – Applications and Theory), Workshop
The objectives of this research are to define and ex-
on Automata, Formal and Natural Languages 2024 (WAFNL 2024)
*
Corresponding author. emplify the challenging parts of the annotation scenario
$ anisimova@ufal.mff.cuni.cz (M. Anisimova); and offer solutions for the observed issues to make the
sarka.zikanova@mff.cuni.cz (Š. Zikánová) annotation process reliable and stable, resulting in a sat-
https://ufal.mff.cuni.cz/mariia-anisimova (M. Anisimova); isfactory inter-annotator agreement. The clarity and
https://ufal.mff.cuni.cz/sarka-zikanova (Š. Zikánová)
reproducibility of the annotation process would also help
0000-0002-2478-2815 (M. Anisimova); 0000-0002-7805-9649
(Š. Zikánová) further automatize the processing of attitudes.
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License
Attribution 4.0 International (CC BY 4.0).
The structure of the paper is outlined in three main
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
sections, namely the brief description of the annotation more convenient to start with, while gradually adapting
scenario, the detailed description of the issues observed, a the evaluation criteria to the task.
brief outline of the future work, and the main conclusions.
Sections 5 and 6 correspond to the Limitations and Ethical 2.1. On identifying attitudes and the
considerations of the presented research.
scope of annotated fragment
The pilot annotation task completed to support the sce-
2. Description of the annotation nario creation has first shown a set of challenges, which
scenario vary in nature, and difficulty and have proven to be either
general to this type of text analysis [2], or particular to
The annotation scenario was developed to annotate the the chosen text type.
attitudes [1] in the diplomatic speeches of the United The first task that the annotator should be able to do is
Nations Security Council [9]. The scenario first offers a to find the attitudinal fragments. This task is not straight-
brief introduction to the Appraisal theory by first describ- forward and lacks direct instructions and explicit descrip-
ing its three main subsystems: attitude, engagement, and tion in the original Appraisal theory documentation [1].
graduation. The three subsystems refer to the descrip- This has been posing a general problem for researchers
tion of our feelings and emotional reactions, sourcing of working with various discourse types [2], in diplomatic
attitudes and the play of voices around opinions in dis- discourse this issue is also of critical importance.
course, and grading the phenomena [1]. It then focuses
on the detailed description of the subsystem of attitude,
which was selected to define emotion and subjectivity
2.2. Attitude identification in diplomatic
in diplomatic speeches. Finally, a practical how-to guide texts
for setting up the annotation environment in the doc- From our empirical experience, there can be several ap-
cano annotation tool [10] is offered along with a set of proaches to the process of identification of an attitude.
annotation labels and references. It is, first of all, possible to judge the presence or ab-
The subsystem of attitude according to the Appraisal sence of an attitude based on the token’s polarity, as if
theory provides a framework for the analysis of evalua- judging each token individually by deciding if it may
tive expressions by categorizing them as being an affect signify the presence of an attitudinal expression. Such
(an emotional reaction), a judgement (an expression of an approach, however, does not allow for capturing all
ethical evaluation), or an appreciation (an evaluation of of the available attitudinal meanings. Let’s take a look at
aesthetics) and defining their polarity and explicitness. Example 1. and analyze it by first highlighting all of the
Each category is then subdivided into a separate tree of positive and negative connotations available to estimate
choices making the system a complex and informative if the available sentiment in separate tokens correlates
structure. with the identification of expressions of attitude.
The dataset that is to be annotated, is also described Example 1: Even items subject to control would go to Iraq
in detail in the scenario. It consists of a corpus of 100 once there is [confidence: POS] that they would not be used
diplomatic speeches, selected according to a set of crite- to rebuild Iraq’s [[weapons: NEG] of [mass [destruction:
ria that allows for proportional representation of diver- NEG]: NEG]] or [[improve: POS] its military capabilities].
sity according to the topic of a speech, the year of the Such complex lexical structures may contain multiple
meeting, and the country the speaker represents. The layers of semantic connotations, which may lead to the
language of the data is English, and the speeches were annotator’s confusion due to the superposition of the la-
either originally presented in English, or were the offi- bels. The annotators may be drawn to find the attitudinal
cial UN translations. The information about the original meaning of every available token conveying attitudinal
language of the speech, as well as the speaker’s affilia- polarity which often leads to viewing an annotated frag-
tion and sex, the topic of the session, and its year are ment in isolation of the broader text meaning.
stored in the metadata of each text. The corpus consists Another approach to attitude identification is based
of 105592 tokens and 7296 types, and the average length on the subjective evaluation of a textual fragment under
of a sentence is 32 tokens. consideration based on the contextual boundaries of the
Our initial goal was for the guidelines to be of a qual- syntactic structure that bounds it, as well as the context
ity that would allow for the full disambiguation of the of the whole text (which in the case of our study is a
task, which means the annotators would select the same speech).
text span and assign completely identical labels to it in Let us take another look at the same sentence, where
terms of attitude type and subtype, sentiment polarity, the possible locations of attitude are defined by its sub-
and explicitness. We therefore apply the strictest ap- jective interpretation and context of the whole structure:
proach to annotations assessment at the time as it is
Even items subject to control would go to Iraq [once Let’s analyze an excerpt from one of the diplomatic
there is confidence that they would not be used to rebuild speeches provided in Example 4:
Iraq’s weapons of mass destruction or improve its military Example 4: Experts in our countries must immediately
capabilities: JUDGEMENT-INVOKED]. begin to analyse it and then draw the appropriate conclu-
Here, the speaker provides an implicit evaluation of the sions.
future possible behaviour of Iraq, expressing their hope Here, there is a definite presence of positive judgement
and inclination towards Iraq not rebuilding its weapons coded by the directive ’must’. experts must begin to anal-
of mass destruction. This scenario is presented as a sub- yse would be our first choice to annotate. The problem
jective evaluation of the possible future behaviour, which is, such a fragment, if non-explicit, does not convey any
means that the attitude in question should be perceived attitude on its own, therefore inclusion of the whole con-
as an implicit evaluation of the desired appropriate action text necessary for understanding of the chosen label is
of Iraq. advised: beginning of analysis is only good if it leads to
Even though this way of finding attitudes leads us drawing appropriate conclusions.
to a somewhat more vague definition of the borders of
the annotated expression, it also provides us with more 2.3.1. On annotating articles
contextual meaning as opposed to the cumulative polarity
of separate tokens. Another required decision is related to the annotation of
Let us exemplify this approach again with the Example determiners. As most of the annotated entities constitute
2: either an evaluative adjective or a collocation of an eval-
Example 2: [A number of these have been under discus- uative adjective and an evaluative noun, it is necessary
sion for some time in this Council: AFFECT-INVOKED], and to decide whether or not articles defining the evaluative
[if we were able to agree on this package, it would be an im- phrase should be included in the annotation or not.
portant step forward in that regard as well: JUDGEMENT- Our solution to this question is related to the overall
INVOKED]. viewpoint on the attitude framework and depends on t
In this example, the speaker implies their emotional If an attitude is expressed explicitly (inscribed tag)
state of unsatisfaction by the amount of time that an issue as in ’a capable management’, ’we are happy about the
is already under discussion while providing their implicit resolution’, ’the valuable lesson’, etc.) and the perceived
evaluation of the appropriate course of future actions. meaning of the adjective+noun collocation is enough to
Our preferred approach is to judge the span of the understand it in the full, annotation of any additional
annotated fragment by the context and the explicitness tokens should be omitted. Usually, it is enough to only
of the evaluation, which would lead to a clearer logic and mark one token or collocation that is used for expressing
accountability of the annotation process. an attitude. There is no need to include any additional
tokens in the annotated fragment, for example, if an at-
titude is found within a phrase [they have thought of a
2.3. On identifying the borders of wonderful solution], we would advise only marking the
inscribed and invoked attitudes evaluative adjective and omit to annotate the article that
Attitude examples observed in the corpus were proven precedes it and the noun that follows. We also advise
to make use of both explicit and implicit (inscribed and against article annotation when a noun that is in colloca-
invoked) ways of expressing an attitude. The diplomatic tion with the attitude-bearing adjective has no attitudinal
discourse is known for its indirectness [5] and a partic- meaning itself (the unfortunate circumstances).
ularly subtle expression of subjectivity. However, the However, if an attitude is a part of the superlative
inscribed and invoked attitudes are not the same regard- form (the best) or is expressed implicitly and the broader
ing the additional context needed for discerning the mes- context is necessary for understanding the expression,
sage’s meaning. any needed number of tokens within one syntactic clause
The inscribed attitudes only require the explicitly sub- could be considered (as in the veto today will not prevent
jective tokens to be annotated, as shown in Example 3: that).
Example 3:France [fully supports: AFFECT-INSCRIBED]
the search for a political solution. 2.4. Overlapping attitudes
This decision would lead to a sufficient increase in the
Another situation that may cause confusion and subse-
expected average length of the annotated fragment for
quent mistakes in annotation includes the cases, where
the invoked annotated attitudes in comparison to the
the two annotated fragments overlap. An overlap hap-
inscribed attitudes.
pens if one attitude occurs within the scope of the an-
Our main solution for annotating invoked attitudes is
notated fragment of another attitude. This may happen
to capture as much meaning as possible by annotating
as few tokens as needed.
in many ways, however, if there are only two attitudi- has led to a parallel annotation experiment as well as
nal elements, the overlap may look one of the following to the creation of a background for the newly-published
ways: corpus of evaluation in diplomatic speeches of the UNSC
1) The first option is for the second label to overlap (CoDipA 1.0).
with the first one as in (a(b)); We are specifically interested in investigating the atti-
2) The second option is for the first element to overlap tude development processes throughout the selected time
with the second one as in ((a)b); frame and in comparing the findings on an inter-conflict
3) The third option is the inclusion of an element (b) scale. Another expected development is the practical
in the middle of the first element (a) following a scheme application of the acquired data on a fine-tuned large lan-
(a(b)a) as in Example 5: guage model (such as Bert or GPT 4) to establish whether
Example 5: There is no doubt that we all want there is a potential for expanding the analysis to a bigger
[Affect-inclination] to resolve this problem [Judgement- scale.
tenacity]. The results of both quantitative and qualitative eval-
Here, the whole sentence is an invoked judgement with uation of the annotation outcomes are expected to be
an inclusion of an inscribed affect (in bold) corresponding published as well.
to the expression of the speaker’s desire.
To technically deliver this solution it is necessary to
allow for overlapping annotation in doccano [10] when 4. Conclusions
creating a project.
In this research paper, we have outlined some of the
challenging parts of annotating attitudes according to the
2.5. Interrupted attitudes Appraisal theory scheme [1] in the diplomatic speeches
The issue of interrupted annotation refers to a situation, and provided our viewpoint on their resolution, as well
where a span of attitudinal text includes a sequence of as technical comments on how to implement this type of
non-attitudinal tokens. Annotating unnecessary tokens annotation in practice.
may lead to a decrease in the quality and reliability of the The first problem that an annotator would encounter
future corpus. As per our observations, such an annota- if they would seek attitude in diplomatic speeches, would
tion scenario occurs solely with the explicitly formulated be attitude identification and deciding on the textual
attitudes, therefore deciding on this type of label is ad- borders of an attitudinal fragment. We have outlined
vised as a first step. these processes and provided examples from our corpus.
Here is an example of an interrupted attitude (Exam- In the next sections, we have offered an overview of the
ple 6), an excerpt from another diplomatic speech. An technically challenging parts of the attitude annotation
invoked judgment of the previous lack of actions of the process, such as overlapping and interrupted types of
Security Council is interrupted by a referral to one of attitudes, as well as their solution.
the resolutions, which does not add any additional atti-
tudinal meaning to the annotated fragment and should Limitations
therefore be excluded from it.
Example 6: Unfortunately, in the past hundred days, the This paper presents an up-to-date analysis of the ongo-
very [limited suspension of the sanctions: JUDGEMENT- ing research based on the annotation project, including
INVOKED-PART1] established by the Security Council res- the development of a discourse-specific annotation sce-
olution 943 (1994) [has also not been entirely fulfilled: nario. The findings from the presented annotation sce-
JUDGEMENT-INVOKED-PART2]. nario should not yet be considered as final, they may
Our solution to resolving it is purely technical and be updated in the final version of the scenario before its
includes the creation of an additional label [none] that is publication.
to be assigned to the tokens excluded from the annotated
fragment.
Ethics Statement
3. Related ongoing and future We honour the ethical code set out in the ACL Code of
Ethics and there are no special ethical issues involved
work during the creation of this research paper.
The acquired annotation scenario has helped us further
define our approach to the category of attitude as it is
understood by the Appraisal theory [1] and adapt it to
the needs of the diplomatic discourse analysis, which
5. Acknowledgements [10] H. Nakayama, T. Kubo, J. Kamura, Y. Taniguchi,
X. Liang, doccano: Text annotation tool
The research reported in this paper was supported by 207- for human, 2018. URL: https://github.com/
01/207PROV and the Czech Science Foundation (project doccano/doccano, software available from
no. 24-11132S, Disagreement in Corpus Annotation and https://github.com/doccano/doccano.
Variation in Human Understanding of Text), and partially
supported by SVV project number 260 698; a part of the
used data comes from the project no. LM2018101 by the
Czech Ministry of Education, Youth and Sports (Digital
Research Infrastructure for Language Technologies, Arts
and Humanities).
References
[1] J. R. Martin, P. White, The Language of Evaluation:
Appraisal in English, Springer, 2005. doi:10.1057/
9780230511910.
[2] M. Fuoli, A step-wise method for annotating ap-
praisal, Functions of Language 25 (2018) 229 – 258.
doi:10.1075/fol.15016.fuo.
[3] M. O’Donnell, Exploring identity through appraisal
analysis: A corpus annotation methodology, Lin-
guistics and the Human Sciences 9 (2013) 95–116.
doi:10.1558/lhs.v9i1.95.
[4] P. C. Cozby, Methods in behavioral research,
McGraw-Hill, New York, 2012.
[5] N. Stanko, Use of language in diplomacy, in: H. S.
J. Kurbalija (Ed.), Language and Diplomacy, Diplo-
Projects, Mediterranean Academy of Diplomatic
Studies, University of Malta, Msida MSD 06, Malta,
2001, pp. 39–48.
[6] M. Anisimova, An introductory overview of
evaluating facts and attitudes in diplomatic dis-
course, in: 2nd Workshop on Automata, For-
mal and Natural Languages – WAFNL 2021
Open Session Proceedings, Faculty of Mathe-
matics and Physics, Charles University, Prague,
2021, pp. 1–4. URL: https://itat.ics.upjs.sk/public/
WAFNL2021OpenSessionProc.pdf.
[7] M. Anisimova, S. Zikánová, Attitude in diplomatic
speeches: a pilot study, 2022, pp. 151–158. URL:
https://ceur-ws.org/Vol-3226/paper17.pdf.
[8] M. Anisimova, Š. Zikánová, Attitudes in diplo-
matic speeches: Introducing the CoDipA UNSC
1.0, in: H. Bunt, N. Ide, K. Lee, V. Petukhova,
J. Pustejovsky, L. Romary (Eds.), Proceedings of
the 20th Joint ACL - ISO Workshop on Interopera-
ble Semantic Annotation @ LREC-COLING 2024,
ELRA and ICCL, Torino, Italia, 2024, pp. 17–26. URL:
https://aclanthology.org/2024.isa-1.3.
[9] M. Schoenfeld, S. Eckhard, R. Patz, H. v. Meegden-
burg, A. Pires, The UN Security Council Debates,
2019. URL: https://doi.org/10.7910/DVN/KGVSYH.
doi:10.7910/DVN/KGVSYH.