It is typical for natural languages that their texts can be understood diferently by individual recipients. A number of scientific disciplines, from cognitive psychology to linguistics, are devoted to this phenomenon. In this study, we focus mainly on linguistic factors, which may lead to diferent interpretations of coherence relations in the text (simply speaking, what is related to what and how). This work presents a pilot typological survey of disagreements in Czech corpus annotations of coherence relations (discourse relations, coreference, information structure) and their common features. Polysemy (polyfunctionality) and semantic underspecification of coherent expressions (e.g. discourse connectives), generic / abstract meaning of autosemantic words, presence of attribution constructions, word order as a potential marker of information structure and text size appear to be essential factors for disagreement in interpretation. In addition, subjective reception of the relative importance of diferent text parts plays an important role, too. Based on the observation of the material, we raise questions and propose possible steps for the ongoing research of variability in the perception of text coherence.
unfamiliarity with the annotation scenario. That is the reason why these data are often re-annotated later. To The availability of digital language resources enables an prevent these kinds of inconsistent analysis of the data, important step forward in linguistic research, both for annotators usually attend frequent trainings; simultaneits theoretical as well as applicational orientation. The ously, their feedback at the beginning of the annotation originally collected data serving mostly for the study of may improve annotation scenario and point out some the lexical studies and those of the study of syntax proper problematic points in the underlying theory. Before regave an impulse to enrich them by various more sophis- leasing data, annotators’ mistakes are searched for and ticated annotation systems dealing with most diferent corrected, e.g. a simple overseeing of phenomena that phenomena, going beyond the sentence boundary and should be marked; nevertheless, some of the mistakes can incl. e.g. text coherence and phenomena related to infer- remain even in the final data. Last, but not least source of encing, and elaborating more levels of granularity in the the disagreement in the annotation is language vagueness, annotation. The annotated data serve for diferent tasks polysemy and homonymy: in some cases, a language itself in the computational processing of natural languages – as allows for several understandings of a sentence. training and testing data for the development of language Computational linguistics ofers several methodologimodels. cal approaches to this variability of the data annotation.
Human data annotation is a process based on interpre- One of the solutions is unification: a gold standard is set, tation of observed phenomena and thus may lead to difer- e.g. by majority voting or by a third judge. ent outcomes. This variation is caused by various factors. Another, more demanding way of data unification is a Some of them are connected with the shortcomings of the joint annotation, when annotators mark the data together, annotation scenario (e.g., not providing instructions for discussing each single case and marking the result of their the solution of some cases) or with the leaks of the under- discussion only. lying theory (e.g., non-intuitive solutions or discerning In order to accept and capture the uncertainty annotatoo fine categories, very close to each other). Other cases tors can face while marking language phenomena, some of inter-annotator disagreement are connected with the annotation scenarios with hierarchical classifications allearning process of annotators: especially the first anno- low the use of more general levels of the classifications, tated batches of data may be influenced by the annotators’ not discerning the finest classification diferences in dubious cases. Another way how to mark the annotators’ certainty is a separate marking of their confidence as a specific feature (e.g., (a) a discourse relation is marked as a conjunction and (b) the annotator was absolutely sure Conference ITAT (Information Technologies—Applications and Theory), 2024: Drienica, Čergovské vrchy, Slovakia $ zikanova@ufal.mf.cuni.cz ( Zikánová) https://ufal.mf.cuni.cz/sarka-zikanova ( Zikánová) 0000-0002-7805-9649 ( Zikánová)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License about his solution). It is necessary to say that annotator’s CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) high certainty does not necessarily mean that his solution In our research, we deal with the annotation variation from a diferent perspective, from the linguistic and psycholinguistic point of view, with focusing on human language understanding. We use data with variations as a source of phenomena that are regularly understood in diferent ways and we search for possible common features of diferent readings. We pay special attention to the cues that are inherent to a language, rather than to the diversity among humans receiving the texts.
Questions of human language understanding have been addressed on a theoretical level, e.g. in psycholinguistics or lexical and syntactic semantics. In our study, we want to take use of our practical long term experience with large amounts of language data and possibly to ofer some new insights into the variation of language interpretation or to contribute to theoretical discussions with practical findings.
is the only possible one; in some cases, another annotator 3.1. Discourse relations can be equally convinced about a diferent reading.
Unification is not the only way how to handle the data. Discourse relations connect so called discourse
arguSome researchers argue that unification may result in ments (clauses, sentences or larger text segments) and
biased data missing important information about variabil- express certain semantic relation between the arguments.
ity of language understanding [
The data in the corpus are in Czech.
perspectives, such as lexical semantics (cf. polysemy of
the word bank as an institution and as a river bank),
morphology (homonymous singular and plural form, like (d) Czech RST Discourse Treebank 1.0 [
The annotation scenario is based on the Crible’s
classification of discourse relations [
Amount of multiple annotations 44 documents, 2084 sentences; 2 annotators 12 documents, 233 sentences; 2 annotators 3 documents, 234 sentences, 4720 words in the original English; 1-2 annotators for each language 5 documents, 63 sentences, 2 annotators 2 annotators, the number of of texts and sentences is not presented
annotators 879 sentences annotated by 6 annotators, 9825 sentences annotated by 3 annotators
Reference
municative importance of discourse arguments, mark- 3.3. Information Structure
ing more important and less important parts (nucleus
and satellite, respectively) in every discourse relation. Information structure of a sentence expresses a
commuRelations with balanced importance of both parts are nicative importance of single parts of a sentence in a
described as multinuclear. given context. In general, it captures a topic (what the
sentence is about) and a focus of a sentence (what new
information is said about the topic), cf. (context: There is
3.2. Coreference a cat under the tree.) It TOPIC is ready for a jump FOCUS.
Coreferential relations connect expressions with the Our data about information structure come from an
same reference, such as The girl looked into her map, she experiment carried out on the data of the Prague
Delooked like she was enjoying the adventure. Madelein had pendency Treebank 2.0 [
A coreferential relation may also hold between a larger text segment, such as a whole thought or paragraph and a summarizing pronoun it / this etc.
We use coreference data including disagreement in the
annotation coming from the Prague Dependency
Treebank 2.0 [
in the linguistic reasons why annotators ascribe diferent meanings to one coherence relation.3 In the present study, we search for general language features of sentences (words, contexts) allowing for variable readings of text structure. For this purpose, we collect 5. Analysis occurrences of inter-annotators’ disagreement in the language corpora (see Table 1) and classify them manually, In our data, which includes the annotation of discourse reputting aside occurrences of disagreement resulting ob- lations, coreference, and information structure, we have viously from other types of reasons (annotator’s mistake, identified seven areas (factors) that repeatedly influence technical solutions of the applied theory). We concen- diferent readings of textual coherence by annotators. trate on the semantic and grammatical features of the examined sentences and expressions.2 5.1. Synsemantic signals of coherence
The results are compared and supplemented by a meta- relations: polysemy analysis of reports on annotations of single corpora; unfortunately, due to space limitations, the annotation re- Some words function primarily in the text as explicit ports often describe reasons of inter-annotators’ disagree- markers of coherence relations (discourse connectives ment very shortly. for discourse relations, some pronouns for anaphoric relations). However, these words are often polysemous (polyfunctional) as lexical units: they can also be used in 4.1. Measuring inter-annotator other, coherence-unrelated roles in the text. For example, disagreement on a text structure conjunctions can have a connecting function in discourse On the most general level, measuring inter-annotator relations, but they can also become particles and funcagreement of textual phenomena concerns with two cri- tion as communication expressions without connecting teria: function (cf. Czech Já peníze nemám, aleCONJUNCTION můj bratr je má. I have no money, butCONJUNCTION my brother (a) How often all the annotators found a certain phe- has. vs AlePARTICLE prosím vás! Co to říkáte? ButPARTICLE nomenon (e.g., a discourse relation). E.g. one annota- please! What are you saying?). tor may ignore a case which should be marked whereas Similarly, in coreferential relations, e.g. the word it the other one does not. This would be a case of a can perform a pronominal function and be part of a corefdisagreement on the existence of the phenomenon. erential chain (She played great. I really liked it.), but (Dis)agreement on the existence is usually measured with it can also function as a grammatical word without any the F1 measure (a harmonic average of precision and re- reference (The weather is fine. It is not raining anymore.). call). The presence of such synsemantic expressions in the text does not signal the presence of a coherence relation clearly; thus, recipients may disagree about the existence of a relation depending on their readings of the function of the polysemous word, as in the discourse annotation example 1: (b) Within the cases where all the annotators agree on the existence of a certain phenomenon, it is measured how often annotators agree on the classification of the found phenomenon. If one annotator assigns a discourse relation the semantic type conjunction, whereas the other one sees it as gradation, it is a case of a disagreement on the type of the phenomenon. (Dis)agreement on the type is prototypically evaluated as a simple percentage match or with the Cohen’s kappa measure.
by a discourse connective přece (because) <Arg1: Neptejte se mě, proč jsem přijel do Prahy.> <Arg2: Je to přece EXPLICATION normální sem přijet.> (1)
Both types of disagreement are relevant to our research: we are looking for linguistic features that can cause one annotator not to recognize a certain type of contiguity while another does. We are equally interested
we interpret the real reasons of inter-annotators’ disagreement correctly: what we see as a variation based on a language feature, could have be seen by an annotator just as his clear oversight. We do not have annotators’ explanations for their solutions. These questions are being solved by the present-day research by Anna Nedoluzhko; for the time being, we find this method appropriate for the present analysis as a pilot study.
be found in [
Many annotation projects adapt their measurement methods to
more precisely suit the phenomena under investigation. E.g. in
the case of discourse relations, the agreement on existence can be
considered strictly as the case where both annotators agree on the
exact scope of both discourse arguments and assign it to a certain
discourse connective as an agreement on existence. For a looser
approach, which respects that the exact localization of arguments
can be dificult in some cases, the mere matching of a discourse
connective can be considered an agreement on existence. In this
case, it does not matter which words exactly the annotators mark
as parts of single discourse arguments [
Don’t ask me why I came. Because EXPLICATION it’s normal to come here.
word přece (after all) expresses the stance of the speaker Neptejte se mě, proč jsem přijel do Prahy. Je to přece normální sem přijet.
Don’t ask me why I came. After all , it’s normal to come here.
(according to [6, p. 63]; multiple annotation of the PDiT-EDA 1.0 [
relations: underspecification Other cases of disagreement are based on the semantic underspecification of words signaling coherence relations: in these cases, the annotators agree on the existence of a certain relation, but they disagree on the assessment of its meaning (disagreement on type). This disagreement is typical for discourse relations, signaled by discourse connectors with a vague meaning, cf. (2): za tím jen okouzlující charakter, neobyčejný konverzační um či ostře nabroušené tužky.
(Dataset of the research reported in [
Both examples, (2) and (3) lead at the same time to another question, namely the nature of the semantic types of discourse relations. In the annotations, we diferentiate the individual types very precisely; but in fact, contrastivity, like causality, can be scalar, gradual, can be located on the same axis with conjunction, and diferent recipients can only perceive diferent degrees of contrastivity or causality. This property of discourse semantic types can be verified using psycholinguistic experiments. (2) 5.3. Autosemantic words in coherence relations: genericity and abstractness <Arg1: Za nabídku by se nemusel stydět ani Don Carleone – nebylo možné jí odolat.> <Arg2: A tak CONJUNCTION / RESULT do roka a do dne dostalo práci 440 shanonských občanů a do pěti let jich bylo už desetkrát tolik.>
Based on the analysis of the data, we make the
assumption that autosemantic words with a concrete,
nonabstract meaning (cf. concrete to bake versus abstract to
<Arg1: Not even Don Carleone would have to be do) and expressions with a specific, not generic reference
ashamed of that ofer – it was impossible to resist. > (the boy vs. the youth as such) are generally more
accessible and representable for the recipients. In this context,
<SAharngn2:onAgnodt asojoCbONwJUitNhCiTnIOaN /yReEaSUrLTan4d40a pdeaoyp,leanodf we observe that words with an abstract meaning or with
within five years, they were already ten times as a generic reference can complicate the understanding
many.> of the text coherence structure: in sentences with these
expressions, inter-annotator disagreement occurs more
([
Diferent understandings of underspecified discourse con- Regarding coreferential relations, Nedoluzhko [10, p.
junctions are also evident in the dataset reported in [
Today I want to talk to you about the mathematics of love. Now, I think that we can all agree that mathematicians are famously excellent at finding love. But it’s not just because of our dashing personalities, superior conversational skills and excellent pencil cases.
Dnes vám chci povědět něco o matematice lásky. Myslím, že se shodneme na tom, že matematici jsou v oblasti lásky proslulí svými schopnostmi. A nestojí (context: interview with child psychiatrists who published the Czech book Children, Family and Stress) - Materiálům, které dnes máte k dispozici, předcházel dlouholetý výzkum. - Zdeněk Dytrych: Od roku 1969, kdy jsme založili v bývalém Výzkumném ústavu psychiatrickém Oddělení pro výzkum rodiny, se hlavně zabýváme touto problematikou.
Měli jsme samozřejmě řadu spolupracovníků a za pětadvacet let jsme v týmu udělali téměř nekonečnou řadu prací.
Tak například rozsáhlý výzkum rozvodovosti. - The materials you have at your disposal today were preceded by a long-term research. - Zdeněk Dytrych: Since 1969, when we founded the Department for Family Research in the former Research Institute of Psychiatry, we have mainly been dealing with this issue.
Of course, we had a number of collaborators, and in twenty-five years we have done an almost endless amount of work as a team. [lit.: endless amount of works (plural) which can mean publications as well, ŠZ] For example, extensive research on the divorce rate.
([10, p. 223–226]; multiple annotation of the PDT 3.0[
Also in the annotation of discourse relations, words of a section of text and his speech. A typical component
with an abstract, non-specific meaning result in the inter- in the attribution construction is the author’s name, the
annotators’ disagreement [
a colon is understood as an explicit discourse connective ([
relates the second sentence to the whole previous
sentence including the verb of thinking phrase
vím, že (I know that).
“<Arg1: Vím, že se nás Rusů bojíte, že nás nemáte
rádi, že námi trochu pohrdáte.> <Arg2: Ale
OPPOSITION Rusko není jenom Žirinovskij, Rusko není
jenom vraždění v Čečensku.>”
“<Arg1: I know that you are afraid of us Russians,
that you dislike us, that you despise us a little.>
<Arg2: But OPPOSITION Russia is not only
Zhirinovsky, Russia is not only murdering in Chechnya.>”
Annotation 2: the discourse connective ale (but)
(7)
relates the second sentence to the content of the an important role in ensuring the coherence of the text
thought only, without the governing verb of think- and can also become subject to diferent interpretations.
ing. In Czech, similarly as in other Slavic languages, the
“Vím, že <Arg1: se nás Rusů bojíte, že nás nemáte word order is relatively free, with few grammatical
rerádi, že námi trochu pohrdáte.> <Arg2: Ale strictions. It is used to express information structure of a
OPPOSITION Rusko není jenom Žirinovskij, Rusko není sentence: the information belonging to the topic is
projenom vraždění v Čečensku.>” totypically placed in the sentence to the left, the focus is
“I know that <Arg1: you are afraid of us Russians, usually located to the right. However, it is also possible
that you dislike us, that you despise us a little.> to use a marked word order, when the topic and focus
oc<Arg2: But OPPOSITION Russia is not only Zhiri- cupy various places in the sentence and are distinguished
novsky, Russia is not only murdering in Chechnya.>” by intonation, the use of focalizers, or deduced from the
([9, p. 777]; multiple annotation of the PDT 2.0 [
[Při využití všech výukových prostor od rána až do večera] 0-subject jsme schopni ročně přijmout ke studiu okolo 2500 studentů.
Lit.: [When using all classrooms from
morning till evening] we_are able a_year to_accept
to_studies about 2500 students.
[When using all our classrooms during the whole
day], we are able to accept about 2500 new students
a year.
(How is your present-day situation?)
Annotation 2:
[Při využití všech výukových prostor od rána až do
večera] jsme schopni ročně přijmout ke studiu
okolo 2500 studentů.
(How will your situation be if you take full
advantage of your present-day capacities?)
([
The Oscar prize. . . for the firm Ilja Běhal & Co. which deals with smith craft and conservatory works mainly in central Moravia.) Nejvíc [kritizují a rozčilují se] neschopní. Lit.: Most [criticize and get_angry] incompetent.
Incompetent employees criticize and get angry most of all. (What happens?)
Nejvíc [kritizují a rozčilují se] neschopní.
(Who criticizes and gets angry most of all?)
([
spired by the information structure. In many kinds of
coherence annotations, we see that annotators difer in
Annotation 1: what they consider to be important, central, at a given
[Zejména FOCALIZER v Olomouci] firma svými place in the text.
výrobky přispívá ke zvýraznění koloritu his- As the previous subsection showed, the variety of
untorického jádra města. derstanding of coherence relations often comes from
cerLit.: [Especially FOCALIZER in Olomouc] firm tain linguistic forms (specific word order pattern, etc.).
with_its products helps accentuation However, the language itself often does not provide a
of_colouring of_historical centre of_city. clue: we cannot tell which phrase or syntactic
construc[Especially in Olomouc], the firm helps to accentuate tion was vague enough to allow for multiple readings.
the colouring of the historical centre of the city with The diversity here comes from the diferent experience
its products. of the recipients, from their expectations and knowledge
(What does the firm do? What can we say about of the world. This type of inter-annotator disagreement
the firm?) is dificult for linguistics to grasp. Nevertheless, since we
Annotation 2: can document it well in our data, we take the liberty of
[Zejména v Olomouci] firma svými výrobky přis- presenting a few of these phenomena here, which can
pívá ke zvýraznění koloritu historického jádra serve as inspiration for e.g. psycholinguistic research.
města. At the local level, subjectivity can be seen in the
per(What does the firm do especially in Olomouc?) ception of importance in the information structure (cf.
([
Their frequency is getting rather higher than lower.
Why?)
<Arg1: Jan Kotík nemaluje jen očima a rukou,>
CONTRAST / ANTITHESIS <Arg2: ale také mozkem.>
<Arg1: Jeho obrazy tedy vyžadují nejen citlivost
a vnímavost,> CONTRAST / ANTITHESIS <Arg2: ale také
přemýšlení.>
<Arg1: Jan Kotík paints not only with his eyes and
hands,> CONTRAST / ANTITHESIS <Arg2: but also with cerned with the features given by the language itself; we
his brain.> only marginally stopped at cases of disagreement that
<Arg1: Therefore his paintings require not only sen- result from the diference of speakers. We have also
forsitivity and receptivity,> CONTRAST / ANTITHESIS <Arg2: mulated some questions that can be the subject of further
but also thinking.> research.
(Czech RST Discourse Treebank 1.0 [
It is a question of how the size of the text afects the detail in all contexts, i.e. distinguish not only the simple variability of understanding in other coherence relations, existence of coherence relations, but also their semantic such as discourse relations and information structure. coloring. What level actually represents a functional and We have not yet conducted research in this direction. suficient understanding of the text? For discourse relations, there can theoretically be more Lexical specificity plays an important role in the underpotential arguments in a large text that are connected standing of autosemantic words, too; these expressions by a discourse connective. If the text is longer, it will do not function primarily as signals of coherence. Corefprobably also be more layered in terms of author’s and erence research shows that for abstract and generic nomreported speech, metacommunication, insertions, etc., inal phrases in a text, recipients determine with dificulty which again ofers more possibilities for diferent under- whether the words have the same content; in contrast, standings of discourse and other relation. On the other for words with a concrete, specific meaning, coreference hand, a longer text can more accurately describe the con- is easier to determine. The same applies to the semantic text in which the discourse relations are interpreted, and concreteness of verbs: for verbs with more vague, genthus contribute to the clarity of understanding. In this eral meanings, it is dificult for annotators to determine regard, another question arises: whether there is a difer- whether or not they are part of discourse arguments. ence in the variability in the understanding of coherence Their meaning seems to be too insignificant, whereas the relations at the beginning of the text (where the text is content of their dependent words is more important. still short, there are few potential members of diferent This observation also applies to the verbs of thinking relations available, but also little context) and in its later and saying in the relation of attribution, where the conparts. tent of reported speech seems to be communicatively more essential than the act of communication itself. In the case of attribution, there is another reason for the 6. Conclusion diverse interpretation of the text: it represents one of the forms of text arrangement (alongside parentheses, In this study, we observed what common features the oc- meta-comments on the communication, etc.), i.e. a comcurrences of inter-annotator disagreement have in coher- plication in the simple basic line of the narrative. It thus ence relations, specifically in discourse relations, coref- provides the possibility for diferent recipients to intererence and information structure. We were mainly con- pret the overall structure of the text diferently.
In addition to individual words, such as various coherence operators or autosemantic expressions, word order can also cause a disagreement in text understanding. Specifically, in Czech and other Slavic languages, word order afects the understanding of the information structure. If expressions with higher communicative dynamism (informativeness) appear in the left, topical part of the sentence, which has a prototypically low communicative dynamism, typical contradictions in their evaluation occur.
In many types of annotation, it turns out that annotators perceive the importance of individual parts of the text and their (hierarchical) connections diferently.
These disagreements are often not so much caused by the special properties of the text as by diferences between the annotators (specifically, it may be knowledge of the language, knowledge of the world, expectations, experience with diferent text genres, etc.). This area seems particularly suitable for future psycholinguistic research focusing on specific domains of coherence. Here, for example, it is possible to examine the influence of respondents’ literacy on the understanding of coreference in abstract words or the process how children learn the text arrangement.
The last factor we dealt with is text dimensions. Its efect on diferent readings was described in coreference (the longer the text, the greater the disagreement in interpretation). For other coherence relations, this factor is still unexplored. We hypothesized that for discourse relations and information structure, text dimensions could influence the degree of disagreement in both directions; the degree of disagreement may also vary by place in the text and amount of preceding context (early vs. later in the text). These ideas suggest possible directions for further research on diferent text comprehension coherence.
the Czech Science Foundation (project no. 24-11132S, Disagreement in Corpus Annotation and Variation in Human Understanding of Text); a part of the used data comes from the project no. LM2018101 by the Czech Ministry of Education, Youth and Sports (Digital Research Infrastructure for Language Technologies, Arts and Humanities).
The author would like to express her gratitude to Prof. E. Hajičová for careful proofreading of the manuscript, dr. J. Mírovský for help with the technical processing of the text and F. Zikánová for the language examples. Thank you all for the pleasant cooperation.