Introduction

The availability of digital language resources enables an important step forward in linguistic research, both for its theoretical as well as applicational orientation. The originally collected data serving mostly for the study of the lexical studies and those of the study of syntax proper gave an impulse to enrich them by various more sophisticated annotation systems dealing with most different phenomena, going beyond the sentence boundary and incl. e.g. text coherence and phenomena related to inferencing, and elaborating more levels of granularity in the annotation. The annotated data serve for different tasks in the computational processing of natural languages -as training and testing data for the development of language models.

Human data annotation is a process based on interpretation of observed phenomena and thus may lead to different outcomes. This variation is caused by various factors. Some of them are connected with the shortcomings of the annotation scenario (e.g., not providing instructions for the solution of some cases) or with the leaks of the underlying theory (e.g., non-intuitive solutions or discerning too fine categories, very close to each other). Other cases of inter-annotator disagreement are connected with the learning process of annotators: especially the first annotated batches of data may be influenced by the annotators' unfamiliarity with the annotation scenario. That is the reason why these data are often re-annotated later. To prevent these kinds of inconsistent analysis of the data, annotators usually attend frequent trainings; simultaneously, their feedback at the beginning of the annotation may improve annotation scenario and point out some problematic points in the underlying theory. Before releasing data, annotators' mistakes are searched for and corrected, e.g. a simple overseeing of phenomena that should be marked; nevertheless, some of the mistakes can remain even in the final data. Last, but not least source of the disagreement in the annotation is language vagueness, polysemy and homonymy: in some cases, a language itself allows for several understandings of a sentence.

Computational linguistics offers several methodological approaches to this variability of the data annotation. One of the solutions is unification: a gold standard is set, e.g. by majority voting or by a third judge.

Another, more demanding way of data unification is a joint annotation, when annotators mark the data together, discussing each single case and marking the result of their discussion only.

In order to accept and capture the uncertainty annotators can face while marking language phenomena, some annotation scenarios with hierarchical classifications allow the use of more general levels of the classifications, not discerning the finest classification differences in dubious cases. Another way how to mark the annotators' certainty is a separate marking of their confidence as a specific feature (e.g., (a) a discourse relation is marked as a conjunction and (b) the annotator was absolutely sure about his solution). It is necessary to say that annotator's high certainty does not necessarily mean that his solution is the only possible one; in some cases, another annotator can be equally convinced about a different reading.

Unification is not the only way how to handle the data. Some researchers argue that unification may result in biased data missing important information about variability of language understanding [1]. Consequently, biased language models are developed based on this data. Therefore, annotators are allowed to mark multiple description of the same phenomenon in some approaches, (e.g., in the Penn Discourse Treebank 3.0 [2], a single discourse relation can be marked as an instantiation and cause at the same time, if the annotator understands it in this way). Other annotation projects publish their data with partial or complete multiple annotations carried out by different annotators; in such data, personal solutions of similar language phenomena can be observed systematically (cf. Czech RST Discourse Treebank, [3]).

Aim of the study

In our research, we deal with the annotation variation from a different perspective, from the linguistic and psycholinguistic point of view, with focusing on human language understanding. We use data with variations as a source of phenomena that are regularly understood in different ways and we search for possible common features of different readings. We pay special attention to the cues that are inherent to a language, rather than to the diversity among humans receiving the texts.

Questions of human language understanding have been addressed on a theoretical level, e.g. in psycholinguistics or lexical and syntactic semantics. In our study, we want to take use of our practical long term experience with large amounts of language data and possibly to offer some new insights into the variation of language interpretation or to contribute to theoretical discussions with practical findings.

Data: Text Coherence Annotation

Multiple reading may result at many language levels and perspectives, such as lexical semantics (cf. polysemy of the word bank as an institution and as a river bank), morphology (homonymous singular and plural form, like sheep or fish), syntax (having an old friend for dinner) etc. Our research is restricted to the area of text coherence in general. Specifically, our data cover multiple annotations of the following phenomena: discourse relations, coreference, and information structure (3.1-3.3).

Discourse relations

Discourse relations connect so called discourse arguments (clauses, sentences or larger text segments) and express certain semantic relation between the arguments. They are prototypically expressed by discourse connectives (conjunctions, subjunctions, discourse adverbs etc.), but they may be formally unexpressed, either. The former type of relations is called explicit discourse relations, the latter relations are implicit. <Arg1: She enjoyed working in the office> <Arg2: be-causeREASON she had pretty flowers there.>

In our data, we work with the data from the following discourse corpora:

(a) Prague Dependency Treebank 2.0 [12] and 3.0 [13]. The annotation scenario of the Prague Dependency Treebank was motivated by the approach of the Penn Discourse Treebank ( [14], following the Lexical Tree-Adjoining Grammar [15]) and is based on the Functional Generative Description [16] as applied in the family of Prague Dependency Treebanks. It discerns 23 semantic types of discourse relations, such as conjunction, disjunction, concession, generalization etc.; the discourse connectives are marked explicitly. The annotation is carried out on so called tectogrammatic (syntactico-semantic) dependency trees which allows the discourse annotation to be related to syntactico-semantic level of a language. The data in the corpus are in Czech.

(b) Enriched Discourse Annotation of Prague Discourse Treebank Subset 1.0 (PDiT-EDA 1.0, [17] The annotation scenario follows the approach of the Prague Dependency Treebank; the annotation is enriched with marking of implicit discourse relations.

(c) Data comparing underspecification of discourse connectives in five languages (English, French, Czech, Hungarian, Lithuanian) as published in [7]. The annotation scenario is based on the Crible's classification of discourse relations [7] discerning 15 discourse relations (e.g., opening, addition, topic-shift). Unlike the Praguian discourse approach, Crible's classification takes into account broader pragmatic aspects of discourse (so called domains), explicitly discerning ideational, rhetorical, sequential, and interpersonal domains where the discourse relations are used.

(d) Czech RST Discourse Treebank 1.0 [3]. The annotation scenario is based on the Rhetorical Text Structure Theory as applied in the Potsdam Commentary Corpus [18]. This theory assumes that text as a whole is built from a smaller segments which are all interconnected by discourse relations, without any part being left aside. It discerns 37 discourse relations (e.g., concession, concession as nucleus, textual preparation). A specific feature of RST is that it puts emphasis on different levels of com-

Coreference

Coreferential relations connect expressions with the same reference, such as The girl looked into her map, she looked like she was enjoying the adventure. Madelein had a great sense of orientation. The arguments of coreferential relations are prototypically noun phrases (nouns, pronouns) including dropped phrases (While [she] walking through the landscape, she admired the nature's beauty.).

A coreferential relation may also hold between a larger text segment, such as a whole thought or paragraph and a summarizing pronoun it / this etc. We use coreference data including disagreement in the annotation coming from the Prague Dependency Treebank 2.0 [12] and 3.0 [13] where coreference is a part of multi-level annotation including discourse and syntactic semantics (see above).

Information Structure

Information structure of a sentence expresses a communicative importance of single parts of a sentence in a given context. In general, it captures a topic (what the sentence is about) and a focus of a sentence (what new information is said about the topic), cf. (context: There is a cat under the tree.) It TOPIC is ready for a jump FOCUS.

Our data about information structure come from an experiment carried out on the data of the Prague Dependency Treebank 2.0 [12] where information structure is marked on dependency trees on the tectogrammatic (syntactico-semantic) level. 1 1 According to the Functional Generative Approach [16], a tectogrammatic tree consists of nodes which prototypically correspond to autosemantic words; the nodes are connected by edges expressing syntactico-semantic relations (e.g., Actor, Patient, Addressee). As for the information structure, each node is ascribed a value of contextual boundness (contextually bound, contextually non-bound, contrastively contextually bound). The nodes are ordered from the left to the right according to their so called communicative dynamism, i.e. measure to which they contribute to the development of information flow in the sentence. The values of topic and focus can be derivated from these two features (contextual boundness and communicative dynamism.)

Methodology

In the present study, we search for general language features of sentences (words, contexts) allowing for variable readings of text structure. For this purpose, we collect occurrences of inter-annotators' disagreement in the language corpora (see Table 1) and classify them manually, putting aside occurrences of disagreement resulting obviously from other types of reasons (annotator's mistake, technical solutions of the applied theory). We concentrate on the semantic and grammatical features of the examined sentences and expressions. 2The results are compared and supplemented by a metaanalysis of reports on annotations of single corpora; unfortunately, due to space limitations, the annotation reports often describe reasons of inter-annotators' disagreement very shortly.

Measuring inter-annotator disagreement on a text structure

On the most general level, measuring inter-annotator agreement of textual phenomena concerns with two criteria:

(a) How often all the annotators found a certain phenomenon (e.g., a discourse relation). E.g. one annotator may ignore a case which should be marked whereas the other one does not. This would be a case of a disagreement on the existence of the phenomenon.

(Dis)agreement on the existence is usually measured with the F1 measure (a harmonic average of precision and recall).

(b) Within the cases where all the annotators agree on the existence of a certain phenomenon, it is measured how often annotators agree on the classification of the found phenomenon. If one annotator assigns a discourse relation the semantic type conjunction, whereas the other one sees it as gradation, it is a case of a disagreement on the type of the phenomenon. (Dis)agreement on the type is prototypically evaluated as a simple percentage match or with the Cohen's kappa measure.

Both types of disagreement are relevant to our research: we are looking for linguistic features that can cause one annotator not to recognize a certain type of contiguity while another does. We are equally interested in the linguistic reasons why annotators ascribe different meanings to one coherence relation. 3

Analysis

In our data, which includes the annotation of discourse relations, coreference, and information structure, we have identified seven areas (factors) that repeatedly influence different readings of textual coherence by annotators.

Synsemantic signals of coherence relations: polysemy

Some words function primarily in the text as explicit markers of coherence relations (discourse connectives for discourse relations, some pronouns for anaphoric relations). However, these words are often polysemous (polyfunctional) as lexical units: they can also be used in Similarly, in coreferential relations, e.g. the word it can perform a pronominal function and be part of a coreferential chain (She played great. I really liked it.), but it can also function as a grammatical word without any reference (The weather is fine. It is not raining anymore.). The presence of such synsemantic expressions in the text does not signal the presence of a coherence relation clearly; thus, recipients may disagree about the existence of a relation depending on their readings of the function of the polysemous word, as in the discourse annotation example 1:

(1) Annotation 1: explicit discourse relation expressed by a discourse connective přece (because) <Arg1: Neptejte se mě, proč jsem přijel do Prahy.> <Arg2: Je to přece EXPLICATION normální sem přijet.>

3 General information on measuring inter-annotator agreement can be found in [19].

Many annotation projects adapt their measurement methods to more precisely suit the phenomena under investigation. E.g. in the case of discourse relations, the agreement on existence can be considered strictly as the case where both annotators agree on the exact scope of both discourse arguments and assign it to a certain discourse connective as an agreement on existence. For a looser approach, which respects that the exact localization of arguments can be difficult in some cases, the mere matching of a discourse connective can be considered an agreement on existence. In this case, it does not matter which words exactly the annotators mark as parts of single discourse arguments [9].

Synsemantic signals of coherence relations: underspecification

Other cases of disagreement are based on the semantic underspecification of words signaling coherence relations: in these cases, the annotators agree on the existence of a certain relation, but they disagree on the assessment of its meaning (disagreement on type). This disagreement is typical for discourse relations, signaled by discourse connectors with a vague meaning, cf. ( 2):

( Different understandings of underspecified discourse conjunctions are also evident in the dataset reported in [7], which contains the original English subtitles of TED talks and their equivalents in four languages. In the following document, the original English conjunction but (underspecified discourse connective with contrastive meaning) is translated using the Czech a (and, underspecified discourse connective with a simple conjunctive meaning).

(3) English original: Today I want to talk to you about the mathematics of love. Now, I think that we can all agree that mathematicians are famously excellent at finding love. But it's not just because of our dashing personalities, superior conversational skills and excellent pencil cases.

Czech translation:

Dnes vám chci povědět něco o matematice lásky. Myslím, že se shodneme na tom, že matematici jsou v oblasti lásky proslulí svými schopnostmi. A nestojí za tím jen okouzlující charakter, neobyčejný konverzační um či ostře nabroušené tužky.

(Dataset of the research reported in [7])

The interchangeability of these words in the given contexts raises certain theoretical questions: for example, what level of text coherence is necessary for the recipient? In the examples given, it seems sufficient to signal that the two arguments are connected by a discourse relation. Which meaning type is specifically involved seems to be irrelevant. Both examples, ( 2) and (3) lead at the same time to another question, namely the nature of the semantic types of discourse relations. In the annotations, we differentiate the individual types very precisely; but in fact, contrastivity, like causality, can be scalar, gradual, can be located on the same axis with conjunction, and different recipients can only perceive different degrees of contrastivity or causality. This property of discourse semantic types can be verified using psycholinguistic experiments.

Autosemantic words in coherence relations: genericity and abstractness

Based on the analysis of the data, we make the assumption that autosemantic words with a concrete, nonabstract meaning (cf. concrete to bake versus abstract to do) and expressions with a specific, not generic reference (the boy vs. the youth as such) are generally more accessible and representable for the recipients. In this context, we observe that words with an abstract meaning or with a generic reference can complicate the understanding of the text coherence structure: in sentences with these expressions, inter-annotator disagreement occurs more often.

Regarding coreferential relations, Nedoluzhko [10, p. 221] states that "The more nouns with abstract meaning and expressions with generic reference in the text, the smaller the agreement. " It is often difficult to estimate, for example, whether concepts of two abstract expressions fully overlap (and are therefore fully coreferential), or one is a part of the other, or they are independent, cf. ( 4 Also in the annotation of discourse relations, words with an abstract, non-specific meaning result in the interannotators' disagreement [5]. This is the case of sentences including verbs with an abstract, general meaning. As the authors say, "The disagreement occurs when it is not clear whether the potential discourse connective refers to the whole sentence as an independent abstract object (discourse argument), or just to its complement, typically a nominal phrase." [5, p. 2003]. Thus, in example (5), the disagreement between annotators shows that it is questionable whether the second part of the sentence (while chimneys. . . ) is related to the whole previous clause including the verbs with abstract meaning (it is possible to note a small, but distinctive difference between. . . ), or just to the nominal phrase (a small, but distinctive difference between. . . ). 4( In fact, this is a disagreement on which level the given phenomenon should be captured (in this case, coreference or discourse). It is rather an academic question how to annotate these cases consistently. As for the recipients themselves, the difference in the annotation does not mean a difference in the understanding of the text, as the language levels and perspectives are inter-related and the annotators can ascribe single phenomena to different levels without understanding the text coherence in a different way.

Attribution: verbs of thinking and saying

Attribution is the relation between the (named) author of a section of text and his speech. A typical component in the attribution construction is the author's name, the verb of thinking or speaking or another form expressing speech (colon, phrases such as according to) and the direct / indirect speech itself (dictum). A language has means how to distinguish the author's speech from the reported speech. Nevertheless, with attributive constructions it is often difficult to distinguish how far discourse relations extend and what is the scope of their arguments, especially when it comes to verbs of thinking and saying. In these cases, annotators often disagree in their interpretations, cf. examples ( 6) and ( 7).

(

Word order

So far, we have observed cases of disagreement between annotators, which result from the lexical properties of expressions ensuring coherence (underspecification vs. specificity, abstractness vs. concreteness) and from the syntactic structure (governing verb of saying/thinking vs. dictum itself). Word order is another area that plays an important role in ensuring the coherence of the text and can also become subject to different interpretations. In Czech, similarly as in other Slavic languages, the word order is relatively free, with few grammatical restrictions. It is used to express information structure of a sentence: the information belonging to the topic is prototypically placed in the sentence to the left, the focus is usually located to the right. However, it is also possible to use a marked word order, when the topic and focus occupy various places in the sentence and are distinguished by intonation, the use of focalizers, or deduced from the context. This freedom in the formal expression of information structure results in some cases in inter-annotator disagreement. Often, annotators interpret differently information structure of the left part of a sentence: some tend to consider it less important, disregarding the used expressions, because it is prototypically a topic position; others are more driven by context and other indicators of possible focus.

This variability applies especially to adverbials located before the verb in the surface word order, focalized phrases and predicate verbs in the left part of the sentence [11]. The example (8) presents an ambiguous interpretation of the conditional phrase at the beginning of the sentence; one of the annotators considers it to be a part of the very message of the sentence, the other as a mere unimportant circumstance. Thus, both perceive the given sentence as a response to a different (unspoken) context, as shown by the contextual questions at the end of each interpretation. (The expressions in topic are underlined; the focus is marked with bold characters.) (8) (How will your situation be if you take full advantage of your present-day capacities?) ([11]; control multiple annotation of the PDT 2.0, [12])

In example (9), there is a collision between two indicators of importance (belonging to the topic / focus): the observed phrase is located at the beginning of the sentence, a place typical for the topic; but at the same time it is emphasized by the focalizer. Annotators perceive its role in the information structure of the sentence differently.

( (What does the firm do especially in Olomouc?) ([11]; control multiple annotation of the PDT 2.0 [12])

In example (10), a striking feature of verbs can be seen: expressions dependent on the verbs often tend to be communicatively more important than the verbs themselves. This can make the role of predicate verbs in the information structure unclear: annotators do not agree whether to classify them as focus or as topic. We have already observed the unclear importance of verbs with respect to dependent parts in examples (5, unclear role of a verb with general meaning in a discourse structure) and (6-7, unclear role of a verb of thinking/saying in a discourse structure, compared to the clear role of dictum).

( As the previous subsection showed, the variety of understanding of coherence relations often comes from certain linguistic forms (specific word order pattern, etc.). However, the language itself often does not provide a clue: we cannot tell which phrase or syntactic construction was vague enough to allow for multiple readings. The diversity here comes from the different experience of the recipients, from their expectations and knowledge of the world. This type of inter-annotator disagreement is difficult for linguistics to grasp. Nevertheless, since we can document it well in our data, we take the liberty of presenting a few of these phenomena here, which can serve as inspiration for e.g. psycholinguistic research.

At the local level, subjectivity can be seen in the perception of importance in the information structure (cf. [21]), i.e. what people see as a topic / focus of a sentence. Furthermore, this variation is found in discourse relations in Rhetorical Structure Theory, which differentiates between a more substantial and a less substantial arguments of a discourse relations (nucleus and satellite, respectively; cf. [8]). See the following example (11) where adjacent sentences have the same syntactic structure connected by the phrase not only -but also. One of the annotators considers both parts of these sentences to have the same level of importance and marks a multinuclear relation of contrast between them. The other one understands the second parts (starting with but also) as emphasized, more important, marking thus the relation as antithesis with the nucleus in the second part. (11) At the global level, in the annotations according to Rhetorical Structure Theory, the perceptual importance of individual parts of news reports differs, too. Typically, while one annotator understands the introductory part as a central message to which details are added in the following text, the other perceives the same part as a preparation to which the own message is associated afterwards. ( [8]).

Text dimensions

Inter-annotator agreement can also be affected by text dimensions. As coreference research shows, the larger the network of possible antecedents for a given word in a text, the greater the disagreement between annotators ([10, p. 221]; cf. the opportunities for disagreement in example 4). The author further states that divergent interpretations of coreference can also be chained: if annotators differ in the interpretation of expressions at the beginnings of the coreference chain, their different interpretations can be reflected in other expressions with a similar meaning in the text.

It is a question of how the size of the text affects the variability of understanding in other coherence relations, such as discourse relations and information structure. We have not yet conducted research in this direction. For discourse relations, there can theoretically be more potential arguments in a large text that are connected by a discourse connective. If the text is longer, it will probably also be more layered in terms of author's and reported speech, metacommunication, insertions, etc., which again offers more possibilities for different understandings of discourse and other relation. On the other hand, a longer text can more accurately describe the context in which the discourse relations are interpreted, and thus contribute to the clarity of understanding. In this regard, another question arises: whether there is a difference in the variability in the understanding of coherence relations at the beginning of the text (where the text is still short, there are few potential members of different relations available, but also little context) and in its later parts.

Conclusion

In this study, we observed what common features the occurrences of inter-annotator disagreement have in coherence relations, specifically in discourse relations, coreference and information structure. We were mainly con-cerned with the features given by the language itself; we only marginally stopped at cases of disagreement that result from the difference of speakers. We have also formulated some questions that can be the subject of further research.

Coherence relations can be divided into formally expressed (e.g. in the discourse structure relations expressed by an explicit discourse connective or an information structure expressed by word order) and unexpressed relations that are understood from the context (e.g. coreference relation between the words text and chapter in a specific text).

In formally unexpressed relations, disagreement occurs naturally: it depends on the recipients what they infer from the context. Formally expressed relations can be also interpreted differently. There may be disagreement on the very existence of a coherence relation; this disagreement is usually based on the polysemy (polyfunctionality) of the linguistic form (expression), which in some contexts functions as a signal of coherence, but not in others. In addition, coherence signals can also lead to a different perception of the semantic type of a discourse relation (in cases where speakers agree on its existence): this is caused by the semantic underspecification of language forms that express coherence (discourse connectives). The general question arises whether, as recipients, we need to understand textual coherence in detail in all contexts, i.e. distinguish not only the simple existence of coherence relations, but also their semantic coloring. What level actually represents a functional and sufficient understanding of the text?

Lexical specificity plays an important role in the understanding of autosemantic words, too; these expressions do not function primarily as signals of coherence. Coreference research shows that for abstract and generic nominal phrases in a text, recipients determine with difficulty whether the words have the same content; in contrast, for words with a concrete, specific meaning, coreference is easier to determine. The same applies to the semantic concreteness of verbs: for verbs with more vague, general meanings, it is difficult for annotators to determine whether or not they are part of discourse arguments. Their meaning seems to be too insignificant, whereas the content of their dependent words is more important.

This observation also applies to the verbs of thinking and saying in the relation of attribution, where the content of reported speech seems to be communicatively more essential than the act of communication itself. In the case of attribution, there is another reason for the diverse interpretation of the text: it represents one of the forms of text arrangement (alongside parentheses, meta-comments on the communication, etc.), i.e. a complication in the simple basic line of the narrative. It thus provides the possibility for different recipients to interpret the overall structure of the text differently.

In addition to individual words, such as various coherence operators or autosemantic expressions, word order can also cause a disagreement in text understanding. Specifically, in Czech and other Slavic languages, word order affects the understanding of the information structure. If expressions with higher communicative dynamism (informativeness) appear in the left, topical part of the sentence, which has a prototypically low communicative dynamism, typical contradictions in their evaluation occur.

In many types of annotation, it turns out that annotators perceive the importance of individual parts of the text and their (hierarchical) connections differently. These disagreements are often not so much caused by the special properties of the text as by differences between the annotators (specifically, it may be knowledge of the language, knowledge of the world, expectations, experience with different text genres, etc.). This area seems particularly suitable for future psycholinguistic research focusing on specific domains of coherence. Here, for example, it is possible to examine the influence of respondents' literacy on the understanding of coreference in abstract words or the process how children learn the text arrangement.

The last factor we dealt with is text dimensions. Its effect on different readings was described in coreference (the longer the text, the greater the disagreement in interpretation). For other coherence relations, this factor is still unexplored. We hypothesized that for discourse relations and information structure, text dimensions could influence the degree of disagreement in both directions; the degree of disagreement may also vary by place in the text and amount of preceding context (early vs. later in the text). These ideas suggest possible directions for further research on different text comprehension coherence.