This paper presents a novel resource designed to study text coherence in the Italian language. The dataset aims to address existing deficiencies in coherence assessment by focusing on human perception of coherence. Recently, it has been integrated into the DiSCoTex benchmark, part of EVALITA 2023 [1], the 8th evaluation campaign for NLP and speech tools in Italian. Our resource aims to provide a comprehensive understanding of coherence, highlighting the influence of both genre and text perturbations on perceived coherence.
metric for assessing the quality of texts generated by
Natural Language Generation (NLG) systems [
Modelling coherence in natural language is essential [15], employed discourse parsers to generate discourse for a wide range of downstream applications. One such relations over sentences. With the advent of neural modapplication is automatic essay scoring in language learn- els, researchers have also explored their application in ing settings, where coherence assessment can provide coherence assessment, see e.g. Lin et al. [15] and Nguyen valuable writing feedback by identifying poorly orga- and Joty [16]. nized paragraphs and abrupt topic transitions [4, 5]. In The importance of building challenging datasets for clinical contexts, coherence modeling is relevant for au- coherence evaluation cannot be overstated. With this retomatic language assessment, as speech irregularities spect, independently from the underlying theories, modindicative of a lack of coherence can serve as markers for els of discourse coherence are typically tested on tasks mental disorders like schizophrenia [6, 7]. Furthermore, such as reordering, which aim to discern an original text coherence has been adopted as an intrinsic evaluation from a corrupted one artificially created by shufling the order of its sentences, or tasks that require systems to detect whether a document contains an intruder sentence from another document [9] or to classify whether a target sentence is contiguous or not with a given passage [17]. However, these approaches have come under criticism because they neglect key aspects of coherence, as noted by Lai and Tetreault [4] and Beyer et al. [18] among passages were left unchanged, while the other half unothers. They fail to identify the qualities that make the derwent a perturbation. More specifically, we devised shufled text incoherent, do not pinpoint the linguistic two distinct perturbation strategies: devices responsible, and overlook the subjective component underlying coherence. Additionally, most existing • swap: it involves swapping the position of two benchmarks are limited to the English language. random sentences in the text passage.
Our contribution In this paper we seek to address • substitution (sub): it consists of replacing one of some of the existing deficiencies in coherence assessment the four sentences with another sentence, correby presenting a novel resource tailored for the Italian lan- sponding to the 10th sentence following the pasguage, designed specifically to study text coherence from sage in the same document. the perspective of human perception. The dataset, which to our knowledge is the first for Italian, has been recently Table 1 contains an example from the corpus for each used as part of a larger benchmark released for DiSCoTex, perturbation type one of the shared-tasks presented at the 8th evaluation campaign of NLP and speech tools for the Italian lan- 2.1. Collecting human ratings guage (EVALITA 2023) [1]. The results of first analyses on the resource shed light on the influence of both genre and text perturbations on perceived coherence1.
we added ten fillers to the dataset, consisting of foursentence passages deliberately chosen to be either highly coherent or highly incoherent. These additional passages 2. Dataset Construction served as a control mechanism to check the reliability and accuracy of each annotator in assigning coherence The construction of our dataset was guided by two dis- scores to the actual texts in the dataset: if an annotator tinct criteria: on the one hand, we intended to explore assigns an out-of-scale coherence value to these texts, it the efect of textual genre on the human perception of co- suggests that they might not have conducted the annotaherence; on the other hand, we wanted to assess whether tion process adequately. and to what extent humans are sensitive to diferent The annotation process has been executed via crowdstrategies introduced to artificially modify an original sourcing. We first used the Questbase2 platform to create text. questionnaires formulating the text scoring process in
As a starting point we selected texts from two distinct the form of questions. Then, we distributed the quessources: the Italian Wikipedia and the Italian speech tran- tionnaires using the crowdsourcing platform Prolific 3, scripts section of the Multilingual TEDx corpus (mTEDx). choosing to recruit only Italian native speakers withThe choice of these sources was meant to obtain a bal- out language disorders as annotators. Considering that anced corpus that was representative of two diferent subjective component underlying coherence that makes language varieties: the former is a ‘standard’ written this concept gradual rather than categorical, people were variety, and the latter a ‘hybrid’ variety combining di- asked to rate each texts on a 5-point Likert scale, where verse genres (e.g., university lectures, newspaper articles, 1 represents the minimum value of perceived coherence conference presentations, and TV science programs) as and 5 the maximum4. A pilot experiment tested the suitwell as diferent semiotic modes, such as written, spoken, ability of the questionnaire from diferent points of view audio, and video [19]. (i.e. the clearness of instructions) and allowed to estimate
Following the approach by Brunato et al. [17], for each the time needed to complete it. After collecting all the text we then proceed to extract passages consisting of responses to the questionnaires, we kept only the most four consecutive sentences, considering them as our unit reliable annotations by filtering out the annotators who of analysis for modeling the coherence annotation task. had failed the attention checks. Specifically, we excluded As for Wikipedia, we relied on the existing segmentation those annotators who rated at least four control texts into paragraph and extract four-sentence passages. For incorrectly, i.e. assigning a value from 1 to 3 to highly the TEDx corpus, as these texts lack such an internal coherent filler passages or a value from 3 to 5 to very structure, we split all the transcripts into passages of four incoherent filler passages. As a result, we retained an sentences. average of 10 annotations per passage for a total of 10,567
After creating all the possible passages, we randomly annotations for the whole dataset. selected 1,064 of them while maintaining a proportional representation from both sources. Half of the extracted
poses at the following link: http://www.italianlp.it/resources/
3https://www.prolific.co/ 4Appendix A contains the instructions given to the annotators when opening the questionnaire on Prolific.
Text passage 1. Clif Burton possedeva uno stile impeccabile ed era capace di produrre giri di basso potenti ma allo stesso tempo rafinati. 2. Specialmente durante gli assoli era solito pizzicare due o tre corde nello stesso momento e lanciarsi in un complesso uso di distorsioni, tapping, bending e applicazioni del pedale wah wah. 3. Il suo stile era molto vario per i canoni di un bassista heavy metal: Burton non suonò mai il basso “come un chitarrista” e mai utilizzò plettri, prediligendo il contatto diretto con le corde, pizzicate a mani nude. 4. Anche per questo, diversamente da altri bassisti heavy metal che utilizzavano bassi a cinque o sei corde, Burton suonava solo bassi a quattro corde, che considerava più adatti al suo stile. 1. È stato teorizzato che le prime stelle dell’universo, le cosiddette stelle di Popolazione III, fossero molto più massicce delle stelle attualmente esistenti. 2. Si è postulata l’esistenza di questa prima generazione di stelle per spiegare l’esistenza di elementi chimici diversi dall’idrogeno e dall’elio nelle stelle più vecchie conosciute. 3. Sebbene fossero più grandi e luminose di tutte le supergiganti note oggi, la loro struttura doveva essere molto diferente, con perdite di massa molto più contenute. 4. Nella maggior parte dei casi la variabilità è dovuta a pulsazioni della superficie stellare.
tion of text coherence, we conducted two types of analyses that examine the relationship between perceived coherence and text structure from distinct perspectives. The first one focuses on the efect of the diferent perturbation strategies artificially introduced to disrupt the internal coherence of rated passages; the second one takes into account solely the subset of original, i.e. unperturbed, texts with the aim of exploring the efect of several linguistic features extracted from each passage on the mean coherence judgments. 3.1. Impact of text perturbations
is .32 for the whole corpus.
After gathering all annotations, we studied their homogeneity by calculating for each passage the mean value and standard deviation of the coherence scores assigned to it5. These statistics were computed for the whole dataset as well as for passages grouped according to the a more standardized structure, with internal coherence text source from which they derived (TED or Wikipedia) remaining relatively stable even when subjected to minor and to the perturbations eventually applied. The purpose alterations, such as changes in sentence order or the inwas to observe how coherence ratings vary among the clusion of an intruder sentence from the same document. diferent groups and understand the efects of the dif- In all groups the standard deviation has low values, ferent artificial perturbations applied to the text. These suggesting a high degree of homogeneity in the ratings, results are shown in Figure 1. especially for Wiki passages.
Observing the trend of the distribution of the mean co- A more in-depth investigation was conducted to asherence ratings for each group, it was possible to see that sess the potential impact of certain factors of each perthe group containing all the original texts was consid- turbation type, such as the distance between swapped ered as more coherent than the ones with the perturbed sentences and the position of the replaced sentence, on texts. However, in all considered groups, texts extracted the distribution of mean coherence scores. Regarding the from Wikipedia were rated as more coherent than those swap perturbation, we assigned a label to each perturbed extracted from TEDx, even when artificially perturbed. text, indicating the distance between the swapped senThis suggests that Wikipedia documents tend to exhibit tences. Dist_0 was given to passages where the swapped sentences were adjacent, Dist_1 to those with one sentence between the swapped sentences, and Dist_2 to those with two sentences between the swapped sentences. morpho-syntactic and syntactic properties of text. An Similarly, we assigned a label to the passage that under- overview of these features is shown in Table 2. These went the substitution perturbation. Pos_1 was given to features were shown to be relevant for modeling aspects passage where the first sentence was replaced, Pos_2 to characterizing the interaction between a reader and a those where the second sentence was replaced, Pos_3 text, such as the human perception of sentence complexwhen the third sentence was replaced, and Pos_4 when ity [20] and of writing quality [21]. As both complexity the fourth sentence was replaced. and quality are properties connected to coherence, we expect that these features will provide valuable insights for our coherence analysis as well.
We then sought correlations between the human perception of coherence and these linguistic features by calculating the Spearman correlation coeficient between the average coherence score attributed to each passage and the average value of each linguistic feature extracted from it. Results are shown in Table 3. Considering only features with p-value below the threshold of 0.05, we observed that the perception of coherence in original texts positively correlates above all with features closely related to length. In fact, the highest correlation (Spearman’s r = 0.32) was obtained with the maximum depth of syntactic tree, followed by tokens_per_sent and n_tokens, Figure 2: Mean coherence judgments attributed to perturbed which captures respectively the average sentence length passages grouped by distance (for swap perturbation) and and of passage length in number of tokens. Also lexical position (for substitution perturbation). richness, measured by the average value of Type/Token Ratio (ttr_form_100) turned out to be among the first top-five correlated features. These findings suggest that
As we can see in Figure 2, in texts perturbed with the longer sentences could contain more information and swap perturbation, the perceived coherence is higher thus lead to a more complete text, that also makes it in those cases where the sentences are adjacent to each more coherent. An interesting result was that the use other, while decreases as the distance between the ex- of pronouns negatively correlated with the perception changed sentences increases. In texts altered via substi- of coherence (Spearman’s r = -0.26). This unexpected tution, the perceived coherence increases especially in observation can be attributed to the potential ambiguity those cases where the sentence substitution occurred at of pronouns, deriving from the fact that the evaluated the ends (thus in the first and last positions), while it passages were extracted from larger texts and might lack generally decreases in cases where substitution involves the necessary context to accurately link the pronoun to the middle positions. its intended referent.
Focusing specifically on original passages derived from Wikipedia, we observed that the presence of features de3.2. Impact of linguistic structure scribing proper syntactic phenomena closely related to Although the previous analysis revealed that perturbed the sentence length, such as the average length of detexts were rated on average as less coherent that original pendency links and of the maximum link (avg_links_len, ones, we also observed that such a perception is influ- max_links_len), along with the presence of nouns modienced by textual genre. Considering the subjective nature ifed by prepositional phrases ( prep_chain), contributed of coherence, we hypothesize that even well-formed texts to increased coherence perception by annotators. These may receive diferent coherence annotations. We thus linguistic features that are typically related to syntactic carried out a final analysis focused solely on the subset complexity may suggest that these texts are also more of original texts, with the aim of investigating the rela- informative, resulting in enhanced coherence perception. tionship between the linguistic profile of these texts and Furthermore, it could be seen that in texts taken from the perceived coherence. Wikipedia the judgement of coherence was positively
To automatically extract linguistic information from influenced by a paratactic structure of the text ( dep_conj). human rated passages, we leveraged Profiling-UD [20], Finally, in the TED original texts it could be observed that a tool for carrying out linguistic profiling investigations the correlation was positive in the case of the distribution in multiple languages based on the Universal Depen- of subordinate propositions (subord_dist) and negative dency framework. Using Profiling-UD, we extracted more in the distribution of main ones (princ_dist). than 130 features for each passage, which capture lexical,
ing and computationally modeling text coherence in the Italian language, focusing on human perception. The investigation into genre and text perturbations revealed a significant interplay between the two dimensions. Interestingly, text passages from Wikipedia were rated on average as more coherent than those extracted from TEDx talks even when presented in a perturbed form. Furthermore, a deeper analysis of the perturbations revealed distinct efects on coherence perception. Modifications that disrupted coherence by altering the sentence order or introducing intruder sentences had varying impacts. Notably, coherence judgments also varied for original texts, and the syntactic structure and complexity-related features emerged as influential factors in human assessment.
In the future we would like to gain deeper insights into the underlying factors that influence coherence perception by also incorporating a diverse range of text genres and perturbations. This deeper understanding of coherence will have significant implications for the development of more sophisticated language understanding and generation systems.
raters:
“Ciao! In questo sondaggio ti chiediamo di leggere dei testi e di valutarne il livello di coerenza, assegnando un punteggio che va da 1 (per nulla coerenti) a 5 (del tutto coerenti).
Innanzitutto, ti diamo una breve definizione di coerenza: in ambito linguistico, questa parola si usa per indicare una caratteristica che riguarda l’organizzazione del significato di un testo.
Un testo è considerato coerente se le singole unità di cui si compone (tipicamente le frasi) sono connesse tra loro in modo da formare un’unità più ampia che il lettore/ascoltatore considera globalmente appropriata, sia dal punto di vista dell’ordine logico-temporale sia rispetto al contenuto principale del discorso. Tuttavia, questa valutazione è molto personale: la coerenza, infatti, dipende sia da fattori legati alla struttura linguistica e al contenuto del testo, sia da fattori soggettivi, come la familiarità del lettore/ascoltatore verso l’argomento, la sua padronanza linguistica, il grado di interesse ecc. Proprio per questo ti chiediamo di valutare ciascun testo con la maggior naturalezza possibile, dal momento che non c’è una risposta giusta o sbagliata: quello che ci interessa è proprio la tua percezione personale! In generale, per orientarti nel giudizio, puoi pensare che un testo molto coerente dovrebbe risultarti facile da comprendere, ben strutturato e non dovresti avvertire discontinuità sul piano logico e del contenuto nel passaggio tra una frase e l’altra. Ad esempio, il testo che segue dovrebbe ottenere un punteggio di 4 o 5:
E quindi che si fa? E quindi mi danno in mano un depliant dell’Università e dicono: "Bene ragazzo. Scegli una facoltà a numero aperto e, nel momento in cui qualcuno ad Economia molla, puoi subentrare te." "Benissimo!" dico. Apro il depliant dell’Università, salto a piè pari Ingegneria per l’eccessiva presenza di Chimica e tra Fisica, Filosofia, Lettere, Matematica e Informatica inizio a decidere che cosa fare.
Stiamo parlando degli anni Trenta. Le aziende, le persone che puntano al futuro, le protagoniste di questa trasformazione, non sono assolutamente associate a queste parole, semmai a: tecnologia; precisione; elettronica; digitale; meccanica; futuro. Tutte parole, queste, associate invece al termine "meccatronica". Solo il cinque per cento dei funghi che potenzialmente esistono sono stati descritti, quindi c’è veramente un mondo da scoprire sotto i nostri piedi.
Inoltre, per la tua valutazione, tieni presente che tutti i testi che leggerai non sono completi. Si tratta infatti di paragrafi di poche righe, estratti da sezioni diverse (es. introduzione, corpo, conclusione) di documenti più lunghi, che provengono da varie fonti (es. testi di Wikipedia, dialoghi trascritti). Infine, ti ricordiamo che il sondaggio è indirizzato alle persone di madrelingua italiana e la sua compilazione richiederà all’incirca 20-25 minuti.
Grazie in anticipo per la partecipazione!"
“Hello! In this survey, we ask you to read texts and evaluate their level of coherence by assigning a score ranging from 1 (not at all coherent) to 5 (completely coherent).
Firstly, we provide a brief definition of coherence: in a linguistic context, this word is used to indicate a characteristic related to the organization of the meaning within a text.
A text is considered coherent if its individual units (typically sentences) are connected in a way that forms a broader unit that the reader/listener perceives as globally appropriate, both in terms of logical-temporal order and the main content of the discourse. However, this evaluation is highly subjective: coherence depends on factors related to the linguistic structure and content of the text, as well as subjective factors such as the reader/listener’s familiarity with the topic, linguistic proficiency, level of interest, etc. That’s why we ask you to assess each text as naturally as possible, as there is no right or wrong answer: what we are interested in is your personal perception! In general, to guide your judgment, you can consider that a highly coherent text should be easy to understand, well-structured, and you should not perceive any discontinuity in logical and content transitions between sentences. For example, the following text should receive a score of 4 or 5:
E quindi che si fa? E quindi mi danno in mano un depliant dell’Università e dicono: "Bene ragazzo. Scegli una facoltà a numero aperto e, nel momento in cui qualcuno ad Economia molla, puoi subentrare te." "Benissimo!" dico. Apro il depliant dell’Università, salto a piè pari Ingegneria per l’eccessiva presenza di Chimica e tra Fisica, Filosofia, Lettere, Matematica e Informatica inizio a decidere che cosa fare.
On the contrary, a text with low coherence should be more dificult for you to understand, poorly connected, and discontinuous on a logical and structural level. For example, the following text should receive a score of 1 or 2:
Stiamo parlando degli anni Trenta. Le aziende, le persone che puntano al futuro, le protagoniste di questa trasformazione, non sono assolutamente associate a queste parole, semmai a: tecnologia; precisione; elettronica; digitale; meccanica; futuro. Tutte parole, queste, associate invece al termine "meccatronica". Solo il cinque per cento dei funghi che potenzialmente esistono sono stati descritti, quindi c’è veramente un mondo da scoprire sotto i nostri piedi.
Furthermore, for your evaluation, please keep in mind that all the texts you will read are not complete. They are short paragraphs extracted from diferent sections (e.g., introduction, body, conclusion) of longer documents, coming from various sources (e.g., Wikipedia texts, transcribed dialogues). Finally, we remind you that the survey is aimed at Italian native speakers, and it should take approximately 20-25 minutes to complete.
Thank you in advance for your participation!"