Argumentative essay writing is a fundamental objective in education for both vocational schools and high schools in Italy, as indicated in [1,2]. It requires students to present arguments supported by personal knowledge or external sources in a coherent and convincing manner. However, writing coherent texts poses both cognitive and linguistic challenges to novice writers and textual competences related to it are frequently claimed to be insufficient, putting pressure on the educational system. Automatically discerning incoherent texts or passages could help teachers to better understand students' problems and give targeted instructions, while students would benefit from more frequent and more timely feedback. However, to date, most NLP research in automatic coherence modelling focused on semantic similarity between two parts of texts using mostly well-formed newspaper or Wikipedia texts, offering little information for educational contexts. In this study, we explore coherence from an educational perspective, utilizing recent language models and data perturbation techniques to probe their value for linguistically informed and informative automatic coherence evaluation for student essays. While large language models have been used successfully in domain general coherence modelling before, we test their effectiveness for text analysis in this domain-specific scenario, taking into account both surface and non-standard language features. We discuss:
• data perturbation techniques to artificially reproduce real-life scenario incoherence in textual data • a custom probing task design • automatic evaluation of coherence using different encoding models
The results of our experiments show the performances of encoder models in recognizing patterns of (in)coherence in a domain-specific educational context such as upper secondary school student essays. The paper is organized as follows: Section 2 provides an overview of previous approaches to coherence modelling and NLP data perturbation with a focus on Italian NLP. Section 3 introduces the data we used for this study, giving information on the research project it originates in as well as on the corpus design and annotation. Section 4 provides a detailed description of our methodology introducing our custom probing tasks (Section 4.1), used Models (Section 4.2.1) and text encoding 4.3 as well as a description of the two analyses performed (Section 4.4 and Section 4.5). Sections 5 and 6 present and discuss our results and Section 7 concludes the article with final considerations.
Coherence modeling is an important task in natural language processing (NLP) with potential impact on other NLP tasks such as Natural Language Understanding or automated essay scoring. Early work on coherence modelling focused on the definition of the phenomenon [3,4,5,6,7] and provides valuable frameworks such as Centering Theory [8,9] and Entity-Grid approach [10].
Following the great development of neural network systems in recent years, many works such as [11,12,13,14] explored coherence modelling implementing further and more sophisticated solutions for the English language.
Recently, the Italian NLP community has approached the topic from an engineering point of view, using Italian pre-trained neural models to distinguish coherent from (mainly synthetically constructed) non-coherent texts [15,16,17,18]. Some efforts were also made for multilingual scenarios [19] demonstrating the encoding capabilities of multilingual models for coherence features.
In data perturbation, dataset entries are corrupted with specific computational operations to simulate noise condition and test the model performance on real world conditions [20]. Many studies on data perturbation and data augmentation in NLP focus on model agnostic methods [20,21,22,23] using random deletion, random swap, synonym replacement, random insertion and punctuation insertion techniques for text classification with limited amount of data. More sophisticated and task-oriented data augmentation approaches are proposed for sentiment analysis [24], hate speech classification [25], hypernymy detection [26] and domain specific classification [27].
The data used in this study originates from a research project, conducted in South Tyrol between 2020 and 2024. The project named ITACA: Coerenza nell'ITAliano Accademico [28] had the aim to study textual competences of students in their first language Italian with particular focus on aspects of text coherence. Within the project various outcomes have been produced: a corpus of Italian student essays collected in Italian South Tyrolean upper secondary schools, a validated rating scale to evaluate coherence in student essays, and coherence ratings for texts in the corpus from three independent raters using the previously developed rating scale. The products are described in the following section.
The ITACA corpus1 is an annotated learner corpus created within the project ITACA: Coerenza nell'ITAliano Accademico [28]. It consists of a total of 636 argumentative essays from Italian L1 upper secondary school students from the autonomous province of Bolzano/Bozen2 during the school year 2021/2022. The texts were collected by asking 12th grade students to type an argumentative essay following precise indications of writing time, text length and topic. The full assignment can be consulted in the Appendix B. While the assignment asked for a minimun text length of 600 words, the average number of tokens in the essay is with 668, just slightly above the minimum length requirement.
The totality of the 636 collected texts constitutes 382,964 tokens. All data were collected digitally and anonymously and underwent subsequent control and cleaning procedures, partly manually, to ensure their integrity and to guarantee the anonymity of the participants. Essays were collected, by asking students to type their essays into an input field in an online form, additional metadata was collected by a subsequent online questionnaire asking for basic socio-demographic information, students' language background, and reading and writing habits. The whole corpus was automatically tokenized, lemmatized and annotated for part-of-speech and syntactic dependencies with the support of project collaborators from Fondazione Bruno Kessler, who also supported the project in the setup of an interface for manual annotation based on Inception [29].
A manual annotation of a subset of 388 texts was performed by two trained annotators and offers detailed descriptions of the text's structure, with a focus on the use of various linguistic features (such as punctuation, connectives, agreements, anaphora, contradictions) that enhance or limit the text's cohesion and coherence. The manual annotation of the corpus was guided by the three sections elaborated in [30] and contained annotations for traits of incoherence referring to 1. segmentation (e.g. splice comma, added comma, not-signed parenthetical clause) 2. logic-argumentative plan (e.g. issues in the use of connectives, contradictions) 3. thematic-referential plan (e.g. critical agreement, critical anaphora, not-expanded comment)
The corpus is accessible through an ANNIS search interface 3 and can be downloaded in various formats from the Eurac Research Clarin Center (ERCC) under the CLARIN ACADEMIC END-USER LICENCE ACA-BY-NC-NORED 1.0 licence 4 . Downloads and further documentation can also be accessed via Eurac Research's PORTA platform 5 .
Each single essay was additionally manually evaluated in a double-blind manner by a panel of six experts who applied a specially created, rating scale, which was subsequently validated to assess textual coherence. The items were rated on a Likert scale from one to ten and referred to three dimensions of coherence (structure, comprehensibility, segmentation). The average structure score 𝜇 is attested at 4.55 with standard deviation 𝜎 = 5. For comprehensibility, 𝜇 = 6.29 and 𝜎 = 1.65, while for segmentation 𝜇 = 5.99 and 𝜎 = 1.79.
In this study, we focus on NLP data perturbation [20,21] and custom probing tasks [31] to evaluate the ability of Italian BERT models of discerning features of coherence given different pre-training conditions and fine tuning.
In our analysis, we aim to evaluate automatic coherence modelling techniques, applying them to student essays with varying degrees of well-formedness and coherence.
We conducted a number of experiments probing whether state-of-the-art coherence modelling techniques based on BERT encodings would be able to distinguish between original, i.e. allegedly coherent texts and those containing features of incoherence identified for student writing before. In our case study, we use data perturbation techniques to reproduce specific students' errors observed during the textual analysis of the ITACA project [28] (see Section 3), in order to apply text modification in a fully controlled fashion. We used representations obtained from BERT [32] models to demonstrate the ability of automatic systems to encode patterns of (in)coherence in a specialized scenario such as Italian student essays and evaluate their potential for educational purposes.
Using data perturbation techniques, we aim to reproduce both general-purpose coherence modelling perturbation strategies and modifications inspired by some of the most salient features of textual (in)coherence observed in the annotation process for the ITACA project. These include incoherent order of arguments and sentences, incorrect use of connectives, overuse of polyfunctional connectives, unresolved co-reference, the use of splice comma and an overuse of paratactical constructions. Assuming that students would not produce the these Although data perturbation can also operate on the character level, we opted for token-and sentence-level approaches maintaining parameters in a controlled setting.
We implemented the following custom probing tasks:
As in other synthetic datasets for coherence modelling [15] this data perturbation technique is to randomly shuffle sentences within the texts.
In order to imitate texts in which the logical connection between phrases is erroneous, we randomly substituted connectives used in the text exploiting both manual and automatic processing with Stanza6 ; To identify the connectives to substitute, we referred to a string matching of all connectives listed in the Lexicon of Italian Connectives (LICO) [33].
Based on the ITACA corpus annotation scheme, we implement a probing task, imitating young writers tendency to use simple polifunctional connectives instead of highly semantically loaded ones. For this, we substitute all connectives in the text by the polyfunctional connective "e".
For a very simplistic approximation of corrupted anaphoric references, we identified pronouns with Stanza and replaced them randomly by other pronouns isoleted from the corpus. To ensure a minimum of correct pronouns, only 50% of the pronouns in the text were corrupted.
A splice comma is the use of a comma to join two independent sentences. The comma can substitute a dot, a colon, or semicolon [34,35,36,37]. In our case, long pause markers such as periods, colons, or semicolons were substituted with a comma. We apply the perturbation to just 50% of the conjunctions in the text to partially keep punctuation unaltered.
Example Sentences under Text Perturbations. The example corresponds to the English "This morning I went to the market. I bought some apples and oranges. Then I went back home and baked a cake"
Coordinating conjunctions extracted with Stanza are substituted with punctuation taken from a list to create paratactic sentences. We apply the perturbation to just 50% of the conjunctions in the text to keep some conjunctions untouched.
Text perturbation examples can be consulted in Table 1 4.2. Models
For our experiments, we test three different BERT-based models to obtain vector representations for our probing tasks.
1. BERT-ita base [38]: trained with Italian data from the OPUS corpora collection 7 and Wikipedia 8 .The final training corpus has a size of 13GB and 2,050,057,573 tokens. 2. GilBERTo9 : RoBERTa based model [39]. The model is trained with the subword masking technique for 100k steps managing 71GB of Italian text with 11,250,012,896 words [40]. The team took up a vocabulary of 32k BPE subwords, generated using SentencePiece tokenizer [41].
Inspired by the works of [42] and [43], the BERT-ita model was fine-tuned using a dataset of high school es-
We retrieved vector representations and performed a binary text classification experiment for each perturbation technique 11 . The model is fed with batch size = 1 with all the texts contained in the set. To overcome the length input limit of 512 tokens imposed by BERT models and process the entire text in a row with no loss of contextual information, we split the text into two segments when reached the max input lenght. Furthermore, we adopted a mean-pooling strategy by calculating the mean between the last hidden state of each contextualized token embedding in the batch across the input sequence length.
The final text representation is the mean of all segment embeddings in the batch.
We first perform a model performance analysis, comparing the model performance in classification for each of the custom probing tasks with each of the three models. Classification is performed with a Random Forest classifier [44], defining each experiment as a binary classification between the original and perturbated texts. The classes were balanced across the entire dataset. To optimize the amount of available data for training and testing, we use 10-fold cross-validation for evaluation. We compare model performance against a majority class baseline (0.5 for balanced binary classification) and against each other using f1 scores.
In a subsequent analysis, we compare the model predictions of our best-performing model with the human coherence ratings provided for the corpus. In order to obtain a single coherence score for each essay, the scores were averaged over the different annotators and the three components (structure, comprehensibility and segmentation; see Section 3). We perform an error analysis by comparing the predictions for unmodified texts with the highest and lowest coherence scores using a random forest classifier trained with the model that achieved the best results in the model comparison. Assuming that all tasks have the same weight, we select the best performing model according to the average f1 score achieved in the model performance analysis (see Section 4.4). The train set for this evaluation corresponds to 90% of the data, while the test set represents the 5% of essays with the highest (𝜇 = 8.28, 𝜎 = 0.36) and the 5% with the lowest coherence scores (𝜇 = 2.63, 𝜎 = 0.51). Finally, we interpret the results, manually investigating texts that were misclassified as modified texts from both tails of the test set.
The classification experiments show the ability of the BERT models to encode the features of (in)coherence represented by the perturbation techniques introduced in Section 4.1. The following sections illustrate our findings for the BERT model comparison and the error analysis conducted on a selected subset of non-modified texts.
F1 scores for most models were very similar with just small differences between the three models. In average, GilBERTo was found to be the best performing model for most tasks, probably due to its higher amount of training data and its lighter model architecture. However, we do not expect these differences to be significant. Except for the improvement in the shuffling task after fine-tuning, the ITACA-bert model remains comparable to its base version, probably due to the scarcity of domain-specific training data. Results showed that models achieved better performance on semantic tasks such as polyfunctional conjunction perturbation or pronoun perturbation while struggling with syntactic probing tasks such as shuffling and splice comma perturbation. For the shuffling task, a considerable improvement can be observed after finetuning (+0.12% from F1 = 0.38 to F1 = 0.50). However, neither of the shuffling models performs better than a random baseline, while the splice comma experiment models performed slightly better, with the BERT-ita and Gilberto models marginally beating the baseline of 0.5. A graphical comparison between model performances can be seen in Figure 1.
A detailed overview of the classification results for single tasks and models can be found in the Appendix A. The tables provide measures of the f1 score for each experiment and model.
To better observe the encoding and classification performance of BERT, we decide to isolate the texts with the highest and the lowest coherence scores according to the average coherence scores as specified in 4.5. The resulting test set corresponds roughly to the 10% of the total number of texts in the corpus. Our expectation is that texts with lower coherence scores have a higher chance to be misclassified as modified texts, while texts with higher coherence scores should not lead the classifiers to identify traits of incoherence as specified in the cus- tom probing tasks. We perform all analysis using the GilBERTo model for text encoding, as it was revealed to be the best performing model when averaging f1 scores on all tasks of the model performance analysis (see Section 4.4). However, we exclude the shuffling task as model performance was below the baseline and therefore too low for interpretation. Thus, we train a random forest classifier with the 90% of the train set, for all custom probing tasks described in Section 4.1.
Our results show that the distribution of misclassified labels is generally skewed toward texts with lower coherence scores, but misclassifications for texts with higher coherence scores were also found. While the splice comma and polyfunctional conjunction (see Figure 2) probing tasks showed clearly more misclassifications on the lower tail of the dataset, also well-rated texts were occasionally misclassified as perturbed texts. On the contrary, the small number of misclassifications on the parataxis and pronoun perturbation probing tasks might suggest that the operationalizations taken in this work are too simplistic to be representative of students' mistakes in the texts and, therefore, not able to pick up on traits of incoherence present in the students' essays. The results of the experiment can be consulted in Appendix A.
Although data perturbation cannot fully reproduce the variability of real-word students' mistakes, our results give precious insights about the ability of BERT encoders to capture degrees of coherence on both syntactic and semantic level. Of course, the efficiency of the data perturbation might be influenced by several factors, such as the fact that the original texts used for our experiments already naturally contain errors of the same or other types. However, we argue that this is the case for any type of data set of unknown quality that is subject to automatic coherence evaluation. Thus, before the evaluation, texts have not been subjected to any review and, excluding other external factors, they reproduce real-world writing conditions. The results of language encoding and classification depend on the difficulty of the perturbation task and on the original training of the BERT model. However, despite the fact that the BERT-ita base and GilBERTo exploit different training strategies, no drastic performance fluctuations have been observed on our selected language tasks. Even though the effects of fine-tuning with domain-specific data is limited to the amount of affordable data, the effect can already be observed by looking at the increment on the shuffling task performance.
The classification of the evaluation set highlighted the potential of data perturbation techniques for the encoding of (in)coherence features. Previous approaches to coherence modelling implemented solutions inspired by theoretical intuitions. In our case, we decided to start from natural textual errors and check the ability of the model in capturing the same features presented in the text. For a more transparent interpretation of results and explanation of individual classification it would be of interest to check how attention maps change according to the tuning of the model [45].
In this paper, we presented an evaluation of coherence modelling techniques for detecting incoherence in student essays based on surface-level features of incoherence. We used the ITACA corpus of Italian upper secondary school essays to perform a number of classification techniques using data perturbation and BERT-based text encoding methods. After a preliminary comparison between pre-trained and fine-tuned models we adopted the best performing one according to our results. The results of the chosen tasks are influenced by the implementation of the perturbation technique, the encoding ability of the model, and the amount and the quality of the data the model is pre-trained on. The best performances are bounded to the model pre-trained with the highest amount of data (GilBERTo). We based our evaluation on simple f1 measures considering this sufficiently indicative of the encoding ability of the model applied to each specific probing task. Since we mainly tested custom perturbation techniques and the encoding abilities of BERT models, future research directions might involve data perturbation techniques enhancement, XAI techiques for model behaviour analysis [46,45] and the exploitation of state-of-the-art generative one shot and few-shot models in a highly domain-specific scenario such as school essays writing.
We thank Fondazione Bruno Kessler Trento for their support on the ITACA corpus and for allowing us to use their student essay dataset for fine-tuning.