This report presents the system developed by the IUSS-NeTS team for the LangLearn evaluation task of the 2023 edition of EVALITA [1], which focuses on tracking the linguistic development of native Italian speakers (L1) and second language learners of Spanish (L2). Our approach focuses on the integration of specific linguistic features and surprisal-based metrics for both Italian and Spanish datasets (CItA and COWS-L2H, respectively). With our tests the system achieves a F-score of 0.67 for the Italian test set and 0.75 for the Spanish test set. These results demonstrate the efectiveness of our methodology in capturing and analyzing the linguistic development of L1 Italian and L2 Spanish learners.
data made available for the task;
• prediction acquired utilizing also extra re- datasets for both Italian L1 and Spanish L2.
language development assessment. It is defined as a binary classification problem where participants are asked to predict the relative order of two essays written by the same student, for both Italian L1 and
to train the models, the task was separated into two sub-tasks: • prediction obtained using only the training sources (e.g. additional data for the training phase).
Building upon prior research in the field [
BY 4.0). 1, 2, 1, 2 . The essays’ chronological placement varied between the first and second year of lower secondary school, contingent upon when they were written; meanwhile, the essay number denoted its sequential position within the given year.
L2 and Heritage speakers); it comprises 3498 brief
essays produced by second language learners at- [
Each academic term spanned a specific time range: 3.2. Surprisal
and F from October to December1.
classification also by using surprisal-based metrics extracted from the constituent texts of the two Lan2.2. Test set gLearn corpora: it has in fact been observed that the surprisal value of a word is associated with The test set for the CItA corpus consisted of 278 behavioral measures of processing complexity [11]. texts organized into 307 pairs, whereas the test set Surprisal-based metrics have already been used in for the COWS-L2H corpus comprised 471 texts or- various contexts, such as the analysis of pathologganized into 320 pairs. It is important to note that, ical speech [12] and Italian linguistic studies [13]. unlike in the train datasets, the essay identifying When referring to surprisal, we are talking about acronyms were presented in a random order. one of the many possible log-transformed probability measures expressing the likelihood that a token follows a sequence of tokens (or it is placed in a 3. System description certain position into a given sequence of tokens).
In this section we present our approach for the models to be used in this case, testing 13 models
automatic tracking of Italian L1 and Spanish L2 on diferent linguistic structures (such as relatives,
language development. Our method can be summa- passives, questions, various typologies of errors) for
rized in a binary classification based on the combina- both Spanish and Italian. One important trait we
tion of linguistic complexity features and surprisal- considered was the low tolerance for errors:
whenbased metrics. ever a morphosyntactic error is present, certain tools
tend to normalize the ill-formed token eventually
ne3.1. Linguistic complexity features glecting the relevant features that, in fact, indicate
The aim of our approach was to evaluate the rel- true mistakes. Expanding on previous studies [14],
evance of specific morphosyntactic features in the the incorporation of surprisal-based metrics was a
assessment of the linguistic competence improve- deliberate efort to preserve the meaningfulness
inment both in L1 and L2. With this in mind, we herent in error occurrences. We set out to evaluate
selected a set of robust Natural Language Process- which model demonstrated the least inclination to
ing (NLP) tools which allowed us to extract linguis- normalize errors, examining the surprisal values
gentic features relevant to the classification task and erated by diferent models. We gave preference to
which was compatible with both Italian and Span- those models that yielded higher values of surprisal
ish. After an extensive evaluation of various tools when faced with various types of errors,
indicat(Tint2, Profiling UD 3, READ-IT4 and CTAP5), we ing a recognition of the mistakes, over models that
decided to rely on CTAP: Common Text Analysis produced lower values leading to normalization of
Platform [
2https://dh.fbk.eu/research/tint/ 3http://www.italianlp.it/demo/profiling-ud/ 6https://huggingface.co/dbmdz/bert-base-italian-cased 4http://www.italianlp.it/demo/read-it/ 7https://huggingface.co/dccuchile/bert-base-spanish-w 5http://sifnos.sfs.uni-tuebingen.de/ctap/ wm-cased
Surprisal-based metrics text surprisal KURT text surprisal MAX text surprisal MEAN text surprisal MEDIAN text surprisal MIN text surprisal OUT F text surprisal OUT N text surprisal Q1 text surprisal Q3 text surprisal SKEW text surprisal STD text surprisal IQR
kurtosis maximum value mean median minimum value outliers (frequency) outliers (number) ifrst quartile third quartile
skewness standard deviation interquartile range performed by both models, we made the decision attribute and the output variable and selected only to introduce an additional preprocessing step which those attributes that had a moderate-to-high posiinvolved utilizing Stanford’s parser Stanza8 for to- tive or negative correlation (close to -1 or 1), discardkenization of both the Italian and Spanish texts, ing the ones with a low correlation (value close to 0). prior to further analysis. Stanza is a Python nat- This made it possible to eliminate attributes that ural language processing toolkit that supports 66 were not significant in our analysis. For the Italian diferent human languages. Stanza’s text analysis language we had initially 266 attributes which bepipeline is language agnostic (in other words, lan- came 40 after the attribute selection phase, while for guage indipendent): the parser has been trained on the Spanish language we had 400 attributes which 112 datasets, and it has been found that the same became 75. To make a good use of the two training neural architecture generalizes eficiently and per- datasets, we performed the analysis with the crossforms well across all languages [15]. Following the validation method (also known as k-fold validation), discussed methodology, we were able to calculate on which is particularly useful to reduce the overfitting both CItA and COWS-L2H corpora 12 statistical problem. To obtain the most comprehensive view measures (table 1) based on the extraction of word we then evaluated several possible classification alsurprisal, which were then added to the linguistic gorithms ofered by Weka. We finally observed that features previously extracted through the CTAP the best performing algorithms were the Logistic tool. regression, J48 and Random Forest, with the last one being the absolute best and thus our choice 3.3. Classification for the final analysis. Random Forest is a class of ensamble methods which aggregates the prediction of diferent classifiers to improve accuracy; Weka makes available the implementation of the algorithm described in [18].
For the classification phase we decided to use Weka [16], since its rich selection of algorithms, ease of use, feature selection, evaluation capabilities and integration options make it a favorable choice for classification tasks 9. Firstly we constructed ARFF ifles associating each text of the two corpora with 4. Results the attributes consisting of the features extracted through CTAP and the 12 surprisal-based metrics. The final results yielded an F-score of 0.67 for the We set our classification in this way: give 1 if it Italian test set and 0.75 for the Spanish test set. is true that the 1 (i.e. the first essay of the Weka facilitated the extraction of diverse informapair) was written before 2 (i.e. the second essay tive measures regarding the classification perforof the pair), otherwise give 0. We pre-processed mance (detailed performance metrics during the our files and we performed attribute selection on 10-fold cross-validation phase can be found in tathe extracted linguistic features and surprisal-based bles 2 and 3). metrics: we calculated the correlation between each It is crucial to interpret all this information collectively to gain an accurate feedback on the classification. Among these measures, the ROC Area stands out as a highly informative index. It gauges 8https://stanfordnlp.github.io/stanza/ 9For an extensive discussion of Weka’s features see [17].
Detailed accuracy by class
TP Rate 0,749 0,696 Weighted average 0,723 Detailed accuracy by class
TP Rate 0,734 0,737 Weighted average 0,735
FP Rate 0,263 0,266 0,265 Class 0.0 1.0 Recall 0,734 0,737 0,735
MCC 0,471 0,471 0,471
ROC Area 0,818 0,818 0,818
PRC Area 0,825 0,824 0,824 the model’s overall performance by calculating the density of diferent part-of-speech categories. These area under the curve in a graph plotting the TP included the density of conjunctions, subjunctive (True Positive) rate (y-axis) against the FP (False mode verbs, pronouns of various types, and auxPositive) rate (x-axis), as the classification decision iliary verbs. Additionally, 2 quantitative features threshold changes. An optimal classifier exhibits were considered, namely the count of sentences and ROC area values close to 1, while 0.5 is akin to a the average sentence length measured in tokens. 15 random guess (similar to a Kappa statistic of 0). attributes pertained to the quantity of various types of syntactic constituents identified within the texts.
5. Analysis and discussion of the ing the sophistication, refinement, and variety of results the lexicon employed. Lastly, only 1 surprisal-based metric was selected, which involved the identificaThe intuition behind our approach was that, in tion of outliers (essentially the count of anomalous tracing the linguistic evolution, we should observe a values that significantly deviated from the remainchange/maturation in the syntactic, morphosyntac- ing observations)10. tic and lexical characteristics of the texts examined.
Nevertheless, it was necessary to make further con- 5.2. Features selected for the Spanish siderations and formulate precise hypotheses: if one could expect an increase in lexical sophistication dataset and variety for both the Italian and Spanish corpus Out of the 75 attributes chosen for the Spanish texts, it was however doubtful what phenomena we dataset, 6 were specifically associated with the denwould have witnessed from the syntactic point of sity of various part-of-speech categories. These inview. Even though we have evidence supporting cluded the density of verbs, adverbs, adjectives, and the theory that, in normal acquisition, functionally others. Additionally, 9 attributes were quantitative higher features (such as those involved in comple- features, encompassing measurements such as the mentation, question or relative clauses formation) mean sentence length in letters, the number of toare developed at later stages of acquisition [19], we kens, the standard deviation of sentence length in wanted to assess as many linguistic features as pos- syllables and letters, and the count of tokens with sible in order to map a large number of phenomena more than two syllables, among others. A total of and characteristics of the students’ essays. Let us 29 features were utilized to track morphological and now examine in more detail which attributes were syntactic complexity. These features encompassed selected for classification. the count of various syntactic constituents, as well as specific indicators such as the number of relative clauses, prepositional phrases per clause, and the
10For a list of the selected features for Italian see https: //drive.google.com/file/d/1FCIn1DsawUR2YYg3L685q09 SZ4NaEtnX/view?usp=sharing occurrence of passive structures. Moreover, 24 fea- References tures were dedicated to assessing the sophistication, refinement, and variety of the lexicon employed. 2 additional indices focused on textual cohesion, particularly examining lemma overlaps. Lastly 5 metrics based on surprisal were selected. These metrics included the interquartile range (i.e., the spread of the middle half of the data), the median, the minimum value, the third quartile (i.e., the middle value between the dataset’s median and the highest value), and the standard deviation 11.
performance achieved on the Spanish dataset surpasses that of the Italian dataset. This discrepancy can potentially be attributed to multiple factors. One plausible explanation is the larger number of available attributes for the Spanish language compared to Italian. Additionally, it is noteworthy that the acquisition of a second language (L2) exhibits more significant variation and development within a shorter temporal period compared to a ifrst language (L1). The elevated significance of surprisal-based metrics in relation to the texts from the Spanish L2 corpus can also be rationalized by a higher occurrence of errors and inconsistency/incoherence within the texts during the initial stages of L2 language learning. It is our belief that the outcomes of this study should be interpreted as motivation to further explore the linguistic characteristics of texts, with a particular emphasis on the development of additional indices pertaining to linguistic complexity and information theory. This pursuit aims to gain a more comprehensive understanding of the similarities and diferences in the progression of L1 and L2 languages.
at IUSS Pavia who discussed with us the preliminary results of these experiments. This research has been partially founded by PON Governance 2014-2020: Next Generation UPP Project - CUP D19J22000240006.
11For a list of the selected features for Spanish see https: //drive.google.com/file/d/16ugOhuL6P8eW1v-C7YaKuV hL8RmifRRv/view?usp=sharing tool supporting automatic complexity analysis, of NLP for Building Educational Applicain: Proceedings of the Workshop on Compu- tions, Association for Computational Linguistational Linguistics for Linguistic Complexity tics, Baltimore, Maryland, 2014, pp. 54–60. (CL4LC), The COLING 2016 Organizing Com- URL: https://aclanthology.org/W14-1807. mittee, Osaka, Japan, 2016, pp. 113–119. URL: doi:10.3115/v1/W14-1807.
https://aclanthology.org/W16-4113. [15] P. Qi, Y. Zhang, Y. Zhang, J. Bolton, C. D. [8] J. Hancke, S. Vajjala, D. Meurers, Readability Manning, Stanza: A python natural lanclassification for German using lexical, syn- guage processing toolkit for many human lantactic, and morphological features, in: Pro- guages, in: Proceedings of the 58th Anceedings of COLING 2012, The COLING 2012 nual Meeting of the Association for ComOrganizing Committee, Mumbai, India, 2012, putational Linguistics: System Demonstrapp. 1063–1080. URL: https://aclanthology.org tions, Association for Computational Linguis/C12-1065. tics, Online, 2020, pp. 101–108. URL: https: [9] S. Vajjala, D. Meurers, Assessing the relative //aclanthology.org/2020.acl- demos.14. reading level of sentence pairs for text simplifi- doi:10.18653/v1/2020.acl-demos.14. cation, in: Proceedings of the 14th Conference [16] E. Frank, M. A. Hall, G. Holmes, R. Kirkby, of the European Chapter of the Association B. Pfahringer, I. H. Witten, Weka: A machine for Computational Linguistics, Association for learning workbench for data mining., Springer, Computational Linguistics, Gothenburg, Swe- Berlin, 2005, pp. 1305–1314. URL: http://rese den, 2014, pp. 288–297. URL: https://aclantho archcommons.waikato.ac.nz/handle/10289/1 logy.org/E14-1031. doi:10.3115/v1/E14-1031. 497. [10] N. Okinina, J.-C. Frey, Z. Weiss, CTAP for Ital- [17] D. Merlini, M. Rossini, Text categorization ian: Integrating components for the analysis of with weka: A survey, Machine Learning with Italian into a multilingual linguistic complexity Applications 4 (2021) 100033. URL: https: analysis tool, in: Proceedings of the Twelfth //www.sciencedirect.com/science/article/pii/ Language Resources and Evaluation Confer- S2666827021000141. doi:https://doi.org/10.1 ence, European Language Resources Associa- 016/j.mlwa.2021.100033. tion, Marseille, France, 2020, pp. 7123–7131. [18] L. Breiman, Random forests, Machine learning URL: https://aclanthology.org/2020.lrec-1.8 45 (2001) 5–32.
80. [19] N. Friedmann, A. Belletti, L. Rizzi, Growing [11] N. J. Smith, R. Levy, The efect of word trees: The acquisition of the left periphery, predictability on reading time is logarithmic, Glossa: a journal of general linguistics 6 (2021). Cognition 128 (2013) 302–319. URL: https: URL: https://www.glossa-journal.org/article //www.sciencedirect.com/science/article/pii/ /id/5877/. doi:10.16995/glossa.5877. S0010027713000413. doi:https://doi.org/10.1 016/j.cognition.2013.02.013. [12] N. Rezaii, J. Michaelov, S. Josephy-Hernandez,
B. Ren, D. Hochberg, M. Quimby, B. Dickerson, A computational approach for measuring sentence information via surprisal: theoretical implications in nonfluent primary progressive aphasia, medRxiv (2022). URL: h t t p s : / / w w w . m e d r x i v . o r g / c o n t e n t / e a r l y / 2 0 2 2 / 1 1 / 2 9 / 2 0 2 2 . 1 1 . 2 5 . 2 2 2 8 2 6 30. doi:1 0 . 1 1 0 1 / 2 0 2 2 . 1 1 . 2 5 . 2 2 2 8 2 6 3 0. arXiv:https://www.medrxiv.org/content/early/2022/11/29/2022.11.25.22282630.full.pdf. [13] J. A. Michaelov, B. K. Bergen, Do language models make human-like predictions about the coreferents of italian anaphoric zero pronouns?, 2022. arXiv:2208.14554. [14] G. Kharkwal, S. Muresan, Surprisal as a predictor of essay quality, in: Proceedings of the Ninth Workshop on Innovative Use