This paper reports on a study aimed at comparing AI vs. human performance in detecting and categorising errors in L2 Italian texts. Four LLMs were considered: ChatGPT, Copilot, Gemini and Llama3. Two groups of human annotators were involved: L1 and L2 speakers of Italian. A gold standard set of annotations was developed. A fine-grained annotation scheme was adopted, to reflect the specific traits of Italian morphosyntax, with related potential learner errors. Overall, we found that human annotation outperforms AI, with some degree of variation with respect to specific error types. We interpret this as a possible effect of the over-reliance on English as main language used in NLP tasks. We, thus, support a more widespread consideration of different languages.
Identifying errors in texts written by second language
(L2) learners is a relevant task in several research areas,
which can also have practical applications in a variety of
fields. Error analysis is a traditional approach adopted in
second language acquisition research for decades
The interest of Natural Language Processing (NLP) in grammatical error correction (GEC) and grammatical error detection (GED) relies on the creation of systems used in Intelligent Computer-Assisted Language Learning (ICALL), Automated Essay Scoring (AES) or Automatic Writing Evaluation (AWE) contexts. ICALL systems integrate NLP techniques into CALL platforms, providing learners with flexible and dynamic interactions in their learning process. AES systems automatically grade written texts with machine learning techniques, as well as AWE systems, which also provide learners with feedback.
Identifying and annotating errors in the performance of L2 learners, while beneficial for both pedagogical and research purposes, presents considerable challenges. This process is typically conducted manually in the case of learner corpora due to the inherent nature of errors as latent phenomena. The manual identification of learners’ errors requires a substantial degree of subjective judgment by human annotators (Dobrić 2023), as well as a considerable investment in terms of time.
The present study aims to contribute to the evaluation of the performance of Large Language Models (LLMs) in the task of automatic grammatical error detection (GED) in written texts produced by L2 learners. In particular: 1.
it evaluates the behaviour of different LLMs in
relation to an error detection task in written
texts produced by L2 learners of Italian, a
language other than English, in line with
recent studies criticising the over-reliance on
English in NLP research
Research on automatic error detection in L2 written
texts, mainly adopting machine learning approaches,
has significantly developed in recent years
The CoNLL-2014 Shared Task on Grammatical Error
Correction
A recent study by Kruijsbergen et al. (2024) focused on L1 and L2 Dutch and explored the capabilities of LLMs in written error detection, with both a fine-tuning and a zero-shot approach through prompting a generative language model (GPT-3.5). Results highlight that the fine-tuning approach largely outperforms zeroshotting, both for L1 and L2.
To evaluate AI performance in automatic GED on L2 written texts, we designed our study based on the following stages: selection of the text sample; error type identification; definition of the gold standard (henceforth, GS); evaluation of LLMs’ annotations; comparison between LLMs and human performance.
We used authentic L2 data derived from a learner corpus
of Italian, the CELI corpus
Table 1. Description of the sentences’ sample. 67
Contrary to previous study
Type AS
PoS article
The outputs of the four LLMs were compared to a benchmark (GS) obtained from the annotation of three researchers. Three Italian trained linguists (i.e., the three authors of this paper) manually annotated the sample texts. The three researchers annotated only the error types described above. Initially there was a substantial agreement between the three linguists (k = 0.61). The three annotators disagreed mostly on the PA error (k = 0.39). Any inter-annotator disagreements were resolved through negotiation until a partial agreement (i.e., two annotators out of three) was reached. The agreement turned out to be improved (k = 0.81). Then, all the remaining disagreements (i.e., the cases that reach a partial agreement) were resolved reaching a perfect annotator agreement prioritising the two annotators’ decision over the third one (k = 1). In the GS, 47 grammatical errors were identified with an average of 4 errors per text, while no errors were found in 32 sentences. On average, each sentence contained 2 errors. ChatGPT-4o (July 2024 version), Copilot, Gemini and Llama3 were evaluated. Several steps were followed to arrive at the final prompt, which can be found in Appendix A. We started giving the prompt in Italian and presenting all the texts together. However, the four LLMs, which were not pre-trained, were able to find a small number of errors. We, then, proposed the prompt in Italian again, repeating the instructions for each text. In this case, the LLMs identified types of errors that were not required. Following the recommendations from Kruijsbergen et al. (2024) on the prompt’s language, the entire prompt was then given in English. The performance improved as a greater number of errors were identified, but still types of errors that were not required. Therefore, we gave a more detailed prompt in English following the recommendation of Coyne et al. (2023). Definitions of the four Italian PoS were provided. Further, we listed the eight error types with descriptions and examples. The texts were presented in numbered sentences. LLMs were instructed to classify each detected error and were informed that there could be more than one error in a sentence as well as no errors at all. The entire prompt was repeated for each text. This last version of the prompt was used for this study. Subsequently, we calculated the inter-annotator agreement between the four LLMs, which resulted to be weak (k = 0.21). 3.3.2. Human annotator groups
LLMs’ performance was also compared to two human groups. Twenty-two L1 (age range: 19-50) and Twenty-seven L2 speakers (age range: 22-40) of Italian took part in the annotation task. They were undergraduate and postgraduate students in humanities and social studies. They were asked to annotate only the error types described above, with definition and examples provided for each type of error. They were also asked to report the incorrect form and to provide the correct one. Then, we calculated the inter-annotator agreement between the raters of the two groups. L1 speakers reached a good agreement (k = 0.52), while the agreement between L2 speakers was poor (k = 0.33).
Four measures were used to compare the performance
of LLMs and human annotators in detecting errors:
Accuracy, Precision (P), Recall (R) and F-score (Fß).
Accuracy was calculated by dividing the number of
correctly identified errors by the total number of
annotated errors. To be consistent with previous works
in GED
Gemini outperformed the other three systems, demonstrating the highest accuracy (65,52%). In contrast, Llama 3 turned out to be less accurate in comparison to the others (51,72%). ChatGPT and Copilot behaved similarly in terms of accuracy (57,47%). LLMs were less accurate than human groups in detecting errors, as L1 and L2 speakers reached much higher values of accuracy (89,66% and 78,16% respectively).
When looking at AI performance, Copilot and Llama3 showed worse P than ChatGPT and Gemini, indicating that they had low ability in detecting true error instances. Conversely, Gemini and Copilot were able to detect a higher number of errors compared to ChatGPT and Llama3. ChatGPT made the best predictions, while Gemini had better R. Human groups outperformed AI systems for R, P, and F-score (Table 2). L1 speakers were able to detect almost all errors and to make correct predictions. On the contrary, L2 speakers had better P and worse R, suggesting they had lowest number of FP but a reduced ability to detect TP.
Figure 1 shows the performance of each group in terms of P and R.
P (%) 65.22 34.78 58.69 45.65 93.02 93.55
R (%) 58.82 69.56 71.05 55.26 89.96 63.04
Fß 63.83 66.75 60.81 47.29 92.39 85.29
To examine thoroughly the performance of the LLMs in GED, we calculated R, P and F-score metrics for each of the eight error types (Table 3).
R(%)
Fß (%) Error type ChatGPT Copilot Gemini Llama3
AO AS AA NE VE PO PA PS AO AS AA NE VE PO PA PS AO AS AA NE VE PO PA PS AO AS AA NE VE PO PA PS
P(%) 50 20 / 20 L2 speakers
AO
Copilot, Gemini, and Llama3 failed to detect various error types exhibiting a high number of FP without detecting true instances. Copilot showed a fair prediction of VE and PS errors. Gemini had better R and P in detecting and correctly predicting AO and VE errors. However, it performed worse on PS errors in terms of both P and R. Llama3 was able to predict AS, VE, and PS errors but showing low values of R. ChatGPT turned out to be the best in predicting all error types, except for the AA error. ChatGPT showed high values of P in the prediction of AO, PA, and PO errors and showed low values of P and R for PS errors.
Human groups performed better than LLMs in detecting each error type. L1 speakers exhibited high values of R and P in detecting all error types but performed less well in making correct predictions on PS errors. L2 speakers demonstrated better R and P in detecting AO and AS errors. Conversely, they were unable to identify all AA errors. Furthermore, they showed a reduced ability in detecting all PO errors and in predicting them correctly.
The main aim of our paper was to investigate whether AI can be a valid support for second language acquisition research, in learner error detection, with specific reference to a language other than English, i.e., Italian. Our study compared the performance of four LLMs among them and also compared with L1 and L2 annotators. A GS, produced by the annotations of three trained linguists, was adopted as benchmark. Given the richness of Italian morphosyntax and the variety of possible morphosyntactic errors that L2 Italian learners may produce, contrary to the few other studies on Italian, this study considered three different error types for two of the parts of speech listed in Table 1, i.e. article and preposition. This methodological novelty can potentially lead to much more fine-grained results, while counterbalancing, like in our case, the low number of annotated texts.
The general finding about human annotators
performing better than LLMs, both in terms of overall
error detection and in terms of error type detection, is
particularly significant if we consider the structural
differences between English and other languages.
Italian, like many other languages, is characterised by
rich morphosyntatic traits, which inevitably have a
considerable impact on the computational processing of
L1 and L2 texts. Our findings may thus be a reflection of
the well-known language bias in NLP, linked to the
dominance of English, which then leads to a number of
scientific but also social inequalities
This study was conducted in the context of CARLA – Corpus Approaches to Research on Language, a research group affiliated with the Department of Italian language, literature and arts in the world (University for Foreigners of Perugia, Italy).
Appendix A The Prompt In this task, we present a text in Italian, produced by a learner of L2 Italian at B1 proficiency level. The text is numbered and divided into numbered sentences. For each sentence, you will have to identify specific errors, if any.
The errors considered in this task involve articles (in Italian "il, lo, la, i, gli, le, un, uno, una"), prepositions (in Italian "di, a, da, in, con, su, per, tra, fra", in their simple forms or associated with articles "del, dalla, negli, etc.", nouns, and verbs).
For each error, you will have to indicate the type, which you can choose from the following list: 1a: Article addition: the learner has added an article where it was not necessary (e.g. "Ho fatto la fatica a salire le scale": "la" should not have been used); 1b: Article omission: the learner did not use the article even though it was necessary (e.g. "Maria ha fatto compromesso con il suo capo": "un" should have been used before "compromesso"); 1c: Article choice: the learner used the wrong article (e.g. "In montagna ci sono i alberi sempreverdi": "i" is wrong, the correct article is "gli"); 2: Verb ending: the verb ending is incorrect (e.g. "Ieri Luca andavo al mare": "andavo" has the wrong ending "o", the correct one is "a" --> "andava"); 3: Noun ending: the noun ending is incorrect (e.g. "Ho comprato tre mela gialle": "mela" has the wrong ending "a", the correct one is "e" --> "mele"); 4a: Preposition addition: the learner added a preposition where it was not necessary (e.g. "Ho comprato a un libro": "a" should not have been used); 4b: Preposition omission: the learner did not use a preposition even though it was necessary (e.g. "Anna Ë andata casa": the preposition "a" is missing before "casa"); 4c: Preposition choice: the learner used the wrong preposition (e.g. "Questo Ë il libro a mio professore": "a" is wrong, the right preposition was "del").
It is possible that there is more than one error in a sentence, but also that there are no errors at all. If you find no errors, do not indicate anything and move on to the next sentence.
Here is the text with the numbered sentences.