Even if the importance of feedback in educational training is widely recognized, delivering timely and effective feedback is not always sustainable, especially in educational settings. To address this challenge, this paper presents the use of an Artificial Agent in both the correction process and online feedback delivery. This approach aims to facilitate the delivery of recursive feedback, thus enhancing the overall learning process. The research design investigates three key phases of the assessment process: test preparation, test evaluation and analysis, and recursive feedback delivery. This paper focuses on the second phase, specifically how the artificial agent and the human agent interact in the correction of student assessments, and details the procedure used to select target texts. The evaluation procedure has been developed and tested. The trial was carried out in the 2023/24 academic year using the open answers submitted by 263 first-year students enrolled in the Master's Degree course in Primary Education at the University of Macerata. The adopted methods allowed for reliable data on approximately 80% of the student submissions. Additionally, the system highlights which texts still require further evaluation and indicates the uncertainty of different blocks, identifying those with more reliable evaluations. Although there is still a long way to go and many developments are possible, the results obtained so far are promising for the adoption of models based on recursive interaction between artificial agent and human agent to make the widespread use of feedback in daily university practice more sustainable.
The importance of feedback in educational training is widely recognized [
The paper presents the strategy to utilize an Artificial Agent (AA) in the correction process and online modes for the delivery of recursive feedback. This approach aims to facilitate the delivery of recursive feedback, thus enhancing the overall learning process.
The work presented in this paper refers to the research carried out in the PRIN AI&F “Artificial
Intelligence and Feedback for Effective Learning” project, in collaboration with teams from the
Universities of Bari and Padua. It is part of research into the development of systems for the
evaluation of open-ended student responses [
The research design investigates three key phases of the assessment process: test preparation, test evaluation and analysis, and recursive feedback delivery. In the first phase, test preparation, the AA collaborates with the human agent (HA) to prepare and organize the test. In the second phase, test evaluation, once the test is completed, it is assessed through an interactive process between the AA and HA. The third phase involves the communication of results to the student, where a recursive interaction between the student and the teacher takes place. After reviewing the teacher’s feedback, the student can provide comments, to which the teacher can respond, fostering an iterative dialogue.
This paper focuses on the second phase, specifically how AA and HA interact in the correction of student assessments. The correction process can be structured as a classification task, which involves grouping student submissions into categories, or as a ranking, where submissions are arranged in order based on a specified parameter. In particular, the procedures used to select Target Texts (TTs) will be detailed. TTs are those texts that the AA uses as reference points to compare against student submissions and identify degrees of similarity. These procedures are directly related to the machine learning process, specifically the steps required to train the system.
It is important to note that within the context of this study the AA is not trained on the whole content of the course in a single instance. Instead, each task was designed to focus on specific content areas or particular aspects of a problem. The distinction between what is considered correct or incorrect is not absolute but depends on the depth of study decided for a specific educational context.
In the initial phase of the experimental process, the goal was to classify student texts. For the
classification of open-ended responses, a variant of the BERT (Bidirectional Encoder Representations
from Transformers) model was used [
The embeddings for student responses and reference texts are generated through this model. Each response is tokenized using BERT’s tokenizer, which splits the text into words or subwords. Dense, high-dimensional numerical vectors, the embeddings, are then computed for each token of the response. The same process is applied to the reference texts (TT), thus allowing the comparison between TTs and students' responses.
Recent studies indicate that embeddings generated by large language models (LLMs) are
extremely effective in capturing complex linguistic structures, with BERT demonstrating superiority
among lightweight models in terms of both accuracy and computational efficiency [
Finally, to assess the similarity between the student’s response and the TT, cosine similarity is used, a measure that quantifies the closeness of two vectors in high-dimensional space [13].
To date, the evaluation procedure has been developed and tested with two trials. In the first trial, the construction of the dataset for the procedures that will be illustrated later was carried out using the answers submitted by 263 first-year students enrolled in the Master’s degree course in Primary Education, attending the course “Foundations of Teaching and Learning” for the 2023/24 academic year. The assessment consisted of three questions:
Q1. Describe what it means to view activity as a coupling between the subject and the environment.
Q2. The following situations can generate conflict: 1) perturbations in the form of resistance, 2) perturbations in the form of gaps. Explain what each situation means and provide an example. Q3. Describe appropriation, specifying the triple integration of in-situation, in-corporation, and in-culturation.
The students’ responses were collected digitally through an online form. Initially, the task was evaluated by a human evaluator (HA1) using a rubric with a grading scale from 1 to 10. After the first tests, it became evident that in some cases the discrepancies between the Artificial Agent (AA) and the human evaluator were due to subjective assessments by HA1. Consequently, a second evaluator (HA2) was asked to review the evaluations.
At a later stage, a third procedure was adopted: all texts were analyzed separately by three evaluators who had shared a common rubric. The cases where there was no agreement between the three evaluations (which exceeded 35%) were then reviewed to reach a consensus. The fact that the non-concordant evaluations exceeded 35% highlighted the objective difficulty in achieving full agreement, even among human evaluators.
The comparison with the results proposed by the AA led to a deep reflection on correction methodologies and the concept of objectivity. As a result of these considerations, the grading scale was adjusted from 10 levels to 4 levels.
Students’ answers were evaluated using different methods, specifically by comparing them with different TT. The goal in exploring different procedures was to identify an effective strategy for constructing the TTs that will tune the AA and therefore will impact the evaluation of student submissions. This paper describes the process that eventually led to the final procedure, which was reached through sequential refinements over 11 tests.
In test 1, students’ answers were divided into two groups: the first group (Block A) served as TT used to train the AA, while the second group (Block B) consisted of the texts that the AA, once trained, had to evaluate independently. The score assigned by the AA was then compared to the score assigned by the human evaluator (HA). The difference between the evaluations of Block B performed by the HA and the AA is defined as Delta, and this value was used as the main parameter for assessing the effectiveness of the procedure.
In test 1, three trials were conducted: in the first trial, Block A consisted of 50 texts from the total of 263, the second trial included 100 texts, and the third trial included 160 texts. The selection of those texts was random. Table 1 presents the results obtained in trial 3 (Block A used 160 TT) for the three questions.
The best results were obtained when using 160 tasks as TT, with 100 texts being evaluated. The analysis of Q3 provided the most acceptable results (71% of cases showed a maximum deviation of one point from the human evaluations on a scale of 10 levels). This difference was attributed to the nature of the three questions. However, correcting 160 students’ answers to achieve this result is not entirely acceptable since it is neither sustainable nor aligned with the desired objective. By further analyzing Q3 to understand where the values were mostly in agreement or disagreement, it was noted that when using a 10-level scale, a difference of 1 or 2 points between evaluators analyzing the same task was very common and difficult to overcome. The first conclusion led to a modification in the number of levels, as previously reported.
From test 1 it emerged that data with higher Similarity scores were more reliable than others (see Table 2). Overall, the average value of Delta is 0.61, whereas a Similarity score higher than 0.975 resulted in an average Delta of 0.26. Similarly, the number of evaluations with Delta 0 (i.e., where the AA’s evaluation matched that of the HA) was 61% in the total sample, rising to 74% for Similarity scores above 0.975. Also, when student responses were shorter (i.e., when the character count was lower than the mean value by an amount equivalent to the standard deviation), the AA often overestimated the score. In fact, the instances where there was a Delta of 3 between the AA and the HA were predominantly linked to these shorter responses. Furthermore, some students included examples in their answers, which were rated positively by HA if relevant, even if the prompt did not explicitly request them. However, for the AA, these examples were not only irrelevant but could skew the evaluation. This is because a response with an example and one without, even if similar in content, would not show the same similarity to TT.
To examine the effect of examples, test 2 used only TTs without examples. This test confirmed the hypothesis, though the difference in Delta was not remarkable (the average Delta decreased from 0.86 to 0.76). Nevertheless, it highlighted the importance of the structure of the TT. The AA does not distinguish between key and minor words that contribute to the overall meaning, particularly in short texts and when the number of available TTs is limited. Redundant words in the TT can affect the similarity measure. This result is evident in Table 3, where providing 20 TTs without examples resulted in a decrease in the average Delta from 0.86 to 0.76.
61 56 56 45 32
Delta 0 [%] 61% 64% 66% 68% 74% 32 27 25 20 11
Delta 1 [%] 32% 31% 29% 30% 26%
Text with examples (N=20) Text without examples (N=20)
After the initial tests, it was decided to use a grading scale with four levels instead of the original ten. Additionally, a new correction of all tasks was carried out by HA, as previously described, with increasing focus on the selection of TT.
In test 3 the TTs were no longer randomly selected as happened in test 1. Instead, concise texts with few redundancies and clearly fitting into one of the four levels were chosen. Sets of 16, 24, and 48 TTs were provided, and each group contained an equal number of texts from levels 1, 2, 3, and 4. The results obtained from 247 instances from the dataset (i.e., 263 total tasks minus the 16 used as TTs in the first trial of test 3) showed an increased agreement between HA and AA agreement compared to previous tests (see Table 4). Additionally, there was no substantial difference between the test using 16 TTs and the one using 48 TTs (see Table 5). In fact, the 16 TTs in the first trial were much more homogeneous and consistent than the 48. Moreover, once the results were ordered and divided into four quartiles according to the similarity score values, it could be observed that where the similarity measure was higher, the results agreed more with the score assigned by HA and thus could be considered substantially more reliable (see Table 6). The findings also indicated greater sustainability, as 16 TTs were used instead of the 160 used in test 1, while achieving similar results, with an average Delta that was not much larger (moving from an average Delta of 0.61 in test 1 with 160 TTs to an average Delta of 0.82 with 16 TTs and an average Delta of 0.60 with 48 TTs) (see Table 4). From this point onward, the focus shifted away from the Similarity threshold (e.g., 0.975) and attention was instead directed to quartile-based analysis.
An additional finding emerged from test 3. As shown in Table 7 the AA better identify the highestquality submissions, particularly those in level 4, where it correctly identifies 72% of the submissions without error. Furthermore, a further attempt using only the 12 level-4 TTs yielded results very similar to those obtained in the trial where 48 TTs were used (12 from level 1, 12 from level 2, 12 from level 3, and 12 from level 4). This further confirms the central role of level-4 TTs. Additionally, as shown in Tables 7 and 8, the system discriminates more easily between submissions from levels 1 and 4.
The previous tests raised the following question: how can the instructor select the 16 significant Target Texts (TTs) without reviewing all the tasks? It became clear that random selection is not viable, meaning the instructor would need to correct a significant number of tasks to extract the 16 most suitable formulations for further analysis. It was considered that, as an initial step, the TT could be based on a passage from the course manual that addresses the topics covered in the question being analyzed.
In test 4, three TTs were used: (A) The entire paragraph related to the test, (B) A selected portion of the paragraph, and (C) Only the passages specifically related to the question. Unlike previous tests, these TTs did not allow for the classification of texts (in previous trials, TTs were provided for all four levels). Results are provided in Table 9, 10 and 11.
It should be noted that by providing the extracted passage from the manual as the TT, the AA could only rank the texts, not classify them, and the ranking was based on similarity. According to the HA assessments, there were 20 responses in level 1, 38 in level 2, 118 in level 3, and 88 in level 4. Therefore, it was decided to instruct the AA to assign the 20 responses with the lowest similarity scores to level 1, the next 38 responses to level 2, the following 118 responses to level 3, and the 88 responses with the highest similarity scores to level 4. This assignment was possible only because the number of tasks assigned to each level by the HA was already known, but this step was crucial to understand whether there was consistency between the similarity score and the HA’s evaluations. In other words, the goal was to determine if the AA’s ranking, based on the similarity between the TT (from the manual) and the students' answers, aligned with the HA’s evaluations.
Tables 9 and 10 present the results of the queries conducted using each TT, as well as the query performed with a TT created by aggregating the three TTs. Table 9 shows the overall data, while Table 10 breaks down the results by quartile. Table 11 shows the average Delta values obtained in the four quartiles.
The first quartile in Table 10 contains the 66 responses with the lowest similarity scores, and so on, where 66 represents approximately one-quarter of the 263 student responses. From Table 9, the best results were obtained with TT C. However, other valuable insights also emerge.
The analysis of Table 9 and 10 provided interesting insights for the continuation of the research. The results obtained by using all three texts together as TTs are similar to those achieved with Text C. This suggests that it is worth considering both the results from using separate TTs and from aggregating multiple texts with similar meanings as if they were a single text.
The second insight relates to the data obtained for each quartile. It is immediately noticeable that the average Delta value across all TTs modalities is higher than the Delta obtained within individual quartiles, as shown by the columns in Table 11. When examining Table 11 vertically, it becomes clear that the Delta value decreases significantly from quartile 1 to quartile 4: the Delta across all TTs modalities in quartile 4 is lower, meaning there is a higher agreement between the evaluations of the HA and the AA. This same trend appears in Table 10, which shows that while there is a total agreement between the AA and HA for 38% across all TTs modalities in quartile 1, this agreement rises to 62% in quartile 4. Additionally, in quartile 4, no TTs modality exhibit a Delta of 3, and the percentage of TTs modalities with a Delta of 2 decreases from 15% to 4%. While the values obtained are not yet fully acceptable for evaluation and feedback purposes, they are valuable for supporting the instructor in their interaction with the AA. As already indicated, and as will be further emphasized later, the selection of the TTs is of fundamental importance for the AA’s evaluation.
In the initial trials, the TTs were derived from an initial manual evaluation by the instructor, which cannot be random. To select, for example, 16 significant TTs, the instructor must analyze far more than 16 students’ answers. Test 4 suggests a way to assist the instructor in identifying the texts to use for constructing the TTs. Of the 15 students’ answers to which the AA assigned the highest similarity scores, 14 of them were placed in the highest level by the HA. In other words, this preliminary automatic analysis by the AA gives the instructor an indication of where to find students’ answers that are most likely to be assigned to the highest level. Shifting the focus to the students’ answers with the lowest similarity scores, among the 15 which were assigned the lowest similarity by the AA, 9 of them were placed in level 1 by the HA.
In summary, test 4 provides some interesting insights. It suggests that a text identified by the instructor and independent of the students' tasks can be used as a TT. This text must be related to the questions, and good results can also be achieved by aggregating multiple texts. The results obtained with such texts are relatively reliable but not yet suitable for definitive evaluations, as the total consistency between the AA and HA is, at best, 54%.
This initial procedure can be used by the instructor to identify tasks belonging to levels 1 and 4 from which to extract the TTs. By reviewing only 20 out of 263 tasks, the instructor can focus on tasks that are highly likely to belong either to level 4 (the ten with the highest similarity) or to level 1 (the ten with the lowest similarity). In this way, the first step of the interaction between HA and AA has been identified, making the entire process more sustainable.
The next step arose from the observation that the AA identifies level-4 tasks more easily. This result is understandable: while level-4 tasks show greater consistency, as they are evaluated for their similarity to a clear model, the lower-level tasks lack coherence with a specific model and instead display consistency with a range of diverse, and not always predictable, possibilities. This led to further reflection on the assignment itself. The prompt required students to address four distinct concepts: 1. what appropriation is, 2. what in-situation is, 3. what in-corporation is, and 4. what inculturation is. If level-4 texts were expected to contain all four concepts, the tasks in levels 1, 2, and 3 could still be coherent in one or more of the four themes; however, inconsistency could manifest in various ways. Therefore, it was decided to provide distinct TTs for each of the individual concepts. A set of texts was constructed, each relating to one of the four required themes (appropriation, insituation, in-corporation, in-culturation), with a fifth category added for examples. For each theme, two sets of TTs were prepared, ranging from 5 to 15 short phrases: the first set consisted of level-4 TTs, and the second set of level-1 TTs (see Table 12).
In this case, the procedure carried out by the AA was no longer a single process. Instead, for each concept, the entire text of each student was compared against a set of TTs. Additionally, the student’s text was compared with each individual TT, and the system reported three similarity values—those obtained from the three TTs with the highest similarity scores. Moreover, the students’ texts were compared with the TT created by combining all 10 TTs related to the same concept.
The results obtained from Test 5 (Table 13), while not showing significant progress compared to previous tests, suggest some promising potential: knowing the correspondence between the student's text and each individual concept allows for providing the student with precise feedback on each specific part of the task.
The subsequent steps took three directions: refining the writing of the TTs; improving the procedure for ranking the students' texts; understanding how to extract feedback from the obtained data to provide to students.
Test 6 aimed to understand whether small modifications, even graphical ones, to the TT would significantly affect the results. For example, the impact of including or omitting capitalized words, numbered or bulleted lists, or presenting the same list in a narrative format was explored. It was observed that a level-4 student text, written in all capital letters, was classified as level 1. Test 2 was repeated after normalizing the students' texts.
The data showed a slight improvement, though minimal. Moreover, this test prompted further investigation into the AI model's indicators to fine-tune it in relation to the desired objectives. In the analysis, the percentage of Delta 0 exceeded 50%, and the average Delta dropped below 0.6. While these results are still not fully satisfactory, they are an improvement over previous test (Table 14).
In Test 7, the focus was more specifically on the difference between ranking and classification (Table 15). In the initial tests, the AA was asked to classify the students' texts, clustering them into the predefined levels based on the TTs, which were also provided for each level. This method did not yield satisfactory results. Subsequently, the decision was made to work on ranking: based on the TTs, the students' texts were ranked, with similarity as the key indicator. In other words, texts with higher similarity scores were more similar to the TT.
While in most previous tests, the focus was on classification—where TTs from four levels were provided and each student text (TS) had to be placed into one level— the goal of test 7 was simply to order the TSs according to similarity, using TTs from a single level. Additionally, a separate analysis was conducted for the four main concepts (plus one additional) required by the task. In previous tests, most TTs included all four concepts. Now, separate queries were performed with TTs representing a single concept. Four queries were conducted, each using seven TTs for one specific concept and level. The overall evaluation was then obtained by aggregating the results for each TS from the separate queries. Logically, level-1 student texts should show a higher relative similarity when compared with level-1 TTs and lower similarity when compared with level-4 TTs. The results are based on 235 student texts, as 28 TTs were excluded from the total of 263. The analysis (Table 15) partially confirms the hypothesis, as the data obtained with the seven level-4 TTs produced a significant ranking. However, the results from the queries using TTs from the other three levels provided much less consistent rankings compared to the HA’s evaluations.
In Test 8, TTs extracted from the manual, similar to those used in Test 4, were reintroduced, but they were modified to better align with the task prompt, removing any unnecessary or accessory information. The results were almost identical to the previous ones, but the small difference allowed for the identification of potential improvements (Table 16). When working with a few TTs and shorter texts, the presence of adverbs, adjectives, lists, examples, and additional details tends to confuse the AA. delta 0 delta 1 delta 2 delta 3 delta 0 delta 1
In Test 9, 28 sentences were provided as TTs, all extracted from the students' texts. Each sentence was numbered and analyzed separately. Since the AA provides the three highest similarity scores and the corresponding three TTs out of the 28 provided, this test allowed for the identification of which TTs were more likely to have higher similarity scores. It was observed that the highest similarity was concentrated on 4 TTs. However, it was challenging to understand why these particular TTs were so frequently matched.
At this point, the task was to engineer the procedure based on the rules extracted from the tests. From the tests, it became clear that short texts—those with a character count below the mean minus the standard deviation (see Test 3)—were difficult to analyze. In our case, these short texts made up less than 10% of the total. In the majority of cases, these texts were classified by the HA at level 1. Therefore, the first rule was to assign all short texts to level 1. For the analysis of the remaining texts, the following considerations were made:
Use ranking logic instead of classification. We can graphically represent the texts as points on a plane, similar to the dispersion of points around a center. The texts in level 4 are closer to the center, while texts in levels 1, 2, and 3 are not only farther from the center but also dispersed in different directions. The similarity measure calculates the distance from the center but not the direction. It was found that the AA’s ability to group the texts was not particularly useful, while ranking them by similarity proved effective.
Divide the results into quartiles and work within each quartile. The most reliable quartiles were the first and fourth. 3. Perform a query for each concept required by the task, using specific TTs for each concept.
In the studied case, the four required concepts were appropriation, in-situation, incorporation, and in-culturation. Additionally, many student texts included examples.
Therefore, four queries should be made, one for each concept. 4. Construct TTs without redundant or conflicting information. These could be sentences with similar meanings but using different language, terms, or graphical structures. A set of about ten sentences per concept should suffice. 5. Prepare the TTs in two phases. In the first phase, student texts are queried using TTs extracted from the manual or the instructor's notes. This identifies the level 4 texts. In the second phase, TTs are extracted from student texts to evaluate the remaining submissions. 6. Use TTs both individually and aggregated. For each concept, a set of around ten TTs is provided. Queries can be run with each individual TT or by aggregating all ten TTs as if they were a single sentence. This approach yields different results—one for each TT and one for the aggregated text—that can either reinforce or weaken the highest similarity result.
Based on these considerations, the following procedure was developed and validated in Tests 10 and 11. The final evaluation is obtained by summing the results from the queries performed for each concept. In this experiment, the same weight was assigned to each concept; however, in other cases, it might be appropriate to assign different weights to each concept. The value for each individual concept is determined as follows: a) Texts in the fourth quartile for each concept are assigned 1 point. Similarly, texts in the first quartile are assigned 0 points. b) Texts in the sixth octile are assigned 1 point if their other similarity values (second, third, and/or aggregated) are consistent. Similarly, texts in the third octile are assigned 0 points if the other evaluations are consistent.
The values assigned using these rules have approximately 90% accuracy.
Using this procedure, each student text is assigned up to four values, one for each concept. Summing these values yields the overall score. Additionally, the instructor receives information on how the student handled each individual concept.
Some texts, after this initial scoring, have blank cells. In the studied case, applying the two rules resulted in a score strongly consistent with the HA’s evaluation for 60% of the sample: 87% had a Delta 0, and 13% had a Delta 1. It should be noted that the Delta 0 between HA2 (non-triangulated evaluation) and HA (triangulated evaluation) was 56%. In other words, in this case, the AA was more reliable than a human evaluation that did not include triangulation.
While the results were satisfactory, having useful outcomes for only 60% of the entire sample was still considered low. Therefore, the results of different queries were used. From these analyses, the following rules emerged, which increased the number of reliable evaluations: c) Texts in the third quartile in the queries with the single TT showing the highest similarity and in the third or fourth quartile in the queries with other individual TTs (that did not show the highest similarity) or with aggregated TTs were assigned 1 point. Similarly, texts in the second quartile in the queries with individual TTs and in the first or second quartile with aggregated texts were assigned 0 points.
Applying these additional rules yielded evaluations for 71% of the sample, although reliability decreased: 79% had a Delta 0, and 14% had a Delta 1.
The next step involved reviewing the texts that had 2 or 3 evaluations instead of 4 after the previous analyses. In these cases, the missing texts fell into the second or third quartile, so an intermediate value between 0 and 1 was assigned. This deduction was only partially correct, as the data analysis, as previously mentioned, indicated that intermediate zones contained less reliable data. In the sample analyzed, 67 texts had 2 or 3 evaluations out of 4. For these 67 texts, the missing cells were filled in with values of 0.25, 0.50, or 0.75 points based on the quartiles of the various similarity measures.
By intervening in this way, evaluations were obtained with approximately 70% accuracy (Delta 0) and 25% Delta 1. This allowed for the correction of 217 texts out of 263, or 85% of the sample, with a Delta 0 of 74% and a Delta 1 of 20%—a performance superior to that of human agents who did not use triangulation.
The adopted methods allowed for reliable data on approximately 80% of the student submissions. Additionally, the system highlights which texts still require further evaluation and indicates the uncertainty of different blocks, identifying those with more reliable evaluations. It should be noted that even in triangulated evaluation, there were "gray areas" where the three evaluators found it difficult to reach a consensus. It is important to emphasize that in the AA's evaluation, over 70% of the texts had a Delta of 0, while the remaining had a Delta of 1.
Although the level of agreement between the AA and HA is higher than that obtained in triangulated evaluation between three evaluators (HA) and the evaluation performed by a single teacher, from a psychological standpoint, especially for the student, such uncertainty can be difficult to accept. Therefore, the evaluation system includes an interactive phase between teacher and student: the system sends the evaluation to the student, who can then respond with comments. This process serves a dual purpose: on the one hand, it allows for the sharing and negotiation of the evaluation; on the other hand, it equips the student with the tools to understand the reasoning behind it. The student is informed about which parts of their work are effective and where there are deficiencies. This process reinforces the recursive feedback system introduced earlier in this paper.
The established procedure is a first step toward automated evaluation and has also highlighted certain limitations. One key limitation, linked to the decision to focus on ranking rather than classification, automatically divides the submissions into four quartiles, each containing the same number of tasks. It is up to the teacher to set the threshold to define the level of acceptability, which may not necessarily align with the average value. In this experiment, threshold values were set to define the four levels: from the automatic calculation, each submission received a score based on the sum of the values obtained for each concept, and the three thresholds (between levels 1 and 2, 2 and 3, and 3 and 4) allowed for the identification of tasks in each level. The teacher independently determines these three thresholds after the AA has ranked the 263 submissions.
Furthermore, continued experimentation is needed to determine how this procedure may be influenced by different types of tasks and academic disciplines. In this context, after the first experiment described here, the procedure was applied to another pedagogical task in a real-world educational setting, and although the reliability obtained was not identical, it was comparable to the initial results. These trials were part of a preliminary investigation into the feasibility of the system, laying the groundwork for future pilot studies in collaboration with schools and other educational institutions. Future research will involve larger-scale pilot studies across various educational institutions to validate these findings and ensure the system’s scalability and effectiveness in different academic contexts.
Future iterations of this system could incorporate topic extraction and clusterization techniques, such as BERTopic [14] and the exploration of other Large Language Models (LLMs) for assessing open-ended responses.
The development of this research will be closely linked to the evolution of AI models. Every day, new and more advanced projects are being introduced by various companies. By the time this contribution is published, the previous model could already be outdated. Looking ahead, generative AI models could assist teachers in the early stages of the process, including the preparation of the task itself. Although there is still a long way to go and many developments are possible, the results obtained so far are promising for the adoption of models based on recursive interaction between AA and HA to make the widespread use of feedback in daily university practice more sustainable.
This research activity is part of the PRIN 2022 project no. 2022ZMYYTH, titled "AI&F – Artificial Intelligence & Feedback for Effective Learning" – CUP D53D23013110006, funded under the PRIN 2022 Call (Funded by the European Union – Next Generation EU – PNRR Mission 4, Component 2, Investment 1.1).
[13] J. Wang, Y. Dong, "Measurement of text similarity: a survey." Information 11.9 (2020): 421. doi: 10.3390/info11090421 [14] M. Grootendorst, BERTopic: Neural topic modeling with a class-based TF-IDF procedure, arXiv preprint arXiv:2203.05794 (2022). Doi: 10.48550/arXiv.2203.05794