Nowadays, there is growing attention on enhancing the quality of teaching, learning and assessment processes. As a recent EU Report underlines, the assessment and feedback area remains a problematic issue regarding educational professionals' training and adopting new practices. In fact, traditional summative assessment practices are predominantly used in European countries, against the recommendations of the Bologna Process guidelines that promote the implementation of alternative assessment practices that seem crucial in order to engage and provide lifelong learning skills for students, also with the use of technology. Looking at the literature, a series of sustainability problems arise when these requests meet real-world teaching, particularly when academic instructors face the assessment of extensive classes. With the fast advancement in Large Language Models (LLMs) and their increasing availability, afordability and capability, part of the solution to these problems might be at hand. In fact, LLMs can process large amounts of text, summarise and give feedback about it following predetermined criteria. The insights of that analysis can be used both for giving feedback to the student and helping the instructor assess the text. With the proper pedagogical and technological framework, LLMs can disengage instructors from some of the time-related sustainability issues and so from the only choice of the multiple-choice test and similar. For this reason, as a first step, we are proposing a starting point for such a framework to a panel of experts following the Delphi methodology and reporting the results.
providing feedback based on specified criteria [
As a UNICEF definition afirmed: “AI refers to machine-based systems that can, given a set of human-defined objectives, make predictions, recommendations, or decisions that influence real or virtual environments. AI systems interact with us and act on our environment, either directly or indirectly. Often, they appear to operate autonomously and can adapt their behaviour by learning about the context" [6, p.16].
To connect this topic to the education field, it is possible to afirm that, in terms of
technological advancements, theoretical contributions and impact on education, the field of Artificial
Intelligence in Education (AIED) has seen success over the past 25 years [
Higher education aims to provide meaningful, relevant courses, where graduates can learn to work and live in an increasingly digital society [14]. In our contemporary education system, our students need to be supported in becoming efective lifelong learners, who must be prepared for the assessment tasks they will encounter in their lives, but they also need to become lifelong assessors, possessing assessment skills acquired through continuous work. It could be possible through the implementation of Assessment for Learning intended as an approach that emphasises the assessment process as an "essential moment of the educational experience, characterised by situations in which learners are enabled to analyse and understand the processes in which they are involved and can thus participate in decisions about their learning goals by becoming increasingly aware of their progress" [15, 16, p.56]
Learners as lifelong assessors must be able to: • Estimate the possession of criteria for evaluating situations or carrying out an assignment; • Seek and understand contextual feedback to construct new knowledge; • Interpret and use feedback to achieve daily goals and challenges [17, 16, p.73].
Students need to gain, during their learning process, the assessment expertise, so the competence required for students to efectively understand assessment criteria and to be able to use the feedback received to close the gap and improve their own learning [19]. They will be supported in this process by teachers who, through continuous assessment and making judgements about the learning products of students, will develop more efective standards of judgement to define the expected competence of students themselves [ 18, 16, p.76]. But how to really support students’ development of assessment expertise? As Sadler pointed out [19], it is necessary to involve students in direct and authentic assessment experiences, supporting them in the acquisition of the concept of quality, and training them in order to make complex judgements according to a multiplicity of criteria [19]. What are the principles to apply in order to create sustainable assessment contexts and then scafold authentic assessment experiences? Boud [17] drew up nine useful principles for reflecting on and designing sustainable assessment and feedback practices. For the author, it is indeed important that there is a timely sharing of clear assessment criteria for students; it is also crucial that students are seen as individuals who can achieve success and, in terms of assessment processes, these must be useful in making students confident in their success and in this sense it seems useful to consider separating feedback processes from the awarding of grades. The focus on learning during the assessment process must take priority over the focus on performance: it seems important therefore to support the development of self-assessment competencies and encourage the use of reflective peer assessment practices. One of the last fundamental aspects is related to the completion of the feedback loop as a tool for reviewing student work and finally, the importance of introducing a review process of assessment practices with the implementation of formative assessment processes is emphasised.
In terms of assessment and feedback in connection with AI systems, Swiecki and colleagues afirmed that “AI-based techniques have been developed to fully or partially automate parts of the traditional assessment practice. AI can generate assessment tasks, find appropriate peers to grade work, and automatically score student work. These techniques ofload tasks from humans to AI and help to make assessment practices more feasible to maintain” [20, p.2].
The power of AI related to the assessment and feedback processes is connected to the fact that, while traditional assessment practices could provide a partial overview about the students’ performance, several AI techniques thanks to their characteristics can promote a wider vision of learning process and progress. In relation to the topic of authenticity and sustainability of assessment, AI systems can help to collect, represent, and assess data in a complex way: authentic assessment processes can be very articulated in terms of task and general design, so AI can be helpful for academics to monitor learning process towards an assessment of student progress [21, 20]. In fact, authentic assessment requires students to “use the same competencies, or combinations of knowledge, skills, and attitudes, that they need to apply in the criterion situation in professional life” [22, p.69]. Authenticity has been recognised as a fundamental element of assessment design that encourages learning. Authentic assessment tries to reproduce the activities and performance criteria often encountered in the workplace and has been shown to have a favourable influence on student learning, autonomy, motivation, self-regulation, and metacognition; qualities that are significantly associated with employability [ 23]. Again, international authors even suggest that the authenticity of the assessment tasks is a need for reaching the expert level of problemsolving. Likewise, strengthening the authenticity of an assessment has the potential to have an encouraging efect on student learning and motivation [24, 25, 26, 22].
Finally, UNESCO [26], which has been in the last years amongst the most influential and active institutions that reflect on the implications of AI in society, provided the following guidelines for AI in assessment.
1. Testing and implementing artificial intelligence technologies is crucial for supporting the assessment of various dimensions of competencies and outcomes. 2. Caution is essential when adopting automated assessment with responses to rule-based closed questions. 3. Employing formative assessment leveraged by artificial intelligence as an integrated function of Learning Management Systems (LMS) is key to analysing student learning data with increased accuracy and eficiency and reducing human biases. 4. Progressive assessments based on artificial intelligence are imperative to provide regular updates to teachers, students, and parents. 5. Examining and evaluating the use of facial recognition and other artificial intelligence for user authentication and monitoring in remote online assessments is paramount.
Over the past few years, Large Language Models (LLMs) have become increasingly prevalent in society and educational settings. These AI-powered models are capable of generating, analysing, and summarising text, as well as engaging in dialogic interactions with humans [27]. One of the most well-known examples of LLMs is OpenAI’s ChatGPT, which is based on GPT 3.5 and GPT 4 architectures. Other notable LLMs include Antropic’s Claude (1 and 2), Bing Chat (another GPT4-based model), and Google Bard. While these models are extremely powerful, there are concerns about data privacy and results consistency [28]. However, there are other options available. With the release of open-source and open-access models such as Meta’s LLAMA and LLAMA 2, as well as TII’s Falcon, and the growth of platforms like HuggingFace, which acts as a repository and framework, there are many possibilities for local LLMs with great capabilities. These models can be customised, fine-tuned, or even trained specifically for one’s use case, allowing for greater flexibility and control [29].
To better understand the possibilities related to the use of these models, Tamkin et al. [
This can be used to gauge student sentiment in feedback, assignments or discussion forums. • Classify: LLMs can classify text into predefined categories. This can be used for assisted grading or categorising student feedback. • Detecting plagiarism: By comparing the similarity between diferent pieces of text, LLMs can help detect potential cases of plagiarism both between students and between students and the source material. • Measure Semantic Similarity: LLMs can measure the semantic similarity between two pieces of text. This can be used to match student queries with relevant answers or resources and help the teacher in the assessment of the student’s work. • Generate Feedback: Based on the assessment of a student’s work LLMs can generate personalised feedback. It would work even better if the LLM would have some teacher’s notes on the assignment to work with. • Assess Knowledge: LLMs can be used to assess a student’s understanding of a topic based on their written submissions, especially if properly trained on correct assignments and having an assessment rubric to refer to.
LLMs are able to analyse massive amounts of text, aggregate it, and then ofer feedback based
on previously established standards [
In specific relation to assessment and feedback practice, the correlation with LLM can be summarised in four diferent points: 1. Automatisation: Assessment and Feedback can be totally or partially automated, although, at the moment, only supervised, human-mediated, assessment is advised [20, 30], there are some cases in which even a complete automation worked very well [29]. 2. Sustainability: relative to the time variable, these models make scalable types of assessment that previously were not. This allows the teacher to always apply the most suitable method of assessment for verifying learning objectives [31, 20]. 3. Objectivity: if trained correctly, LLMs should not have bias, they tend to be more objective and consistent than a human being and adhere to the established criteria. 4. AI in the loop: LLM, teachers and students can be part of the same process in which the
IA is assigned only those tasks in which it is super-human [32, 33].
In light of the evidence already produced by the international literature on the topic of AI and education, this research study aims to create and validate a model for the use of AI in Educational Assessment in Higher Education. The work is based on one main research question: • Could university teachers use AI tools to adopt approaches that support more efective, sustainable and authentic assessment?
The designed model takes into account the existing literature connected to the topic of Assessment for Learning, Authentic Assessment and Sustainable Assessment [15, 22, 17]. Starting from these literature pieces of evidence, we are working on the development of a model useful to adopt AI in the assessment processes in the Higher Education context. The model considers the role that AI plays in the assessment and feedback practices connected to academics and students in the virtuous cycle of the learning spiral. The model itself will be assessed following four diferent levels proposed by Kaptelinin and colleagues [ 34] through the Activity Theory checklists for the design, the evaluation and the use of technology: • Design: we will introduce this checklist in order to evaluate the design process itself. • Evaluation Phase 1: in the first phase of the evaluation process, thanks to the introduction of the Delphi study approach and then, thanks to the collaboration of the experts, we will use the checklist connected to the Activity Theory to assess the structure of the model itself and collect prompts and suggestions. • Evaluation Phase 2: in the second phase of the evaluation process, we aim to introduce an evaluation of how we are going to propose the use of the model itself, again based on the validated checklists. • Use: in this last phase, we will introduce a specific checklist to evaluate the model and its related impacts on the teaching, learning and assessment processes.
Every checklist is developed following four diferent areas: 1. Means and ends: the extent to which the technology facilitates and constrains the attainment of users’ goals and the impact of the technology on provoking or resolving conflicts between diferent goals. 2. Social and physical aspects of the environment: integration of target technology with requirements, tools, resources, and social rules of the environment. 3. Learning, cognition, and articulation: internal versus external components of activity and support of their mutual transformations with target technology. 4. Development: developmental transformation of the foregoing components as a whole [34].
• AI-Mediated Summative Assessment: level focused on assessment processes connected to Technology Enhanced Assessment practices, so the power of AI in connection to the possibility of introducing assessment and feedback timely, customised and informed by AI data [30]. • AI-Mediated Formative Assessment: level focused on the power of the AI implementation in assessment and feedback in order to monitor the whole learning process and to guide formative design actions and students’ self and peer assessment processes [35, 36].
To validate the proposed model, we planned to introduce in our research the Delphi Study technique, intended as “a scientific method to organise and manage structured group communication processes with the aim of generating insights on either current or prospective challenges [...] the Delphi technique builds on the anonymity of participating experts who are invited to assess and comment on diferent statements or questions related to a specific research topic” [37, p.2].
In a Delphi survey, the opinions, generated by the individuated group of experts across the multiple discussion rounds organised on a specific topic, are collected. The multi-round structure can be introduced sequentially, or immediately thanks to specific software. The structured group communication process should create a convergence or a divergence of opinions, producing a more dynamic and accurate collection of data in comparison to traditional opinion-polling techniques. This method allows researchers to focus on the sharing process, reducing risks related to group dynamics that may emerge during in-person collaborative processes [38, 39, 40, 41, 42, 37].
To sum up, the Delphi process can be divided in the following diferent steps: 1. Defining and recruiting experts: experts could be professionals with specific knowledge and relevant experience in a particular discipline and research area. The panel size is calibrated depending on the study’s purpose. 2. Developing Delphi questionnaire: a Delphi questionnaire can be structured from primary data (interviews) or literature analysis to enhance validity. Experts can be involved with Paper-based or e-Delphi. 3. Round 1: this phase can be qualitative, really powerful to generate ideas (e.g. open-ended questions) or quantitative (e.g. rating scale). To certify the rigour of each round, Kilroy and Driscoll (2006) suggest that the response rate should not fall below 70%. 4. Analysis and design of Round 2: phase characterised by the results analysis and in connection with non-consensus issues, another questionnaire containing non-consensus issues and the Round 1 results are sent out to the experts (Round 2). The feedback sustains the experts’ comparison of their initial opinions with the group result. Additional rounds are organised until the achievement of an acceptable level of consensus free of issues or controversies [43].
Starting from JISC’s [30] and UNESCO’s [26] guidelines, we developed our model called AIMAAS (AI-Mediated Assessment Academics and Students), composed of two diferent levels of application and interpretation, with a focus on the implementation of AI in Technology Enhanced Assessment and Formative Assessment processes. The model revolves around three focal points, i.e. with respect to the cyclic and balanced intersection of AI, teachers and students, following Vygotskij and Leont’ev’s model of artefact mediation [44]. AI Mediated Summative Assessment level of implementation describes the three elements (Academics, AI and Students) and the connection between them as follows.
This first level (Fig. 1) focuses on general, usual, assessment. In this kind of process, AI can give the assessment a formative twist and significance, adding to the interaction with the student, but keeping the whole process sustainable for the academic. In this approach, most of formative exchanges would be between AI and Student, and mostly one-way (i.e. AI to Student), with early feedback on the product, and a final report on the assessment and future actions being the most important. It is important to notice that the AI’s final feedback to the student must be, at this stage, moderated by the academic. This approach is supported by the capability of AI in connection to the possibility of introducing assessment and feedback timely, customised and informed by AI data [31].
Elements of this process are: • AI: -Constructive role: AI can help teachers with the construction and delivery of early feedback and assessment. In connection, academics can define and share rubrics and assessment criteria to scafold the assessment process. -Feedback mechanism: academics play a key role as actors who can give reinforcement feedback to the AI system itself, always to improve jointly developed evaluation processes. -Evaluation and Reporting: the relationship between AI and students is characterised by the exchange of the students’ products to be assessed and then AI as the producer of specific reports that contain suggestions for learning improvement. AI with the role of tutor that shares early and timely feedback supported by the academics’ expertise. • Academics: -Experts provision: academics as experts able to build and share tailored information to sustain AI actions. -Feedback management: academics as professionals who are able to manage timely, personalised and AI-informed feedback. • Students (students’ product): -Product creation: students as crucial actors are able to build specific products to be assessed thanks to the collaboration between academics and AI. -Guidance role: students as important elements to guide the AI Mediated Assessment processes with focused feedback.
AI Mediated Formative Assessment level of implementation describes the three elements and the connection between them as follows:
The second level (Fig. 2) focuses on proper formative assessment processes. In this approach the lecturer has designed the teaching to follow this approach, and the assessment is continuous, not relegated to the final stages of the course. Most interactions are bi-directional and occur between AI and Students, and Students and the lecturer. In this model, AI is directed by the lecturer and impersonates various roles, always in the form of collaboration with the students as a mentor, a tutor, or a peer. At the same time, AI capabilities to monitor the whole learning process and to inform formative design actions will be employed to support the academic [36, 37].
Elements of this process are: • AI: -Constructive role: the relationship between AI and academics is set up with a dynamic process of exchange in terms of expertise, resources and tasks. -Feedback mechanism: the data produced by AI can be fundamental for monitoring and redesigning academics’ teaching and assessment practice. -Evaluation and Reporting: AI can play the role of student and the students can act as teachers in order to support and give prompts to AI that, at the same time, can play the role of tutor, mentor or group member (peer teaching relationship with students). • Academics: -Expertise Provision: in connection with AI, academics define roles, rules and criteria for AI itself. -Feedback management: in terms of relationships with students, academics pay attention to the assessment of the whole learning process, giving timely, personalised and AI-informed feedback. • Students (students’ product): -Constructive role: students can activate critical thinking actions on AI answers, in order to stimulate deep and complex reflective processes, through specific students’ inquiry to produce efective insights for AI. At the same time, students can discuss, share results produced by academics and AI, and also activate peer learning and assessment processes. -Guidance role: students can generate feedback on AI Mediated Assessment itself.
As previously mentioned, the model will be assessed and validated using the Delphi method [
Starting from the opportunities connected to the use of AI in education from both perspectives of students and teachers [27], it is important to understand how to better include these new opportunities to enhance teaching, learning and assessment processes in Higher Education contexts. For this purpose, our research is contextualised in an academic environment that has to cope with constant renewal in terms of approaches and strategies to deal with a major change at design, organisational and conceptual level. In connection with the topic of assessment and feedback from a perspective of assessment for learning, sustainability and authenticity [15, 17, 22], it is important to reflect and design specific formative and practical actions to sustain students and teachers in the implementation of AI systems as powerful agents to support the progress of the educational system. In terms of future research perspective, the designed actions include, after the validation of the AI-MAAS model through the Delphi study, experimentation using the model with academics, with a following phase which will comprehend the impact analysis and the assessment of the eficacy. [11] W. Holmes, M. Bialik, and C. Fadel, "Artificial intelligence in education," Globethics Publications, 2023. [12] W. Holmes and I. Tuomi, "State of the art and practice in AI in education," European Journal of Education, vol. 57, no. 4, pp. 542-570, 2022. [13] W. Holmes, M. Bialik, and C. Fadel, "Artificial intelligence in Education: Promises and implications for teaching & learning," The Center for Curriculum Redesign, 2019. [14] J. Nieminen, M. Bearman, and R. Ajjawi, "Designing the digital in authentic assessment: is it fit for purpose?," Assessment & Evaluation in Higher Education, vol. 48, no. 4, pp. 529-543, 2023. [15] K. Sambell, L. McDowell, and C. Montgomery, "Assessment for learning in higher education," Routledge, 2013. [16] V. Grion and A. Serbati, "Valutazione sostenibile e feedback nei contesti universitari.
Prospettive emergenti, ricerche e pratiche," PensaMultimedia, 2019. [17] D. Boud, "Sustainable assessment: rethinking assessment for the learning society," Studies in continuing education, vol. 22, no. 2, pp. 151-167, 2000. [18] D. J. Nicol and D. Macfarlane-Dick, "Formative assessment and self-regulated learning: A model and seven principles of good feedback practice," Studies in higher education, vol. 31, no. 2, pp. 199-218, 2006. [19] D. R. Sadler, "Formative assessment: Revisiting the territory," Assessment in Education, vol. 5, no. 1, pp. 77-84, 1989. [20] Z. Swiecki, H. Khosravi, G. Chen, R. Martinez-Maldonado, J. M. Lodge, S. Milligan, N.
Selwyn, and D. Gašević, "Assessment in the age of artificial intelligence," Computers and Education: Artificial Intelligence, vol. 3, 100075, 2022. [21] V. Murphy, J. Fox, S. Freeman, and N. Hughes, "“Keeping it Real”: A review of the benefits, challenges and steps towards implementing authentic assessment," All Ireland Journal of Higher Education, vol. 9, no. 3, 2017. [22] J. T. Gulikers, T. J. Bastiaens, and P. A. Kirschner, "A five-dimensional framework for authentic assessment," Educational technology research and development, vol. 52, no. 3, pp. 67-86, 2004. [23] V. Villarroel, S. Bloxham, D. Bruna, C. Bruna, and C. Herrera-Seda, "Authentic assessment: creating a blueprint for course design," Assessment & Evaluation in Higher Education, vol. 43, no. 5, pp. 840-854, 2018. [24] J. Herrington and A. Herrington, "Authentic assessment and multimedia: How university students respond to a model of authentic assessment," Higher Educational Research & Development, vol. 77, no. 3, pp. 305-322, 1998. [25] K. Sambell, L. McDowell, and S. Brown, "But is it fair?: An exploratory study of student perceptions of the consequential validity of assessments," Studies in Educational Evaluation, vol. 2.3, no. 4, pp. 349-371, 1997. [26] S. Gielen, ¥. Dochy, and S. Dierick, "The influence of assessment on learning," in M. Segers, F. Dochy, & E. Cascallar (Eds.), Optimising new modes of assessment: In search of quality and standards, pp. 37-54, Dordrecht, The Netherlands: Kluwer Academic Publishers, 2003. [27] F. Miao, W. Holmes, R. Huang, and H. Zhang, "AI and education: A guidance for policymakers," UNESCO Publishing, 2021. [Online]. Available: https://doi.org/10.54675/PCSP7350 [28] E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G.
Groh, S. Günnemann, E. Hüllermeier, S. Krusche, G. Kutyniok, T. Michaeli, C. Nerdel, J. Pfefer, O. Poquet, M. Sailer, A. Schmidt, T. Seidel, M. Stadler, J. Weller, J. Kuhn, and G. Kasneci, "ChatGPT for good? On opportunities and challenges of large language models for education," Learning and Individual Diferences, vol. 103, 102274, ISSN 1041-6080, 2023. [Online]. Available: https://doi.org/10.1016/j.lindif.2023.102274 [29] L. Chen, M. Zaharia, and J. Zou, "How is ChatGPT’s behavior changing over time?," arXiv preprint arXiv:2307.09009, 2023. [Online]. Available: https://arxiv.org/pdf/2307.09009.pdf [30] P. P. Martin, D. Kranz, P. Wulf, and N. Graulich, "Exploring new depths: Applying machine learning for the analysis of student argumentation in chemistry," Journal of Research in Science Teaching, pp. 1–36, 2023. [Online]. Available: https://doi.org/10.1002/tea.21903 [31] M. Webb, "A Generative AI Primer," JISC, 2023. [Online]. Available: https://nationalcentreforai.jiscinvolve.org/wp/2023/05/11/generative-ai-primer/#31 [32] V. González-Calatayud, P. Prendes-Espinosa, and R. Roig-Vila, "Artificial intelligence for student assessment: A systematic review," Applied Sciences, vol. 11, no. 12, 5467, 2021. [33] D.C. Englebart, "Augmenting human intellect: A conceptual framework," SRI Summary Report AFOSR-3223, October 1962. [Online]. Available: https://www.dougengelbart.org/pubs/augment-3906.html [34] T. W. Malone, "Superminds: The surprising power of people and computers thinking together," Little, Brown Spark, 2018. [35] V. Kaptelinin and B. Nardi, "Acting with Technology: Activity Theory and interaction
Design," MIT Press, 2006. [Online]. Available: https://doi.org/10.5210/fm.v12i4.1772 [36] E. R. Mollick and L. Mollick, "Assigning AI: Seven Approaches for Students, with Prompts," June 12, 2023. [Online]. Available at SSRN: https://ssrn.com/abstract=4475995 or http://dx.doi.org/10.2139/ssrn.4475995 [37] OpenAI, "Teaching with AI," 2023. [Online]. Available: https://openai.com/blog/teachingwith-ai [38] D. Beiderbeck, N. Frevel, H. von der Gracht, S. L. Schmidt, and V. M. Schweitzer, "Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements," MethodsX, vol. 8, 101401, 2021. [39] S. Aengenheyster, K. Cuhls, L. Gerhold, M. Heiskanen-Schüttler, J. Huck, and M. Muszynska, "Real-Time Delphi in practice - A comparative analysis of existing software-based tools," Technological Forecasting and Social Change, vol. 118, pp. 15-27, 2017. [40] T. Gnatzy, J. Warth, H. von der Gracht, and I. L. Darkow, "Validating an innovative realtime Delphi approach—A methodological comparison between real-time and conventional Delphi studies," Technological Forecasting and Social Change, vol. 78, no. 9, pp. 1681-1694, 2011. [41] T. Gordon and A. Pease, "RT Delphi: An eficient, ’round-less’ almost real time Delphi method," Technological Forecasting and Social Change, vol. 73, no. 4, pp. 321-333, 2006. [42] H. P. McKenna, "The Delphi technique: a worthwhile research approach for nursing?,"
Journal of advanced nursing, vol. 19, no. 6, pp. 1221-1225, 1994. [43] P. L. Williams and C. Webb, "The Delphi technique: A methodological discussion," Journal of advanced nursing, vol. 19, no. 1, pp. 180-186, 1994. [44] S. Chuenjitwongsa, "How to conduct a Delphi study," Medical Education, 2017. [Online].
[45] N. V. Cong-Lem, "Vygotsky’s, Leontiev’s and Engeström’s cultural-historical (activity) theories: Overview, clarifications and implications," Integrative Psychological and Behavioral Science, vol. 56, no. 4, pp. 1091-1112, 2022.