=Paper=
{{Paper
|id=Vol-3605/5
|storemode=property
|title=Enhancing Authentic Assessment in Higher Education: Leveraging Digital Transformation and Artificial Intelligence (short paper)
|pdfUrl=https://ceur-ws.org/Vol-3605/5.pdf
|volume=Vol-3605
|authors=Loredana Perla,Viviana Vinci
|dblpUrl=https://dblp.org/rec/conf/aixedu/PerlaV23
}}
==Enhancing Authentic Assessment in Higher Education: Leveraging Digital Transformation and Artificial Intelligence (short paper)==
https://ceur-ws.org/Vol-3605/5.pdf
Enhancing Authentic Assessment in Higher Education:
leveraging Digital Transformation and Artificial
Intelligence
Loredana Perla1, Viviana Vinci2
1 University of Bari, Department of Education, Psychology, Communication, Bari, 70122, Italy
2 University of Foggia, Department of Humanities, Foggia, 71121, Italy
Abstract
The COVID-19 pandemic necessitated universities to adopt online assessments, incorporate hybrid
teaching methods, and diversify their evaluation tools. This transformation centered on principles of
authenticity, accessibility, and responsible data management. Technology, artificial intelligence (AI),
and assessment analytics have played pivotal roles in reshaping assessment practices. Automatic
grading tools like Knowledge Tracing have proven valuable for both in-person and online learning.
Nonetheless, ongoing debates revolve around issues of data literacy and striking a balance between
humanistic and technocratic educational approaches. This proposal delves into the realm of e-
assessment tools, the evolution of assessment objectives, and the challenge of establishing trust in
online assessment processes. Universities are actively working to enhance assessment methods, with a
strong emphasis on authenticity and robust student support. They are also integrating e-assessment
technology to ensure genuine and effective evaluations. The proposal's overarching objective is to
contribute to the ongoing discourse surrounding the promotion of authentic and continuous assessment
within higher education. It aims to explore the opportunities presented by digital transformation, online
learning, and artificial intelligence in this educational context. Additionally, we provide an overview of
a pilot study aimed at investigating the potential application of a multimodal computer vision system to
enhance assessment methods in the university setting and create a supportive learning environment.
Keywords
Authentic Assessment, Artificial Intelligence, Higher education 1
1. Hybrid teaching and technological revolution in the assessment
area
The COVID-19 pandemic has necessitated teachers to fundamentally reassess and restructure
their pedagogical approach, entailing substantial infrastructural and instrumental adaptations,
along with concomitant cultural shifts [1, 2]. In confronting this exigency and the attendant
radical transformation, digitalization has assumed an indispensable role [3].
Nevertheless, it has also unveiled manifest challenges, resistance, and a palpable exigency for
pedagogical support within the academic community [4, 5]. This paradigmatic shift has gradually
reshaped the landscape of pedagogy, engendering a more hybrid and adaptable instructional
modality, extending even to the realm of assessment methodologies.
Specifically, the advent of the COVID-19 pandemic compelled academic institutions to
embrace online assessment solutions, resulting in a notable proliferation of hybrid teaching
modalities [6]. UNESCO [7] provided a summary of the components of effective professional
development for remote and hybrid mode teaching, deepening the foundational knowledge and
AIxIA 2023 - 1st International Workshop on High-performance Artificial Intelligence Systems in Education, November 6-
9, 2023, Rome, Italy
Loredana Perla wrote the paragraph 1; Viviana Vinci wrote the paragraph 2. All authors made contributions to the
article and have reviewed and approved the submitted version.
loredana.perla@uniba.it (L. Perla); viviana.vinci@unifg.it (V. Vinci)
0000-0003-1520-0884 (L. Perla); 0000-0002-4091-0098 (V. Vinci)
© 2023 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
skills specific to the teaching profession, also in the field of assessment and self-assessment
(Table 1).
Table 1
Foundational knowledge and skills specific to the teaching profession (Extracted and adapted from
UNESCO 2023, p. 51)
Student Assessment Self-assessment
Teachers must possess the following Teachers should possess the following
competencies: competencies:
Assessment Proficiency: Teachers need to Regular Self-Assessment: Teachers should
assess student learning, achieve objectives, engage in consistent self-assessment by
monitor progress, and use assessment data to gathering feedback from students and
support students and adapt teaching methods colleagues. They should use this data to enhance
and curriculum. their teaching and make necessary adjustments
Adaptive Assessment: Teachers should use to facilitate students' satisfactory learning and
appropriate assessments for in-person or development.
remote learning to ensure reliable evidence of Seeking Support: Teachers should be proactive
student progress. in seeking support from colleagues and
Diverse Assessment Formats: Teachers should supervisors for areas they identify as needing
be skilled in developing and implementing improvement.
various formats for both formative and
summative assessments, tailored to different In the context of hybrid and remote teaching,
teaching modes. teachers should also:
Technology Integration: Proficiency in various Utilize New Teacher Resources: They should be
assessment technologies and platforms enables proficient in accessing and using new resources
students to demonstrate understanding through that aid in their professional development, such
diverse methods like written work, videos, voice as recording their teaching sessions and
recordings, games, presentations, projects, and reviewing these recordings either individually or
activities. collaboratively with peers.
Student Self-Assessment: Teachers should
teach students how to self-assess their
understanding and encourage them to seek
support when needed.
In the context of hybrid and remote teaching,
teachers must also:
Technology-Enhanced Assessment: Be adept at
applying assessment techniques that leverage
educational technology in all instructional
modes.
Communication of Modes: Teachers should
help students understand various knowledge
transmission modes for effective demonstration
of lesson objectives, especially in remote and
hybrid teaching.
Varied Assessment Methods: Use various
assessment methods to assess knowledge
mastery, giving students flexibility in
demonstrating their understanding without
being limited to specific approaches.
This paradigm shifts also ushered in a greater degree of flexibility in the selection of
pedagogical tools. Notable adaptations included an increased emphasis on formative evaluation,
incorporation of oral presentations, peer assessment, and practical project-based assignments,
facilitated through platforms/tools such as Teams, Canvas, Turnitin, and PowerPoint. Moreover,
this period witnessed the widespread adoption of open-book examinations administered
remotely and the implementation of continuous assessment methodologies.
Concomitantly, the exigencies of the pandemic engendered a requisite reevaluation of
assessment design. This recalibration hinged on foundational principles centered on authenticity,
accessibility, judicious automation, pedagogical continuity, and robust security considerations, as
elucidated by Jisc [8].
The seminal report entitled "Assessment Rebooted" [9] underscored the imperative of
personalized evaluation frameworks that are inherently student-centric. These frameworks,
while ensuring the academic integrity, also espouse responsible data handling, equitable
evaluation practices, inclusivity, and the amelioration of awarding disparities.
In the midst of this evolving cultural landscape, the proposal aims to contribute to the
discourse surrounding the promotion of authentic and continuous assessment within the realm
of higher education. It seeks to explore the opportunities afforded by digital transformation,
online learning, and artificial intelligence in this context. Specifically, the proposal will delve into
several key areas, including the utilization of e-assessment tools, the redefinition of assessment
objectives, and the challenges associated with establishing trust in online assessment practices.
The ongoing dialogue underscores the pervasive influence of the technological revolution in
the field of assessment. As evidenced by the EADTU Report edited by Rossade et al. [10], the
development of online assessment strategies in educational institutions is intricately linked to
technology. This includes the utilization of tools such as quizzes and video-based assessment, as
well as the integration of AI and assessment analytics. These technological advancements are
poised to drive innovation in assessment, particularly in areas like academic writing and
grammar. The evolution of AI and assessment analytics promises to enhance existing practices
and reshape the dynamics of teaching and learning within educational institutions.
Through a meticulous examination of the research landscape and a comprehensive literature
review spanning from 2016 to 2021, del Gobbo et al. [11] shed light on the growing significance
of automatic grading and feedback tools and methods (AGFTM) within online learning systems.
These tools play a critical role, particularly in the assessment of short answers and essays.
One specific AI-supported strategy, as highlighted by Mollick and Mollick [12], involves the
creation of low-stakes tests. These tests can be integrated into various educational contexts, such
as discussions, lectures, individual exercises, homework assignments, or in-class group activities.
When administered regularly, they have proven effective in aiding students in retaining
information over the long term and identifying areas where their comprehension of the study
material may be lacking. Additionally, low-stakes tests provide instructors with invaluable
insights into students' knowledge and understanding, enabling them to adapt their teaching
methods effectively. AI emerges as a valuable tool for educators in generating practice tests,
quizzes, and short-answer questions, including those designed for assessing students' knowledge
during lectures.
Casalino et al. [13] elucidate the application of deep learning techniques in the realm of
Learning Analytics, with a particular focus on the Knowledge Tracing (KT) methodology. KT
employs predictive modeling to ascertain a student's likelihood of successfully completing
exercises based on their historical performance. This approach allows for the modeling of student
behavior over time, offering actionable feedback for both students and educators. Moreover, KT
has the capacity to promptly notify teachers if a student encounters difficulties in acquiring
essential skills, enabling timely interventions. This methodology proves effective in both in-
person and online learning contexts, particularly in blended instructional models.
In the realm of critical perspectives, it is worth noting the thematic series edited by Raffaghelli
et al. [14], which delves into various aspects related to the research problem of data literacy in
Higher Education. This series explores the concept of "data culture" within the broader
institutional context, shedding light on the growing tension between a neo-humanistic
perspective and technocratic imperatives, particularly as artificial intelligence and the Internet
of Things continue to advance as marketable innovations.
In addition, the utilization of AI in educational settings presents several notable risks. These
include concerns related to confabulation, where the generation of plausible yet inaccurate
responses can occur. Moreover, biases, encompassing gender, racial, or ideological biases, may
also manifest within AI systems. Privacy considerations and legal complexities further add to the
intricacies of AI integration. Lastly, there are instructional risks to consider, including the
potential for AI to disrupt traditional learning methods [15]. These critical viewpoints underscore
the multifaceted nature of AI incorporation in education, warranting a careful and vigilant
approach to its implementation.
2. Pilot Study design: towards the Hybridization of Assessment
We present an overview of a pilot study encompassing teachers and students from two
universities in Southern Italy, namely Bari and Foggia, conducted in collaboration with an
international network of university partners. The primary aim of this project is to explore the
application of a multimodal computer vision system to enhance assessment methods within the
university context and foster a supportive learning environment. This transformation seeks to
shift the role of assessment from merely evaluating learning achievement to actively promoting
learning development.
Research hypothesis: the main hypothesis of this study is that the hybridization of
assessment, using a multimodal computer vision system based on AI, will significantly improve
the authenticity, personalization, and flexibility of assessment practices in higher education.
Furthermore, it is hypothesized that this approach will foster a more supportive learning
environment and promote learning development.
Participant selection: two groups of participants will be selected, one from the University of
Bari and the other from the University of Foggia, both consisting of teachers and students using
the AI-based multimodal computer vision learning environment. The sample size will be a
minimum of 100 participants from each university. An international network of universities with
specific expertise in AI application in educational systems, including the College of Computing at
Georgia Tech, the Leibniz Institute for Research and Information in Education, the University of
the West of England, and the Department of Education at the University of Salamanca, will be
involved in the research and system design phase.
Implementation of the Multimodal System: the multimodal computer vision system will be
implemented in collaboration with the Department of Educational Sciences, Psychology,
Communication, and the Department of Computer Science at the University of Bari Aldo Moro.
This system will include components such as Knowledge Tracing Systems, Sentiment Analysis
techniques, formative assessment tools, profiling systems, personalized assessment methods,
and the utilization of extensive language models.
Data collection: data will be collected through questionnaires, interviews, analysis of learning
data generated by the system, and assessment data. The data will include student feedback,
assessment results, knowledge tracking data, and sentiment analysis data.
Data analysis: the collected data will be analyzed to assess the effectiveness of the multimodal
system in enhancing the authenticity, personalization, and flexibility of assessment practices.
Advanced statistical techniques will be used to compare results across different universities and
identify challenges and successes in implementation. Qualitative interviews will be conducted
with teachers and students to gain a better understanding of the challenges and successes
encountered while using the system, with a particular focus on cultural and organizational
differences between the involved universities. These interviews will provide further insights into
the factors influencing the effectiveness of AI-based online education.
Different forms of continuous and learning-based assessments will be experimented with in
the pilot study, using platforms such as Moodle or Learning Management Systems (LMS). This
component will focus on how these forms of assessment can be integrated into the AI-based
online learning environment. Continuous assessment forms may include: student self-
assessment: students will be encouraged to monitor their own learning progress using digital
tools within Moodle or LMS platforms and reflect on their development; quizzes and periodic
activities: online quizzes and periodic activities will be created within the Moodle or LMS
platforms for students to assess their understanding and continuous learning; real-time
feedback: real-time feedback tools within Moodle or LMS platforms, such as chatbots or comment
systems, will be used to provide immediate feedback to students during their learning activities;
personalized eportfolios: students will be encouraged to create and maintain personalized
eportfolios within Moodle or LMS platforms to document their learning, reflections, and progress
within the course. Data collected from various forms of continuous assessment within Moodle or
LMS platforms will be analyzed to understand how these practices impact student learning,
motivation, and participation. The effectiveness of continuous assessment forms in promoting
active learning and student engagement will be evaluated.
Expected Results: A key outcome of this pilot project is to experiment and assess the efficacy
of automated support tools for university educators. These tools aim to streamline the efficient
and timely processing and comprehension of data from diverse sources, thereby providing a
comprehensive understanding of student learning and engagement. It is expected that this pilot
study will provide a clear understanding of the impact of the hybridization of assessment in
higher education and the effectiveness of the multimodal AI-based system. It is hypothesized that
the system will improve the effectiveness of assessment, promoting authentic and personalized
learning. Additionally, the study may reveal specific challenges related to the implementation of
this technology on an international scale and suggest possible solutions to address them. This
pilot study will serve as a solid foundation for further extensive research and may contribute to
the development of guidelines for implementing similar systems in other educational institutions.
The hybridization of traditional university assessment practices with online technologies and
AI necessitates structural adjustments. These modifications encompass redefined roles for both
teachers and students, alterations in the dynamics of time and space, and a heightened emphasis
on feedback and error management, as summarized in Table 2.
Table 2
Technological Hybridization of Assessment
‘Traditional’ Learning Assessment Technology-mediated formative assessment
Assessment of learning Assessment for learning
Individual perspective Self, peer, and co-assessment perspective
Final, summative moment, with few tools Continuous assessment and through narrative
and reflective tools
Teacher centrality Student engagement and feedback
Wide margin of subjectivity and discretion Automation in verification and reduction of
subjectivity
Uniform and non-adaptive tools (tests) Adaptivity, personalization, and progressive
question complexity
Based on predominantly linguistic-symbolic Heterogeneity of communicative channels and
communicative channels and mediators visual, iconic, simulated, hypermedia mediation
Negatively perceived by students Positively perceived (ease of use, interaction,
error reversibility)
This transformative shift entails an augmentation of several key elements, including
authenticity, personalization, choice, flexibility, and a reduction in high-stakes assessments.
These modifications are intended to bolster academic integrity and promote student well-being.
Establishing trust in online assessment practices represents a multifaceted challenge,
encompassing trust in the technology itself, its deployment, the organizational framework,
privacy considerations, and the equitable treatment of individuals within the assessment process.
For instance, the Universidade Aberta in Portugal has pioneered a student-centric virtual
pedagogical model that provides an array of flexible assessment methods, including options for
continuous assessment or a single endpoint examination. Central to these innovative approaches
is the integration of e-assessment technology, aligning with the overarching concept of authentic
assessment [16].
Mas-Garcià and colleagues, within the EADTU Envisioning report 2023 [17], also indicated 10
Methods and activities for learning and assessment with generative AI (in particular, AI text
generators and Large Language Models such as ChatGPT) that can be applied individually or
combined to improve assessment practices, including AI and gamification where appropriate:
1. Infographic Deliverables: require students to create infographics to communicate concepts
effectively and meet specific requirements.
2. Utilize Tests: randomize questions in tests within Learning Management Systems (LMSs)
to assess theoretical knowledge with time limits.
3. Leverage Portfolios: encourage metacognitive reflection through portfolios, where
students explain their approaches, justify decisions, and self-evaluate their work.
4. Implement Co-Assessment: engage students in co-assessment with peers, including
qualitative assessments in final deliverables.
5. Incorporate Oral Tests/Interviews: use oral tests and interviews, including presentation
videos, to assess content and personalize the process.
6. Contextualized Questions: Avoid generic questions and frame them in real-world contexts
related to the course.
7. Use of Chat (Text-Based AI): integrate chatbots or AI for gamified activities and encourage
critical evaluation of responses.
8. Promote Collaborative Work: foster collaboration through discussions, debates, and shared
working processes.
9. Increase Feedback: provide personalized feedback at multiple points during the learning
process, linking it to assignments.
10. Synchronous Tests: consider synchronous tests, including collaborative elements and
practical simulations, to assess applied or practical knowledge.
As elucidated by Cardona and colleagues [18], the infusion of AI into education has the
potential to reshape assessment practices, particularly within the realm of formative assessment.
This evolution necessitates a balanced approach that maintains human oversight, with a primary
focus on students, teachers, and other stakeholders. AI has the capacity to enhance formative
assessments by delivering real-time feedback, adapting to individual learner abilities, and
seamlessly integrating assessments into the learning process. However, it remains paramount to
retain human involvement to ensure fairness, address potential biases, and confront the practical
implications of AI in education. Active participation of teachers and students in the design and
evaluation of AI-augmented assessment tools is imperative to render them both valuable and
user-friendly. Addressing issues related to bias and fairness in AI-enabled assessments is equally
crucial, ensuring that feedback loops enhance the student learning experience while upholding
the principles of privacy and trust within AI systems.
References
[1] T. Estermann, et al., The impact of the Covid-19 crisis on university funding in Europe.
Lessons learnt from the 2008 global financial crisis, Switzerland: EUA, 2020.
[2] L. Perla, V. Vinci, A. Scarinci, Hybrid mediation and Digital Scholarship in Higher Education,
in: G. Ubachs, S. Meuleman, A. Antonaci (Eds.), Higher Education in the new normal: the role
of online, blended and distance learning, European Association of Distance Teaching
Universities, Maastricht, 2021, pp. 58-70.
[3] B. Williamson, R. Eynon, J. Potter, Pandemic politics, pedagogies and practices: digital
technologies and distance education during the coronavirus emergency, Learning, Media and
Technology, 45(2), 2020, pp. 107-114.
[4] C.E. Tømte, T. Fossland., P.O. Aamodt, L. Degn, Digitalisation in higher education: mapping
institutional approaches for teaching and learning, Quality in Higher Education, 25(1), 2019,
pp. 98-114.
[5] J. Fernández Río, M. López Aguado, A. Pérez Pueyo, D. Hortigüela Alcalá, J. Manso Ayuso, La
brecha digital destapada por la pandemia del coronavirus: Una investigación sobre
profesorado y familias. Revista complutense de educación, 2022.
[6] L. Perla, A. Scarinci, I. Amati, Metamorphosis of space into digital scholarship: A research on
hybrid mediation in a university context, In: L. Agrati, D. Burgos, & L. Perla et al (Eds.),
HELMeTO 2020. Bridges and mediation in Higher Distance Education, volume 1344 of
Communications in Computer and Information Science book series, Springer, Cham, 2021.
doi:10.1007/978-3-030-67435-9_18.
[7] UNESCO, Preparing teachers to deliver hybrid education. A framework for Latin America and
the Caribbean. UNESCO and IDB, 2023.
[8] JISC, The future of assessment: five principles, five targets for 2025, 2020. URL:
https://repository.jisc.ac.uk/7733/1/the-future-of-assessment-report.pdf.
[9] Emerge and JISC, From 2020s quick fixes to future transformation, 2020. URL:
https://repository.jisc.ac.uk/7854/1/assessment-rebooted-report.pdf.
[10] K. D. Rossade, J. Janssen, C. Wood, G. Ubachs, Designing Online Assessment - Solutions that
are Rigorous, Trusted, Flexible and Scalable. European Association of Distance Teaching
Universities, Maastricht, 2022.
[11] E. del Gobbo, A. Guarino, B. Cafarelli, L. Grilli, P. Limone, Automatic evaluation of open-ended
questions for online learning: A systematic mapping, Studies in Educational Evaluation, 77
(2023) 101258. doi:10.1016/j.stueduc.2023.101258.
[12] E. Mollick, L. Mollick, Using AI to Implement Effective Teaching Strategies in Classrooms: Five
Strategies, Including Prompts, 2023.
[13] G. Casalino, L. Grilli, P. Limone, D. Santoro, D. Schicchi, Deep learning for knowledge tracing
in learning analytics: an overview, in: TeleXbe, 2021.
[14] J.E. Raffaghelli, S. Manca, B. Stewart, P. Prinsloo, A. Sangrà, Supporting the development of
critical data literacies in higher education: building blocks for fair data cultures in society.
International Journal of Educational Technology in Higher Education, 17(58), 2020. doi:
10.1186/s41239-020-00235-w.
[15] E. Mollick, L. Mollick, Assigning AI: seven approaches for students with prompts, 2023.
[16] V. Villarroel, S. Bloxham, D. Bruna, C. Bruna, C. Herrera-Seda, Authentic assessment: creating
a blueprint for course design, Assessment & Evaluation in Higher Education, 43(5), 2018, pp.
840-854. doi:10.1080/02602938.2017.1412396.
[17] X. Mas-García, L. González-García, D. Gómez-Cardosa, G. Garcia-Brustenga, 10 Methods and
activities for learning and assessment with generative AI, in: EADTU, The Envisioning Report
for Empowering Universities. (nr. 7). 2023, pp. 35-39. doi: 10.5281/zenodo.7993634.
[18] M.A Cardona, R.J. Rodriguez, K. Ishmael, Artificial Intelligence and the Future of Teaching and
Learning. Insights and Recommendations, U.S. Department of Education, Office of
Educational Technology, Washington, DC, 2023.