In this report, we summarize the discussions that took place during the workshop on Automated Assessment and Guidance of Project Work at AIED 2023 in Tokyo, Japan. The workshop organizers and participants discussed the critical considerations and challenges in automating assessment and guidance for Project-Based Learning (PBL). ∗Corresponding author.
The rapid pace of job automation has led to a significant increase in workforce displacement
[
The projects’ learning outcomes and the success of the project can vary significantly depending on the PBL implementation, monitoring, and guidance. However, monitoring a PBL activity is not easy due to the process’s unstructured and complex nature. Unlike the relative structure of direct instruction, each PBL can have a diferent degree of instructor support or student choice. Moreover, students behave in an unstructured way, they move around the classroom, they alternate between the use of a computer and working on a physical prototype, and they also work collaboratively at times and individually at other times. For example, if a project requires the students to design a pet robot, the students will have to write a code on the computer, draw a prototype, and build the robot. Each of these tasks will be conducted both in groups (via discussions) and individually (by writing the code).
As a part of the AIED 2023 workshop on the Automated assessment and guidance of project
work, participants brainstormed the critical considerations when designing automated
assessments and guidance for PBL. As it is today, the majority of multi-modal learning analytics
innovations are learning-design agnostic, even though learning analytics that are designed
considering the particularities of the learning task design have more chances to impact students’
learning [
Consideration of Project Phase Guidance strategies can vary based on the project stage, and automated guidance should account for the specific project phase. Projects can have diferent structures [ 8] and can involve diferent phases (e.g.: problem definition, solution design, implementation, prototyping) [9] During each stage, students interact diferently and studies on projects need to account for the resulting diferences in learning [ 10]. For instance, the initiation phase might require more guidance for idea generation and problem definition, while the execution phase might focus on application and problem-solving. Authenticity of Student Output With the advent of AI and other digital tools, there’s a risk that students might use these technologies to produce their work. The participants discussed the importance of creating projects complex and interdisciplinary and cannot be solved solely by the use of AI tools or other resources available online.
Criteria for Successful Output Defining what constitutes a successful project can be challenging. The participants proposed to base success criteria on the students’ own objectives, allowing for personalized learning experiences and outcomes.
Assessing Beyond Content Knowledge Project-based learning is not just about content mastery. It ofers opportunities for students to demonstrate other skills such as creativity, teamwork, and communication. Automated assessment systems should be designed to evaluate these essential skills.
Types of Assessment - Formative vs. Summative Formative assessments provide continuous feedback, helping students refine their understanding and approach throughout the project. In contrast, summative assessments evaluate the overall learning and project outcome at its conclusion. Both types of assessments have their place in PBL and should be integrated into the automated system.
Scalability in the assessment of Domain-Specific PBL Depending on the course, PBL can be tailored to cover specific content knowledge. This specificity means that generic assessment tools or guidelines might not be directly applicable. For instance, an automated assessment designed for a PBL project in environmental science might not be suitable for one in ancient history. Additionally, there is often a limited number of data on each domain-specific PBL, and the data on all PBL processes is often aggregated. This limitation poses a challenge for automated systems that rely on vast amounts of data to refine and improve their assessment algorithms. To address these challenges, there might be a need for more customized automated assessment tools for each domain. While this approach might seem resource-intensive, it ensures that the assessments are tailored to the unique requirements and challenges of each domain-specific PBL.
During the workshop, participants were divided into groups and tasked with creating mind maps using Miro boards to explore challenges related to the automation of assessment and guidance in project work. This activity aimed to foster a deeper understanding and generate discussions around the challenges and considerations in automating interactions during ProjectBased Learning (PBL), the evaluation of project work, and the privacy concerns inherent in such automation.
Detecting Interactions While learning, students interact with each other (talking, gazing, joint attention), with the teachers, with the content (ofline and online), and with the technologies and tools around them (building prototypes) [11, 12, 13]. Participants delved into the complexities of detecting interactions that occur during PBL. Recently, various types of data have been captured in an attempt to measure these interactions. Text, speech, sketch, and handwriting analysis have been used to understand and predict the evolution of students’ learning [14, 15]. Facial expressions have been measured and have been used to predict learning outcomes [16], and physiological markers have been measured to detect afective states [ 17]. Actions and gestures such as joint visual attention, head pose, or eye contact have also been used to estimate student engagement and attention [18, 19, 20]. Participants highlighted the challenges arising from the diverse ways in which individuals from diferent cultures might express themselves diferently. Even though the use of multimodal learning analytics is increasing, the majority of models are developed and evaluated with participants from Western populations [21]. Predictive features that apply to a certain population might not apply to another and can cause unfitting conclusions and biases. This can particularly be the case for behaviors and emotions models as it is well documented that the behavior and expression of emotions are diferent in diferent countries. For example, Akeshi and colleagues showed that East Asian cultures perceive another face as being angrier, unapproachable, and unpleasant when making eye contact as compared to individuals from Western European culture [22]. Automated Evaluation The conversation around automated evaluation focused on the challenges related to providing relevant feedback to students. It was noted that students might embark on diferent pathways during PBL, leading to varied progress and results. The importance of ensuring that the feedback is pertinent and constructive, considering the diverse approaches students might adopt during their project work, was emphasized. Privacy Privacy emerged as a pivotal theme, with discussions revolving around minimizing the use of Personally Identifiable Data (PID). The groups underscored the necessity for AI, used in automating guidance and assessment, to be explainable and transparent. The discussions also highlighted the importance of being cognizant of the data collection methods, the populations on which data is collected, and the potential biases that may be inherent in the collected data. This is crucial to ensure ethical and unbiased modeling of learners and to address any disparities and inequalities that might arise from the automation processes.
The AIED 2023 workshop delved into the complexities and considerations of automating assessment and guidance in Project-Based Learning (PBL), highlighting the importance of nuanced approaches due to PBL’s unstructured nature. Discussions emphasized the need for specificity in automated guidance across diferent project phases and the importance of evaluating intangible skills and maintaining the authenticity of student output. Challenges in detecting diverse student interactions and providing relevant, constructive feedback were explored, with a focus on addressing potential biases and ensuring applicability across diferent cultures. Privacy and ethical considerations, including minimizing the use of Personally Identifiable Data and maintaining transparency in AI, were deemed essential to address disparities and inequalities in automation processes. The insights from the workshop are important for refining future innovations and implementations in automated PBL assessments, aiming for enhanced and equitable learning experiences. Taking tools to task, The Internet and Higher Education 45 (2020) 100729. Publisher: Elsevier. [7] L. Yan, L. Zhao, D. Gasevic, R. Martinez-Maldonado, Scalability, sustainability, and ethicality of multimodal learning analytics, in: LAK22: 12th International Learning Analytics and Knowledge Conference, 2022, pp. 13–23. [8] P. Skillen, Project Based Learning (PBL): Am I Doing it Right?, 2019. URL: https:// plpnetwork.com/2019/02/14/pbl-right/. [9] L. Honglin, N. Yifan, The Construction of Project-Based Learning Model Based on Design Thinking, in: 2022 4th International Conference on Computer Science and Technologies in Education (CSTE), IEEE, 2022, pp. 173–177. [10] M. C. English, A. Kitsantas, Supporting student self-regulated learning in problem-and project-based learning, Interdisciplinary journal of problem-based learning 7 (2013) 6.
Publisher: Purdue University Press. [11] T. Anderson, D. R. Garrison, Learning in a networked world: New roles and responsibilties, in: Distance Learners in Higher Education: Institutional responses for quality outcomes.
Madison, Wi.: Atwood, 1998. [12] R. M. Bernard, P. C. Abrami, E. Borokhovski, C. A. Wade, R. M. Tamim, M. A. Surkes, E. C.
Bethel, A meta-analysis of three types of interaction treatments in distance education, Review of Educational research 79 (2009) 1243–1289. Publisher: Sage Publications. [13] M. G. Moore, Three types of interaction, Taylor & Francis, 1989. [14] P. Blikstein, M. Worsley, Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks, Journal of Learning Analytics 3 (2016) 220–238. [15] B. Sherin, A computational study of commonsense science: An exploration in the automated analysis of clinical interview data, Journal of the Learning Sciences 22 (2013) 600–638. Publisher: Taylor & Francis. [16] M. Worsley, P. Blikstein, Using learning analytics to study cognitive disequilibrium in a complex learning environment, in: Proceedings of the Fifth International Conference on Learning Analytics And Knowledge, 2015, pp. 426–427. [17] K. Muldner, W. Burleson, K. VanLehn, “Yes!”: Using tutor and sensor data to predict moments of delight during instructional activities, in: International Conference on User Modeling, Adaptation, and Personalization, Springer, 2010, pp. 159–170. [18] M. Raca, Camera-based estimation of student’s attention in class, Technical Report, EPFL, 2015. [19] M. Raca, R. Tormey, P. Dillenbourg, Sleepers’ lag-study on motion and attention, in: Proceedings of the fourth international conference on learning analytics and knowledge, 2014, pp. 36–43. [20] B. Schneider, K. Sharma, S. Cuendet, G. Zuferey, P. Dillenbourg, R. Pea, Leveraging mobile eye-trackers to capture joint visual attention in co-located collaborative learning groups, International Journal of Computer-Supported Collaborative Learning 13 (2018) 241–261.
Publisher: Springer. [21] Y. H. V. Chua, J. Dauwels, S. C. Tan, Technologies for automated analysis of co-located, reallife, physical learning spaces: Where are we now?, in: Proceedings of the 9th international conference on learning analytics & knowledge, 2019, pp. 11–20. [22] H. Akechi, A. Senju, H. Uibo, Y. Kikuchi, T. Hasegawa, J. K. Hietanen, Attention to eye contact in the West and East: Autonomic responses and evaluative ratings, PloS one 8 (2013) e59312. Publisher: Public Library of Science San Francisco, USA.