Reflecting on the Actionable Components of a Model for Augmented Feedback

Abstract

In this paper, we introduce the concept of “augmented feedback” as an enhanced version of traditional educational feedback enriched by digital data and artificial intelligence. To provide an operational definition of augmented feedback, we acknowledge previous research in the fields of technology-enhanced learning and learning analytics. We argue why augmented feedback constitutes a promising research direction for the future of learning. We define the actionable components for a new model of augmented feedback. We also point to relevant use case scenarios stemming from existing projects in learning analytics and artificial intelligence in education, which employ the concept of augmented feedback to various degrees. In doing so, we also point out various open questions and challenges that the notion of augmented feedback implies.

1. Introduction

is going through a series of transformations [ 1 ]. First and foremost, the COVID-19 pandemic heavily disrupted the educational world as we knew it, popularising e-learning as a valid and often preferred mode of instruction. The transformation to digital learning is accompanied by the difusion of new technologies such as smart wearables, depth cameras or virtual and augmented reality devices. The new technologies are accompanied by a “data revolution” in education, driven by advances in data science and Learning Analytics [ 2 ].

While modes of delivering education are transforming, the educational objectives are changing too. The educational institutions are reconsidering themselves and their learning programs by drifting away from performance-oriented learning towards the lifelong acquisition of skills and competencies [ 3 ]. The focus on student competence acquisition as a central educational goal requires a cultural shift in teaching in the sense of learner-centeredness (”shift from teaching to

In this context, learners are continuously required to nurture their competencies by acquiring new knowledge and mastering new skills. The required skills span from basic to complex, as well as from highly specific to transversal ones. Competency-oriented learning requires MILeS 22: Proceedings of the second international workshop on Multimodal Immersive Learning Systems, September 13, nEvelop-O learners to develop skills, knowledge, attitudes, and meta-cognitive abilities through digital tools in a limited time [ 5 ]. Successful learners of the 21st century need not only to be able to acquire cognitive skills but also should be able to self-regulate and adapt dynamically, learn physical interactions and psychomotor skills, and develop approaches to learn all these skills more eficiently.

A way to cater towards this significant demand for learning in the near future is to ofer learners continuous instruction and timely feedback. Novice students, for example, need direct and regular feedback at shorter time intervals, while delayed feedback has proven to be efective for more advanced students [ 6 ]. Given this background, one-size-fits-all feedback is not desirable. Therefore, feedback should be tailored to the individual. Especially in times of Covid-19 and the consequent learning in isolation, the traditional one-to-many teaching model seems unable to ifll such a gap appropriately. Moreover, for reasons of limited time-related and mental capacities, teachers, coaches and professors are often not able to always provide individual feedback where it is urgently needed.

In light of this “feedback gap”, we are motivated to answer the research question, How can educational technologies, learning analytics and artificial intelligence best support learners by providing continuous feedback, promoting skills development and knowledge mastery?

2. Augmented Feedback: a definition

In education science, “feedback” is arguably the most critical pedagogical intervention. Black & Wiliam and Hattie & Timperley show that feedback has one of the most powerful influences on students’ learning goal achievement [ 7, 6 ]. The purpose of feedback is to reduce discrepancies between current understanding or performance and a desired goal [ 6 ].

Feedback can be defined as information presented to the learner after any input to shape the perceptions of the learner [ 8 ]. Feedback can be seen as an exchange between the sender, who is the teacher or human expert and the recipient of the feedback, who is the learner [ 9 ].

What does the term “augmented” then add to the notion of “feedback”? From the point of view of information theory, feedback augmentation can be seen as an “enrichment” of human expert feedback. The augmentation occurs through digital information, such as audio messages, video or 3D animations. “Augmented feedback” does not mean more complex feedback for the learner. As pointed out by the Cognitive Load Theory, having complex explanations on what needs to be improved or which further steps to take can increase the cognitive load and confuse the learner [ 10 ].

Conversely, it is crucial for the learner to have access to personalised support which fits students’ needs to support learning as well as possible. Data-driven systems allow to ”get to know” the learner better and ofer more individualized feedback along with the teacher. This does not mean that they should receive feedback as often as possible. Instead, it is important that learners receive feedback at the ”right” time [ 6 ]. The augmentation, therefore, is not only intended for increasing the availability of feedback which is helpful for scafolding instruction, but also for providing the appropriate feedback at the right time.

Allowing learners to receive feedback anywhere at anytime means expanding the definition of the sender from only a human expert towards Artificial Intelligence (AI) systems supporting the same. The widely accepted characterisation of AI in education is reflected in Intelligent Tutoring Systems (ITS). ITSs, in their original definition, implement an Expert Model that replaces the human expert [ 11 ]. The ITS profiles the learner’s characteristics, strengths and weaknesses. Through a Tutoring Model, the ITS provides personalised and adaptive instructions to the learners. ITSs and other AI educational applications have been researched for more than ifty years. As for other AI domains, the researchers in this field have debated to which extent AI needs to be used for replacement rather than augmenting capability [ 12 ]. While AI has the potential to be always available for the learner and can provide immediate and personalised feedback, the debate seems to be converging that augmentation is the most reasonable choice, as the assistance of human experts is needed and their expertise is hard to replace fully [13].

Especially when providing feedback, the human expert considers typically various aspects such as technical/physical aspects, mental aspects, goals, and possible improvements. An AI that wishes to automate feedback for learning must position the learner in this space of hypothesis and have a holistic vision of the learners’ state, taking various contextual information into account. The AI needs to weigh and balance all the aspects and prioritise those that require feedback. Moreover, when automating feedback with AI, the AI is given high degrees of autonomy and it may make decisions that can be considered unethical and not aligned to the human expert’s goals or with the learner’s expectations. This ethical problem is generally referred to as the “alignment problem” [14].

This paper moves the first steps toward an integrated model for augmented feedback which consider a tight intertwining between the AI and the human expert.

3. The Components of the Augmented Feedback Model

Figure 1 presents the first example of the Augmented Feedback model. In the proposed model, the feedback is an exchange between the Human and the AI expert with the learner. The first proposed version of the model is learner-centred and features four concentric components.

The first component reflects the Domains of Learning of the learner. The domains taken into consideration are (1) the physical domain, (2) the mental domain and the (3) the emotional domain. These domains reflect the ones proposed by Bloom (the psychomotor, the cognitive and the emotional domain). We also add one, the (4) metacognitive domain, which refers to all the aspects such as learning strategies and approaches which are a relevant target for feedback.

The second component is the Temporality of Feedback. The feedback may be given in (near) real-time to optimise the connection between actions and rewards. Alternatively, the feedback can be given in retrospect, for example, at the end of the session, to debrief the mistakes and what can be done to correct them. The temporal component also deals with frequency of the feedback: how often should the feedback be given to the learner to be meaningful?

The third component is the Modality of the feedback. Referring back to its definition, which formulates the feedback as an information exchange, the feedback can be given through multiple modalities which reflect both the modes of interaction with the learning environment as well the senses of the learners. Examples of feedback modalities can be, for example, audio, speech, text or visuals such as images, video or 3D animations. The theory of multimedia learning explores this area [15].

The fourth component is the Connotation of feedback which refers to how the feedback is proposed to the learner. Styles of providing feedback are, for example, answering the questions Where am I going? (FeedUp), How am I going? (FeedBack), Where to next? (FeedForward) [ 6 ], or the Feedback Sandwich, a mixture of positive reinforcing and negative corrective feedback [16]. Alternatively, feedback invites the learner to reflect and figure out what needs to be improved by themselves. Moreover, feedback can look at specific mistakes and possibly show how to correct them. Connotation also looks at the feedback hierarchy. While the learner is deliberately practising, there can be multiple mistakes simultaneously.

Finally, outside the concentric circles, we find the two “feedback senders”: the human expert and the AI. The involvement of these two actors can be diferent and should also be considered a component of the Augmented Feedback model on its own. Humans can be involved in the co-design of the AI tools, in the data annotation, in the revision of the AI feedback or retrospect by providing annotations. For example, the AI can generate feedback texts via a generative language model [17] and then let a human edit the feedback text. Alternatively, the human could write feedback texts and then an AI can elaborate those afterwards.

Furthermore, combining a dashboard of AI-generated indicators with the option to give feedback manually using templates as proposed by the Learning Analytics Cockpit [18] is a viable option for implementing augmented feedback. Such a system could be expanded to allow to massively scale the distribution of feedback, e.g. through AIs in active learning mode analysing and replicating a teacher’s way of providing feedback.

4. Conclusions

With this paper, we aimed to introduce a new feedback model which we coined as “augmented feedback”, intended as an enhanced version of traditional educational feedback through augmentation by digital data and artificial intelligence. After defining the notion of feedback augmentation, we briefly explained the actionable components which combine the strengths or artificial intelligence with the nature of capabilities of humans to provide feedback. The aim of this contribution is to open a discussion with the research community with the intention to further refine the proposed model. [13] K. Holstein, V. Aleven, N. Rummel, A Conceptual Framework for Human–AI Hybrid Adaptivity in Education, in: I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, E. Millán (Eds.), Artificial Intelligence in Education, volume 12163, Springer International Publishing, Cham, 2020, pp. 240–254. URL: http://link.springer.com/10.1007/978-3-030-52237-7_20. doi:1 0 . 1 0 0 7 / 9 7 8 - 3 - 0 3 0 - 5 2 2 3 7 - 7 \ _ 2 0 , series Title: Lecture Notes in Computer Science. [14] B. Christian, The Alignment Problem: Machine Learning and Human Values, 2020. [15] R. E. Mayer, Multimedia learning, Psychology of Learning and Motivation 41 (2004) 85–139.

URL: http://www.sciencedirect.com/science/article/pii/S0079742102800056. doi:1 0 . 1 2 0 7 / s 1 5 3 2 6 9 8 5 e p 4 1 0 2 \ _ 2 . [16] D. M. Dochef, The feedback sandwich, J. Phys. Educ. Recreation Dance 61 (1990) 17–18. [17] A. Filighera, S. Parihar, T. Steuer, T. Meuser, S. Ochs, Your answer is incorrect... would you like to know why? introducing a bilingual short answer feedback dataset, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 8577–8591. URL: https://aclanthology.org/2022.acl-long.587. doi:1 0 . 1 8 6 5 3 / v 1 / 2 0 2 2 . a c l - l o n g . 5 8 7 . [18] O. Karademir, A. Ahmad, J. Schneider, D. Di Mitri, I. Jivet, H. Drachsler, Designing the Learning Analytics Cockpit - A Dashboard that Enables Interventions, in: F. De la Prieta, R. Gennari, M. Temperini, T. Di Mascio, P. Vittorini, Z. Kubincova, E. Popescu, D. Rua Carneiro, L. Lancia, A. Addone (Eds.), Methodologies and Intelligent Systems for Technology Enhanced Learning, 11th International Conference, Lecture Notes in Networks and Systems, Springer International Publishing, Cham, 2022, pp. 95–104. doi:1 0 . 1 0 0 7 / 9 7 8 - 3 - 0 3 0 - 8 6 6 1 8 - 1 \ _ 1 0 .

References

[1] T. Performance , 2022 Learning Trends, 2022 . URL: https://tier1performance.com/ 2022-learning-trends/.

[2] W. Greller , H. Drachsler , Translating learning into numbers: A generic framework for learning analytics , Educational Technology and Society 15 ( 2012 ) 42 - 57 .

[3] Y. Directorate-General for Education , Key competences for lifelong learning , Publications Ofice of the European Union, LU , 2019 . URL: https://data.europa.eu/doi/10.2766/569540.

[4] J. B. Biggs , C. Tang , Teaching for quality learning at university, 4 ed., Open University Press, Buckingham, England, 2011 .

[5] M. Binkley , O. Erstad , J. Herman , S. Raizen , M. Ripley , M. Miller-Ricci , M. Rumble , Defining Twenty-First Century Skills , in: P. Grifin , B. McGaw , E. Care (Eds.), Assessment and Teaching of 21st Century Skills , Springer Netherlands, Dordrecht, 2012 , pp. 17 - 66 . URL: https://doi.org/10.1007/ 978 -94 -007-2324-5_2. doi:1 0 . 1 0 0 7 / 9 7 8 - 9 4 - 0 0 7 - 2 3 2 4 - 5 \ _ 2 .

[6] J. Hattie , H. Timperley , The power of feedback , Rev. Educ. Res . 77 ( 2007 ) 81 - 112 .

[7] P. Black , D. Wiliam , Assessment and classroom learning , Assess. Educ . 5 ( 1998 ) 7 - 74 .

[8] J. V. Dempsey , G. C. Sales (Eds.), Interactive Instruction and Feedback , Educational Technology Pubns, Englewood Clifs , N.J , 1993 .

[9] S. Ramani , K. D. Könings , S. Ginsburg , C. P. van der Vleuten , Feedback Redefined: Principles and Practice , Journal of General Internal Medicine 34 ( 2019 ) 744 - 749 . URL: https://doi.org/ 10.1007/s11606-019 -04874-2. doi:1 0 . 1 0 0 7 / s 1 1 6 0 6 - 0 1 9 - 0 4 8 7 4 - 2 .

[10] J. J. G. Van Merriënboer , J. Sweller , Cognitive load theory and complex learning: Recent developments and future directions , Educational Psychology Review 17 ( 2005 ) 147 - 177 . URL: http://link.springer. com/10.1007/s10648-005-3951-0. doi:1 0 . 1 0 0 7 / s 1 0 6 4 8 - 0 0 5 - 3 9 5 1 - 0 , publisher: Kluwer Academic Publishers-Plenum Publishers.

[11] M. C. Polson , J. J. Richardson , E. Soloway, Foundations of intelligent tutoring systems ., Erlbaum Associates Inc., Hillsdale, NJ, USA, 1988 . URL: http://search.ebscohost.com/login. aspx ?direct=true&db=psyh&AN= 1988 -97172-000&lang=de& site=ehost-live, pages: 293 Publication Title: Foundations of intelligent tutoring systems .

[12] H. Hassani , E. S. Silva , S. Unger , M. TajMazinani, S. Mac Feely, Artificial Intelligence (AI) or Intelligence Augmentation (IA): What Is the Future? , AI 1 ( 2020 ) 143 - 155 . URL: https://www.mdpi. com/2673-2688/1/2/8. doi:1 0 . 3 3 9 0 / a i 1 0 2 0 0 0 8 , number : 2 Publisher: Multidisciplinary Digital Publishing Institute.