Elaborative Feedback in L2 Reading Games Matthew Pattemore Roger Gilabert Judit Serra University of Barcelona University of Barcelona University of Barcelona Barcelona, Spain Barcelona, Spain Barcelona, Spain matthew.pattemore@ub.edu rogergilabert@ub.edu juditserra@ub.edu ABSTRACT (SLA) feedback has frequently been investigated in terms In combination with instruction, feedback can assist of written CF or oral CF (see Li, 2010 for a general meta- students in comprehension, engagement, and strategy analysis; see Lyster & Saito, 2010 for a meta-analysis of development in second language acquisition. Outcome oral CF; Kang & Han, 2015 for a meta-analysis of written feedback allows learners to understand whether they are CF). This study focuses instead on the relatively newer correct or incorrect. Elaborative feedback can be used to subject of digital or audiovisual CF. remind learners of the underlying processes behind the activities or encourage them to use learning strategies. This This research is being conducted within the iRead project, study is part of a European project called iRead which aims which aims to develop a suite of educational software for to create and test personalised learning technologies to android tablet computers, aimed at helping children to assist primary school children in reading development. We develop their reading skills. It is currently localised into investigate to what extent elaborative feedback, rather than four languages – English, Spanish, German, and Greek. The solely outcome feedback, is taken up by EFL learners. software covers over 250 aspects of phonology, Data analytics from the games will be collected, including morphology, morphosyntax, and syntax, and consists of response correctness, feedback type received, and the learning games, a graded e-reader, and an analytics impact of the feedback on subsequent responses. Results application. The software is presently only available for will be discussed in light of both serious games and SLA researchers, but aims to have a public release after trials theories interested in the effects of feedback on second have been conducted. This study will be focusing on the use language reading development. of the learning games application by 10- and 11-year-old English as a foreign language (EFL) learners in Spain, and Author Keywords how the feedback in those games is designed and what Serious games; feedback; second language acquisition; impact it may have on their game behavior and reading reading skills. development. ACM Classification Keywords These types of games are known as serious games as they J.5. Linguistics; K.3.1. Computer-assisted instruction are designed with a prevalent gaming element, but include (CAI). the intent to improve learning (Ratan & Ritterfield, 2009), INTRODUCTION as opposed to games simply for entertainment purposes. Feedback is information given to learners to allow them to Ritterfield, Cody, and Vorderer (2009) define serious evaluate their output. Corrective feedback (CF) is games as intrinsically motivating because they are fun to information given to help learners fix errors in their output play, but containing content that is complex enough to and/or understanding. Based on cognitive interactionist provide learning opportunities. This differs from theories of language acquisition (e.g. Long, 1996), there is a ‘gamification’ as, in serious games, learning is expected to belief that CF assists learners to notice and attend to happen within the game, while in gamification the gamified language errors in a manner that is beneficial for their long- elements are intended to encourage learning external to the term language development. Many studies have found that, game through greater engagement and motivation (Landers in combination with instruction, feedback can assist 2015). The iRead project, then, can be said to be gamifying students in comprehension, engagement, and strategy the reading development process by introducing (among development. In terms of Second Language Acquisition other things) a serious literacy game. Serious games are a multimodal environment where users are engaged in Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). multiple cognitive tasks concurrently (Johnson et. al. 2017). In: J. Arnedo-Moreno, C.S. González, A. Mora (eds.): Proceedings of the CORRECTIVE FEEDBACK 3rd International Symposium on Gamification and Games for Learning (GamiLearn’19), Barcelona, Spain, 22-10-2019, published at http://ceur- Rod Ellis (2010) laid out a framework for investigating CF, ws.org which can be easily adapted to include digital feedback. He listed six areas of study: type of feedback; individual difference factors; contextual factors; engagement with feedback, and learning outcomes. Form of feedback synthetic voices have been found to increase processing Regarding form of feedback, researchers have proposed demands (Sinatra et. al. 2013). This all adds up to a many categories of feedback (see e.g. Hattie & Timperley, cognitively demanding environment in which working 2007). A central distinction is between outcome feedback memory may play a role in facilitating feedback use. and elaborative feedback (Johnson et. al., 2017). Outcome Engagement with feedback feedback is simply when the learner is informed if their Engagement with feedback is another dimension that will be answer is correct or incorrect. In addition to mere investigated. Affective engagement will be studied through knowledge of result, outcome feedback can provide direct smaller scale studies involving think aloud protocols, as or indirect forms of feedback (also called input-providing or seen in Hookham et. al. (2016), in order to find out what the output-pushing feedback). Direct/Input-providing feedback learners perceive from the feedback and how they feel identifies what the right answer should have been (e.g. about it, and to what extent it assists or detracts from the recasts), while indirect/output-pushing feedback highlight learning experience. Finally, behavioural engagement will exactly where the error was made, and encourages the be observed through the data provided by the software, to learner to self-correct. There is evidence (Yang & Lyster show whether, and in what way, learners take up the 2010, Lyster & Saito 2010) that indirect/output-pushing feedback, as demonstrated by Smith et. al. (2016). feedback better assists long-term development if learners have previously been introduced to the forms. Learning outcomes Finally, an analysis of feedback would be sorely lacking Elaborative feedback (also known as process feedback) without a measure of Ellis’ final dimension, learning gives learners more details, such as explaining why an outcomes. Research from the Language and Reading answer is correct or incorrect, providing information about Research Consortium (2015) has shown that around 90% of the specific concepts or processes involved in the task, or the variance in reading comprehension can be explained by guiding learners to particular strategies that will help them word recognition and listening comprehension skills, both with the task. These types of feedback are not mutually modulated by vocabulary. Using pre-, post-, and delayed exclusive, and indeed it is difficult to give elaborative post-tests, in conjunction with the data provided by the feedback without also giving outcome feedback. There is software, we will have a measure of how effective the little evidence of the effectiveness of different types of feedback is in creating reading development gains. feedback in serious games for SLA, and Benton et. al. (2018) found that many commercially available early There is a lack of studies on the effects of different types of literacy games contained limited elaborative feedback. feedback in the context of serious games for SLA. This is However, there is some evidence from the fields of botany true both for within game learning (how well does the (Moreno, 2004) and electrical circuits (Mayer & Johnson, feedback help learners progress through the game) and 2010) that providing outcome and elaborative feedback external learning (how well has the feedback assisted the together in a game helps to develop learning more than just learner in their overall reading development). There is outcome feedback. limited understanding of how learners engage with feedback in the context of serious games for SLA, and how Complexity of feedback that might be moderated by individual differences between One additional aspect that might be profitably explored is the users, such as working memory. These are the gaps that complexity of feedback. In terms of feedback complexity this research project aims to help fill. generally, Shute (2008) points out that there is no evidence for more complex feedback providing greater results, and AIMS AND METHODOLOGY some evidence that it worsens results. It seems possible that The aim of this study is to gain a better understanding of the elaborative, metalinguistic feedback may be challenging to effects of feedback in serious literacy games. This will use for some lower level learners, or that these more involve gathering data to attempt to answer a number of the explicit forms may be beneficial to older learners. As Li questions posed above, such as: (2010) notes though, there is surprisingly little research on 1) In the context of serious literacy games, does outcome metalinguistic complexity of feedback. plus elaborative feedback lead to better in-game results than Individual differences and feedback outcome feedback alone? Ellis also suggests individual differences as an area to study 2) Are different types of elaborative feedback taken up for CF. While there are many possible individual differently by learners playing serious literacy games? differences that could be explored, including age, language aptitude, gender, prior knowledge, spatial ability, and 3) Do individual differences in working memory affect motivation, this research will target working memory. uptake of elaborative feedback by learners playing serious Working memory is of particular relevance due to the literacy games? multimodal setting, which increases processing demands, in addition to the audio only feedback delivered (mostly) by a 4) How do 10- and 11-year-old users of serious literacy synthetic voice. Audio feedback is considered preferable to games engage with the feedback provided to them on a reduce processing demands (Johnson et. al. 2017) but cognitive, affective, and behavioural level? The primary data collection for the iRead program will for additional study, including the use of think-aloud consist of one academic year’s worth of use of the system, protocols, or using the application under feedback/no for at least one hour a week, by several classes of 10- and feedback conditions for a short period of time. 11-year-old EFL learners in Spain. Additionally, one or two classes of students at the same grade level but who will not be using the system (but rather taking part in their regular reading development activities) will also undergo the pre- and post-tests in order to serve as a control group. Figure 2. Study design Figure 1. iRead literacy games with both outcome and elaborative feedback Outcome feedback in the game is presented in numerous ways, including visual indicators (e.g. a bridge breaking) Several variables will be involved in the research. Type of and as oral speech (e.g. “Great job!”). Elaborative feedback feedback will be an independent variable. Working memory is all presented orally to the learner when they make an and engagement will be moderating variables. error. By undertaking (primarily) classroom research with The dependent variables will fall into two categories: data a commercial quality application, the study will have strong internal to the system, and data external to the system. ecological validity. In addition, the large numbers of participants involved and constant digital data collection Measures will help to strengthen any findings. Internal measures will include data on which features and games learners are using, errors and error recovery, and timing of responses. These will allow us to judge the CONCLUSION effectiveness of the feedback, and whether the different types are taken up in different ways. External measures will This is an ongoing study and as such does not yet have final be assessed with a pre/post-test design, which will allow us results. However data collection has already begun and to measure reading development and motivation. At the there are some emerging trends from the pilot testing and start of the academic year, all students taking part in the early data. In general, the elaborative feedback is not project will be given pre-tests, including tests of: proving to be more effective than simple outcome feedback. 1) Reading skills, including measures of word and non- The 10- and 11-year-old learners in the study are largely word recognition, listening comprehension, and oral seen to ignore longer feedback explanations and make reading fluency in L1 and L2. second attempts before the elaborative feedback has finished playing. In early think-aloud sessions, participants 2) Vocabulary size, as measured by a children’s productive regularly refer to the outcome feedback (e.g. tiles turning vocabulary test developed by Anthony and Nation (2017). red or green) as helping them understand their progression 3) Working memory, as measured by a backwards digit through the topic, but rarely mention using the longer audio span. hints. This finding is in contrast to those of Moreno (2004) and Mayer & Johnson (2010). A possible explanation for 4) A questionnaire gauging individual motivation for this is to do with the age of the participants – our learners learning English, reading, and using learning games. are primary school children rather than university students. Twice during the year, learners will complete interim Another preliminary finding relates to the primacy of engagement questionnaires with additional questions mechanics over language. Many participants struggle specifically relating to the application they have been using. initially with some of the game mechanics and make At the end of the academic year, students will undergo post- numerous non-language errors. On the other hand, by their tests covering all the topics mentioned above in order to third game trial, learners seem to take actions by finding assess development over time. mechanical similarities with previous trials, rather than due To investigate cognitive and affective engagement, smaller to understanding and utilizing the target language points. samples of students will be taken out from their classrooms Further data and analysis will be available for discussion at 12. Laura Benton, Asmina Vasalou, Kay Berkling, Gamilearn’19. Wolmet Barendregt, and Manolis Mavrikis. 2018, April. A Critical Examination of Feedback in Early ACKNOWLEDGMENTS Reading Games. In Proceedings of the 2018 CHI This project has received funding from the European Conference on Human Factors in Computing Systems Union’s Horizon 2020 research and innovation programme (p. 373). ACM. under grant agreement No 731724. 13. Roxana Moreno. 2004. Decreasing cognitive load for novice students: Effects of explanatory versus corrective feedback in discovery-based multimedia. REFERENCES 1. Michael Long. 1996. The role of the linguistic Instructional science 32, 1-2: 99-113. environment in second language acquisition. In 14. Richard E. Mayer and Cheryl I. Johnson. 2010. Adding Handbook of second language acquisition. Wiley. instructional features that promote learning in a game- like environment. Journal of Educational Computing 2. Shaofeng Li. 2010. The effectiveness of corrective feedback in SLA: A meta‐analysis. Language Learning Research 42, 3: 241-265. 60, 2: 309-365. 15. Valerie J. Shute. 2008. Focus on formative feedback. 3. Roy Lyster and Kazuya Saito. 2010. Oral feedback in Review of educational research 78, 1: 153-189. classroom SLA: A meta-analysis. Studies in second 16. Anne M. Sinatra, Valerie K. Sims, Shannon K.T. language acquisition 32, 2: 265-302. Bailey, and Maxine B. Najle. 2013, September. Differences in the performance of older and younger 4. Eunyoung Kang and Zhaohong Han. 2015. The adults in a natural vs. synthetic speech dichotic efficacy of written corrective feedback in improving L2 listening task. Proceedings of the Human Factors and written accuracy: A meta‐analysis. The Modern Ergonomics Society Annual Meeting 57, 1: 1565- Language Journal 99, 1: 1-18. 1569). 5. Rabindra Ratan and Ute Ritterfeld. 2009. Classifying 17. Geoffrey Hookham, Bridgettte Bewick, Frances Kay- serious games. In Serious games: Mechanics and Lambkin, and Keith Nesbitt. 2016, September. A effects. Eds. Ute Ritterfeld, Michael Cody, Peter concurrent think aloud study of engagement and Vorderer. Routledge. 32-46. usability in a serious game. In Joint International 6. Ute Ritterfeld, Michael Cody, and Peter Vorderer. Conference on Serious Games. Eds. Tim Marsh, 2009. Serious games: Explication of an oxymoron- Minhua Ma, Manuel Fradinho Oliveira, Jannicke introduction. In Serious games: Mechanics and effects. Baalsrud Hauge, and Stefan Göbel. Springer. 241-219. Eds. Ute Ritterfeld, Michael Cody, Peter Vorderer. 18. Shamus P. Smith, Daniel Hickmott, Erica Southgate, Routledge. 3-9. Ross Bille, and Liz Stephens. 2016, September. 7. Richard N. Landers. 2014. Developing a theory of Exploring playlearners’ analytics in a serious game for gamified learning: Linking serious games and literacy improvement. . In Joint International gamification of learning. In Simulation & gaming 45, Conference on Serious Games. Eds. Tim Marsh, 6: 752-768. Minhua Ma, Manuel Fradinho Oliveira, Jannicke 8. Cheryl I. Johnson, Shannon K.T. Bailey, and Wendi L. Baalsrud Hauge, and Stefan Göbel. Springer. 13-24. Van Buskirk. 2017. Designing effective feedback 19. Language and Reading Research Consortium. 2015. messages in serious games and simulations: a research Learning to read: Should we keep things simple?. review. In Instructional Techniques to Facilitate Reading Research Quarterly 50, 2: 151-169. Learning and Motivation of Serious Games. Eds. Pieter 20. Laurence Anthony and Paul Nation. 2017. Picture Wouters and Herre van Oostendorp. Springer. 119-140. Vocabulary Size Test (Version 1.2.0) [Computer 9. John Hattie and Helen Timperley. 2007. The power of Software]. Tokyo, Japan: Waseda University. feedback. Review of educational research, 77, 1: 81- Available from 112. http://www.laurenceanthony.net/software/pvst 10. Rod Ellis. 2010. A framework for investigating oral and written corrective feedback. Studies in Second Language Acquisition 32, 2: 335-349. 11. Yingli Yang and Roy Lyster. 2010. Effects of form- focused practice and feedback on Chinese ESL learners’ acquisition of regular and irregular past tense forms. Studies in second language acquisition 32, 2: 235-263.