Using Analytics and Artificial Intelligence to Support Language Learner Decision Making

Abstract

Technology use is deeply rooted within language learning. From the early use of language labs and more recent use of multi-media, we have seen the wide use of technology by language learners. These technologies provide detailed tracking of learner activities that can be harnessed in a way that adapts the learning environment to better meet learner needs. This adaptation can be made by computer programs or humans when analytics, machine learning, and artificial intelligence are used to support the sensemaking and adaptation process. This paper presents an autonomy framework in the context of analytics use. It explores this framework through the discussion of several projects that aimed to enable language learner autonomy by using analytics to help language learners understand their abilities or by recommending potential learning materials and paths to language learners.

Keywords

• 1 Computer Assisted Language Learning (CALL)
• Mobile Assisted Language Learning (MALL)
• Learning Analytics
• Learner Modelling
• Open Learner Models
• Learning Analytics Dashboards

1. Introduction

The use of technology in language learning began with the introduction of tapes where learners would listen to someone talking in the second or additional language (L2) and repeat the utterance after having listened to it. As technology has progressed, we have seen the introduction of more and more complex technologies from computers to mobile phones and the software that runs on these devices.

When the dominant supplementary form of technology transitioned to computers, we largely transferred (1) the type of repeat-after-me speaking activity that was performed with cassettes, and (2) the simple worksheet activities that focused on vocabulary development or grammar-translation approaches to learning. These were reasonable choices given the capability computers at that time. This focus did not shift substantially as technologies progressed. We saw similar patterns of learning activities in the transition to using mobile devices to support language learning.

Since the introduction of these more advanced technologies, we have seen an increase in the provisioning of automated feedback. This change has been facilitated by the availability of data via the Internet and advances in artificial intelligence (AI) and natural language processing (NLP). These technological improvements have meant that we no longer need to elicit incredible amounts of information from experts and then codify that information in software systems. Rather, we can learn from data using the techniques provided by AI and NLP. Consistent with these advances, we are seeing an increase in the development of analytics and adaptive supports for language learners (e.g., [ 1 ] [ 7 ])

Analytics in Language Learning

Most of the analytics used in computer assisted language learning are simple. They also tend to be used when the learner is given constrained tasks. Expediency fueled part of this focus on simple analytics such as number of correct or incorrect answers or number of activities attempted. These types of analytics are easy to obtain, especially when providing learners with constrained tasks, such as cloze items, verb conjugation, and vocabulary translation. These types of constrained tasks are also easy to program and have reduced content-creation burdens because items can be automatically generated from a dictionary or corpus. Another contributing factor is the available technologies. It is still difficult to auto-grade things like essays using pedagogically meaningful approaches; by virtue of the algorithms used, most successful auto-grading ignores the features that are pedagogically meaningful and instead uses relatively simple linguistic features to predict a holistic score [ 8 ], [ 9 ]. This leads to accurate models but fails to provide information that the learner or a teacher can act on. If these stakeholders cannot act on the information from the model, then system developers cannot add appropriate support or adaptation to a system that is meant to scaffold L2 acquisition.

When the analytics have been more complex (e.g., RosettaStone [ 10 ] and Duolingo [ 11 ]) they are still performed in highly constrained settings that support the automated assessment process because the structure imposed by the constraints reduces ambiguity. However, we can and should expect more as computers are now capable of far more. Alelo has been working on developing situated language learning software for over a decade. This software still constrains the learners activities by giving them a mission to complete [ 12 ], [ 13 ]. The learner then interacts with non-player characters within the provided environment to achieve the missions goal. While Alelos software constrains the task, it provides a fairly open environment in which learners can practice their L2: they can choose who to communicate with and how to communicate with those non-player characters.

While Alelo has been focusing on oral language, others have been focusing on writing assessment and support. This is one area where more advanced analytics are beginning to shine. Liaqat and colleagues have focused on combining peer feedback with automated feedback on key rubric elements as a way to facilitate learner use of auto-scoring results [ 5 ]. Litman and her team have been focusing on creating analytics that will identify problems in essay argumentation [ 14 ], and others have been focused on giving feedback about rhetoric in academic writing [ 15 ]. Grammar correction is also an active area of NLP research, with some going as far as trying to detect which errors are due to negative transfer so that the software can automatically provide more targeted feedback that should help the learner modify their mental model and improve their understanding of the L2 [ 7 ].

It should be noted that most of these analytics have only been developed for high-resource languages such as English. Tools to support low or extremely low resource languages are still at the stage of counting errors in highly constrained tasks (e.g., multiple choice items, matching a word in the L2 to its pair in the learners first language ) when analytics are provided at al [ 16 ], [ 17 ].

The more advanced analytics that are discussed above could be used to promote learner autonomy when the output is understandable to learners or the system can use those analytics to adapt the activities to an individual learners needs. 1.2.

Analytics-Driven Adaptation

Using analytics to adapt computer-based learning dates back to the 1970s [ 18 ], [ 19 ], with several advances occurring in the 1980s [ 20 ] and 1990s [ 21 ] [ 23 ]. Providing analytics to learners or teachers so that they could understand and adapt learning began in the 1990s [ 24 ], [ 25 ]. While a wide array of analytics and adaptation processes have been trialed, all of them depend on the same high-level process: Data is obtained, analytics are applied, and the analytics are shared with a decision maker (see Figure 1). Regardless of who is acting on the analytics, a model of what the learner knows or can do is often created and then used to drive adaptation. This model is created via analytics that are applied to learnercreated artefacts and learner actions. The model is continually updated as the learner does new activities. This updating allows for new adaptations that should promote learner growth.

There are three types of decision makers when analytics are used to adapt learning environments or experiences: the software system, the teacher, and the learner. When the software is the decision maker,

Data

Analytics

System

Teacher Learner

the adaptation is performed automatically, and the analytics are kept in their original form. For the other two types of decision makers, the analytics often need to be transformed into something that is humanreadable. In many cases, this may require considerable effort as the target population may lack the background knowledge needed to use the raw analytics. This need is especially pronounced when advanced analytics that involve machine learning or modern techniques from artificial intelligence are being used.

While the approaches used to communicate this information have varied historically, it is now common to visualize some representation of the analytic or model [ 26 ]. It is common for these visualizations to take the form of charts even though other approaches have been trialed. Visualizing the analytics allows the teacher to review the system-generated analytics and adapt the learning activity or experience based on those analytics, their knowledge of the learner or class, and their knowledge of how people learn. A similar approach can be used with learners. When given to learners, these visualizations are called student facing dashboards or open learner models [ 26 ], [ 27 ], and they can enable learners to reflect on their learning, reason about their learning, and adapt their learning activities.

2. Analytics Autonomy Framework

Many assume that using analytics takes power away from learners but that largely depends on who is using those analytics and how they are being used. Analytics can be used to support learner decision making [ 4 ], [ 28 ]. Across the range of analytics, there are three high-level types of learner autonomy: telling (little autonomy), partial autonomy, and full autonomy.

In the telling model of autonomy, the software or teacher tells the learner what they should do. In this case, the only choice that the learner has is whether to comply with the instructions. This is the case in the majority of intelligent tutoring systems, a type of adaptive software that aims to help individual learners improve their knowledge or skills [ 29 ]. This approach has been shown to be effective in many areas of mathematics and physics [ 27 ]. It has also been shown to support student learning of computer languages [ 30 ].

In the partial autonomy model, the learner is given some choice in how to proceed. This model is supported through the use of two key mechanisms. The first mechanism is to restrict learner choices. When using this mechanism, the system or teacher will give the learner a set of options to choose from. The second one is the use of persuasive technologies. These mechanisms give the learner considerable control, but they reduce autonomy by using system features and design to persuade or influence the learner in a specific direction. Under this model the computer or teacher is trying to influence student decisions but still allows the student to make the decision for themselves.

The full autonomy model gives the learner control over all choices. Under this model, the teacher or computer is trying to support student decision making but ceding control to the learner. There are cases, where a persuasive technology approach could fit under this model: it would require that the learner can choose the goals that the persuasive technology is nudging them towards. It would also require that the learner has control over the persuasive mechanisms that are used.

While I have defined, three classes of autonomy, it is possible for hybrid models to exist. Hybrid models that combine different types of autonomy based on the learning environment are arguably the more appropriate choice in many settings. In all cases, analytics are generated based on data. The analytics are then used to support one of these models by allowing the software to make an adjustment automatically, giving the teacher access to the analytics to inform their decision-making, or giving the analytics to the student to inform their decision making.

3. Exemplars of Analytics Use in Language Learning

The below content will walk through several examples of hybrid models for supporting learner autonomy in language learning. These examples do not cover the full range of possibilities but illustrate how some of these options can be balanced with concerns over analytics quality or preferences for approaches to supporting teaching and learning. 3.1.

Mixed-partial Autonomy

In this model, a combination of persuasive approaches and restricted choice are used to support partial learner autonomy.

3.1.1. ProTutor

ProTutor [ 4 ], [ 31 ] was a pronunciation tutor that aimed to also support the vocabulary acquisition of post-secondary learners of Russian as a foreign language. In this system, students would complete games (e.g., memory) and other simple activities (e.g., matching, flashcards) that reinforced their knowledge of vocabulary. The sequencing of this vocabulary was tied to the curriculum and textbooks used in their first- and second-year Russian language courses. Once they had achieved a sub-task within an activity (e.g., matched a word to its image), they would record themselves saying the vocabulary item. Their speech was then analyzed and analytics about their pronunciation accuracy were provided to learners using a variety of communication approaches. One compared their performance against that of an expert speaker, and another compared their current performance to their previous performance. There were also charts to draw learner attention to their strengths and weaknesses. Weaknesses were framed as things that they should work on more.

The analytics in this system were unreliable. Despite this, they could be used to suggest what learners should work on (Figure 2) because they were able to sufficiently distinguish performance. If learners followed these recommendations, they would get targeted instruction and activities that specifically aimed to support the development of their pronunciation of the selected characters in the context in which that student struggled with correctly pronouncing the identified character. For example, if a learner struggled with reducing , they would get items that contained unstressed characters that is, the items they would receive would require the reduction of .

If learners decided to ignore these recommendations, they would continue through the curriculum. The curriculum sequence was matched to that of their course. However, learners could exercise some choice: they were always given a list of 5 items to choose from. So, they could avoid certain activities for a while but would eventually have to do some of them because the incomplete curricular items would not disappear.

Social comparison and several design features around the completeness of activities, effort invested so far, and social comparison were used to persuade the learner to continue working on their pronunciation of difficult sounds. This system seemed to support learner motivation to continue working on improving their pronunciation even though their ability to pronounce words correctly was not a graded component of their courses.

3.1.2. VocabNomad with Goal-based Gamification Support

VocabNomad was a mobile communication and vocabulary acquisition support tool [ 2 ], [ 32 ], [ 33 ]. This tool combined gamification (i.e., a specific type of persuasive technology) with analytics and learner goal setting to support vocabulary acquisition. Learners were given the ability to set goals and then the analytics were used to drive gamification elements that encouraged learners to meet their selfdetermined goals (Figure 3: centre and right). With the pie chart, learners could specifically monitor whether they were meeting their goals as to which macro-skills they were interested in developing.

The goal-setting aspect of this version of VocabNomad, restricted learner choice based on a variety of factors. It would only allow learners to have a certain number of goals at a time, and it would only allow them to select goals that they had the appropriate background knowledge to achieve. If the learner did not have this background, the system would suggest goals that would move them towards their desired goal by helping them to fill in the necessary background knowledge (Figure 3: left). 3.2.

Full Autonomy with Telling

Another version of VocabNomad [ 34 ], [35], used a different approach to supporting learner autonomy while encouraging learning. In this version, learners had full control over what topics they studied. However, the system would make inferences about learner knowledge and expose them to new, related content (Figure 4). Learners were not required to engage with this content, but they could not control their exposure to it: the system always made it available. The inference was based on the theories of fast and extended mapping [36], [37] and would show the synonyms and near synonyms of words that the system believed the learner knew.

There was no way of knowing whether the analytics had made the correct inference, so the system was designed to make low-risk adaptations. It was thought that exposing learners to new words would not harm them and could benefit them, so this was the approach that was taken. This approach was shown to support the development learner vocabulary knowledge under certain usage conditions [38]: when they were working meaningfully with the content. It was also associated with an increase in their willingness to attempt communication in a second-language environment [ 34 ].

Full Autonomy with Persuasion

In this example, learners were engaged in an experience sampling task [ 3 ], [39], [40]: they were prompted at several times throughout the day to create a log of their current experiences (Figure 5). Three of these prompts had them rate their affective state (how they were feeling) using existing scales, and another asked them to report on their most recent attempt at communicating in English [ 34 ]. At the end of the day, learners were asked to reflect over their entire day and report on their experiences. This application had no automated analytics. Rather, learners were meant to reflect on their experiences, create the analytic, and make decisions based on that analytic.

The mobile application would alert learners that it was time to complete a report. The learner could then postpone the reporting or complete it. The reporting options would also time out: the learner had a limited window in which they could complete the requested report. This automated reminder system was the only persuasive aspect that was programmed into the app. Looking at the interface for the communication and contextualization screens shows that learners could choose to respond in whichever media they felt was appropriate. They could also use a language of their choice. Learners chose to always respond using their L2 during the study that we conducted. Some occasionally supplemented their responses with photographs.

What we saw from learners who used this system was that they would reflect on their activities and that this would push them to make changes because they would see the misalignment between their behaviours and goals. As one learner said For me it was very bad that when, its the question, like how many times did you try to communicate. I just keep it, call to my friends and speak with them and then answer. So, its just like, motivator, to use English. When working with motivated post-secondary students this was effective from the perspective of motivating them to change their own learning behaviours without the need for additional intervention.

4. Lessons from Using Analytics to Support Autonomy

All the above examples and my work has been with adult learners, most of whom had a history of relative success. Many of the tools that I briefly introduced above also visualized analytics to support learners metacognitive processes. This suggests a bias in my work towards enabling some level of learner autonomy.

In general, I would recommend hybrid models of autonomy where you can adjust the level of autonomy based on individual learner needs. So, those who can handle increased autonomy get it and those who need more guidance or support get that support. Approaches like this also enable you to change the amount of autonomy based on how students are doing.

I also recommend enabling reflection and supporting analytics use with processes that move students towards being able to (1) make sense of analytics on their own and (2) plan based on the analytics they are given because this creates a skillset that can allow students to continue to grow independently of the learning environment. In many cases, the simplest approaches can be the best. Having students selfrate their abilities and activities and then plan based on those ratings can be highly effective, especially when supported through (1) discussions with a teacher who can help with the goal setting and analytics use process or (2) carefully designed software features that enable this process in more constrained settings.

5. Acknowledgements

This work was done over more than a decade and in collaboration with an amazing group of students and other collaborators.

Funding was received from the Natural Sciences and Engineering Research Council of Canada (NSERC), the Social Sciences and Humanities Research Council of Canada (SSHRC), the National Research Council Canada (NRCC), and Google Research.

6. References

[35] B. Mboutsiadis, C. Demmans Epp, and D. Anayatova, VocabNomad: A context -sensitive mobile application - University English for academic purposes case study, presented at the TESOL International Convention & Language Expo (TESOL), Toronto, Canada, 2015. [36] S. Carey, Beyond Fast Mapping, Lang Learn Dev, vol. 6, no. 3, pp. 184 205, Jun. 2010, doi: 10.1080/15475441.2010.484379. [37] S. Carey, Bootstrapping an d the Origin of Concepts, Daedalus, no. Winter, pp. 59 68, 2004. [38] C. Demmans Epp and K. Phirangee, Exploring mobile tool integration: Design activities carefully or students may not learn, Contemporary Educational Psychology, vol. 59, no. 2019, p. 101791, Oct. 2019, doi: 10.1016/j.cedpsych.2019.101791. [39] S. Consolvo and M. Walker, Using the Experience Sampling Method to Evaluate Ubicomp Applications, IEEE Pervasive Comput., vol. 2, no. 2, pp. 24 31, Apr. 2003, doi: 10.1109/MPRV.2003.1203750. [40] J. M. Hektner, J. A. Schmidt, and M. Csikszentmihalyi, Experience sampling method: measuring the quality of everyday life. Thousand Oaks, Calif: Sage Publications, 2007.

References

[1] Y. Cui and S. Bull , Context and learner modelling for the mobile foreign language learner , System , vol. 33 , no. 2 , pp. 353 367 , Jun. 2005 , doi: 10.1016/j.system. 2004 . 12 .008.

[2] C. Demmans Epp , Developing an Adaptive Mobile Tool to Scaffold the Communication and Vocabulary Acquisition of Language Learners, in Handbook of Mobile Teaching and Learning , 2nd ed., Y. Zhang and D. Cristol, Eds. Berlin, Heidelberg: Springer, 2019 , pp. 1 26 .

[3] C. Demmans Epp , English language learner experiences of formal and informal learning environments , in Learning Analytics and Knowledge , Edinburgh, Scotland, 2016 , pp. 231 235 , doi: 10.1145/2883851.2883896.

[4] C. Demmans Epp and G. I. McCalla , ProTutor: Historic open learner models for pronunciation tutoring , in Artificial Intelligence in Education (AIED) , Auckland, New Zealand, 2011 , vol. 6738 , pp. 441 443 , doi: 10.1007/978-3- 642 -21869-9_ 63 .

[5] A. Liaqat , C. Munteanu , and C. Demmans Epp , Collaborating with Mature English Language Learners to Combine Peer and Automated Feedback: a User-Centered Approach to Designing Writing Support , Int J Artif Intell Educ , Jul. 2020 , doi: 10.1007/s40593-020-00204-4.

[6] A. Liaqat , G. Akcayir, C. Demmans Epp , and C. Munteanu , Mature ELLs Perceptions Towards Automated and Peer Writing Feedback, in Transforming Learning with Meaningful Technologies , 2019 , pp. 266 279 , doi: https://doi.org/10.1007/978-3- 030 -29736-7_ 20 .

[7] L. Wanderley and C. Demmans Epp , Identifying negative language transfer in writing to increase English as a Second Language learners metalinguistic awareness , in Bringing Together Writing Tool Design, Writing Analytics and Writing Pedagogy Workshop at the International Conference on Learning Analytics and Knowledge (LAK) , Frankfurt, Germany (online) , 2020 , pp. 1 4 , [Online]. Available: https://writinganalytics.zhaw.ch/wpcontent/uploads/2020/03/LAK20_Writing_Analytics_Workshop-Identifying- negative-languagetransfer-in-writing-to-increase-English-as-a- Second- Language-learners%E2% 80 % 99 - metalinguistic-awareness.pdf.

[8] R. Correnti , L. C. Matsumura , E. Wang , D. Litman , Z. Rahimi , and Z. Kisa , Automated Scoring of Students Use of Text Evidence in Writing , Reading Research Quarterly, vol. 55 , no. 3 , pp. 493 520 , 2020 , doi: https://doi.org/10.1002/rrq.281.

[9] Z. Ke and V. Ng , Automated Essay Scoring: A Survey of the State of the Art , pp. 6300 6308 , 2019 .

[10] Rosetta Stone . http://www.rosettastone.com/ (accessed Mar. 31 , 2012 ).

[11] L. von Ahn , Duolingo: Learn a Language for Free While Helping to Translate the Web , in Proceedings of the 2013 International Conference on Intelligent User Interfaces , New York, NY, USA, 2013 , pp. 1 2 , doi: 10.1145/2449396.2449398.

[12] W. L. Johnson , A. Valente , and R. Heuts , Multi -Platform Delivery of Game-Based Learning Content, presented at the New Learning Technology Conference , 2008 .

[13] W. L. Johnson , S. Marsella, and H. Vilhjálmsson , The DARWARS Tactical Language Training System , presented at t he Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), Orlando , 2004 .

[14] Z. Rahimi , D. Litman , R. Correnti , E. Wang , and L. C. Matsumura , Assessing Students Use of Evidence and Organization in Response-to-Text Writing: Using Natural Language Processing for Rubric-Based Automated Scoring , Int J Artif Intell Educ , pp. 1 35 , Mar. 2017 , doi: 10.1007/s40593-017-0143-2.

[15] D. Simsek , S. B. Shum , A. De Liddo , R. Ferguson , and . Sándor, Visual Analytics of Academic Writing, in Learning Analytics and Knowledge , New York, NY, USA, 2014 , pp. 265 266 , doi: 10.1145/2567574.2567577.

[16] D. Lothian , G. Akcayir, and C. Demmans Epp , Accommodating Indigenous People When Using Technology to Learn Their Ancestral Language , in International Workshop on Supporting Lifelong Learning co-located with the 20th International Conference on Artificial Intelligence in Education (AIED) , Chicago, Illinois, 2019 , vol. 2395 , pp. 16 22 , [Online]. Available: http://ceurws.org/Vol- 2395 /.

[17] D. Lothian , G. Akcayir , A. Sparrow , O. Mcleod , and C. Demmans Epp, SoundHunters: Increasing Learner Phonological Awareness in Plains Cree , in Artificial Intelligence in Education, Cham, 2020 , pp. 346 359 , doi: 10.1007/978-3- 030 -52237-7_ 28 .

[18] B. Carr and I. P. Goldstein , Overlays: a Theory of Modelling for Computer Aided Instruction , Massachusetts Institute of Technology Artificial Intelligence Laboratory, AI Memo 406 , 1977 .

[19] E. Rich , User Modeling via Stereotypes, Cognitive Sci. , vol. 3 , no. 4 , pp. 329 354 , 1979 .

[20] J. R. Anderson , C. F. Boyle , and G. Yost , The Geometry Tutor, in 9th International Joint Conference on Artificial Intelligence (IJCAI) , 1985 , vol. 10 , pp. 1 7 , [Online]. Available: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.75.7590.

[21] A. T. Corbett and J. R. Anderson , Knowledge Tracing: Modeling the Acquisition of Procedura l Knowledge, UMUAI , vol. 4 , no. 4 , pp. 253 278 , 1995 , doi: 10.1007/BF01099821.

[22] S. Ohlsson , Learning from Performance Errors, Psychol . Rev., vol. 103 , no. 2 , pp. 241 62 , 1996 .

[23] G. Weber , Episodic Learner Modeling, Cognitive Science , vol. 20 , no. 2 , pp. 195 236 , 1996 .

[24] S. Bull , P. Brna , and H. Pain , Extending the Scope of the Student Model, UMUAI , vol. 5 , no. 1 , pp. 45 65 , 1995 .

[25] J. Kay , A Scrutable User Modelling Shell for User -Adapted Interaction , PhD , Basser Dept. of Computer Science, University of Sydney, 1999 .

[26] C. Demmans Epp and S. Bull , Uncertainty representation in visualizations of learning analytics for learners: Current approaches and opportunities , IEEE TLT , vol. 8 , no. 3 , pp. 242 260 , 2015 , doi: 10.1109/TLT. 2015 . 2411604 .

[27] R. Bodily et al., Open learner models and learning analytics dashboards: a systematic review , in Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK) , 2018 , pp. 41 50 , doi: 10.1145/3170358.3170409.

[28] S. Bull and J. Kay , Open Learner Models as Drivers for Metacognitive Processes , in International Handbook of Metacognition and Learning Technologies, R. Azevedo and V. Aleven , Eds. Springer New York, 2013 , pp. 349 365 .

[29] K. VanLehn , The Behavior of Tuto ring Systems , Int. J. Artif . Intell. Ed., vol. 16 , no. 3 , pp. 227 265 , Aug. 2006 .

[30] D. Barrow , A. Mitrovic , S. Ohlsson , and M. Grimley , Assessing the Impact of Positive Feedback in Constraint-Based Tutors , in Intelligent Tutoring Systems (ITS) , Montreal, Canada, 2008 , pp. 250 259 .

[31] C. Demmans Epp , ProTutor: A pronunciation tutor that uses historic open learner models , MSc , University of Saskatchewan, Saskatoon, Saskatchewan, Canada, 2010 .

[32] S. Tsourounis , C. Demmans Epp , and R. M. Baecker , V ocabNomad: Exploiting gamification in a mobile vocabulary learning tool, presented at the 31st Annual CALICO Conference , Athens, OH, USA, 2014 .

[33] S. Tsourounis and C. Demmans Epp , Learning dashboards and gamification in MALL: Design guidelines in practice, in The International Handbook of Mobile-Assisted Language Learning, A . Palalas and M. Ally , Eds. Beijing, China: China Central Radio & TV University Press Co., Ltd ., 2016 , pp. 370 398 .

[34] C. Demmans Epp , Supporting English Language Learners with an Adaptive Mobile Application , Doctoral, University of Toronto, Toronto, ON, Canada, 2016 .