Inferring Knowledge Acquisition through Web Navigation Behaviour

Abstract

As we witness the growing popularity of online learning, we address the problem of knowing if users are actually learning. The traditional assessment approaches involve tests, assignments and peer assessments. We explore if there is a way to measure learning and personalise the user learning experience in an unobtrusive manner. My PhD proposes using data-driven methods to measure learning by mining user interaction data to identify regularities that could be indicators of learning.

Keywords

• Navigation behaviour
• Knowledge acquisition

Introduction

MOOCs were rst introduced in 2006 and quickly gained popularity in 2012 [ 8 ]. There are many well-known providers such as Coursera, Udacity and edX. with 81 million registered learners in total [ 3 ]. While each platform has its own structure and style, they can generally be divided into two categories: cMOOCs and xMOOCs. These terms were rst proposed by Stephen Downes as an xMOOC resembles more of a traditional course and cMOOCs focus more on generating knowledge through communities [ 9 ].

The goal of this project is to examine the hypothesis of whether knowledge acquisition and learning can be inferred from users navigational behaviours. These interactive behaviours can be low-level events such as a mouse click or high-level events such as searching certain keywords. Both low-level and highlevel events will be included in the investigation as it is shown that context may be spread across multiple events and composition of these events needs to be taken into consideration to nd appropriate interpretations [ 5 ]. Measuring learning and its properties has always been an important practice in education. Traditional methods include modelling students automatically using an assessment system such as knowledge tracing [ 14 ], knowledge assessment [ 13 ], peer review [ 4 ] or endorser review [ 6 ] or using automatic analysis tools such as adaptive assessment functionality [ 11 ]. These systems are convenient as modelling is performed automatically but they also lack personalisation that suits the needs of individual learners [ 10 ] such as content customisation based on individual goals and personal trajectory that optimizes the learning process. Using Web navigation behaviour to assess learning can be applied in consideration of individuality such as di erent level of expertise.

Another motivation for using Web navigation behaviour to measure properties of learning is to provide context to assessments. Additional information such as engagement, material coverage and self-e cacy can potentially o er context and explanation to traditional assessment results. Work has been done on MOOC platforms to measure properties of learning such as using various indicators to measure engagement [ 2 ] and using Bayesian networks to predict dropout [ 7 ]. Our research will be conducted on the MOVING platform which is a cMOOC platform (see Fig. 1).

The MOVING platform will measure properties of learning through the data generated by the following four sources, these measurements will serve as the best practice in our investigation: { The adaptive support training widget which will display charts about the usage of the di erent features of the platform. { Self-assessment data from user answers to questions provided by the adaptive training support and their written feedback. { User prior knowledge level according to self-reported prior knowledge assessment. { Progress data from the curriculum progress widget which will show the status of user progress in the entire curriculum. 3

Research Challenges

To investigate the relations between Web navigation behaviours and learning, three research questions are identi ed: RQ1: What are the traditional assessment methods on MOOC platform and what properties of learning can we investigate?

We explore the current state of the art assessment methods, how they are used and their advantages/disadvantages. We will discuss the challenges of these methods in terms of MOOC settings. We will investigate the properties that are associated with learning, for example, level or engagement or participation. In particular, how are these properties indicate learning and how to measure them.

The research question will be approached by conducting a literature review on assessments and properties of learning.

RQ2: Can we use Web navigation behaviour to measure these properties of learning

We explore whether a user's interactive navigational behaviours can be used as a reliable and e ective way to measure their knowledge acquisition. Two approaches will be taken to investigate the research question. The rst one is literature-driven. Potential interactive behaviours that are connected to learning, in general, can be extracted from the related literature. The second approach is data-driven when we identify frequent navigational behaviours/patterns using pattern-mining algorithms.

RQ3: How do we evaluate our ndings?

If correlations were discovered between navigation behaviour and properties of learning, how can we evaluate these ndings? It is possible that we can evaluate by analysing the longitudinal data and compare the results with our earlier discoveries. Also, we may be able to gather data from other platforms to evaluate these correlations. 4

Contribution

The main contributions of this project will be in the area of learning analytics and human-computer interaction. The project will collect learner-generated interac

He Yu tive weblogs, perform analysis and use the best practice to discover connections that can predict learning, which will be contributions to learning analytics [ 12 ]. It focuses on the interaction between the users and an online learning platform, which suggests contributions to human-computer interaction.

Using Web navigation behaviour to measure learning can be complementary to assessments in many aspects, for example: { Unobtrusiveness. Traditional assessments can be used to motivate the students, however, in certain online learning settings it can potentially pose more problems such as higher dropout rate. It has been expressed that it is a problem if the assessments become obtrusive and learners can be reluctant towards them [ 1 ]. Assessing users by their automatically generated data is less disruptive to their learning process, it can be a signi cant improvement to current assessment methods and online learning platforms. { Automatic analysis. The implementation of e cient assessments is still problematic despite and advancements in learning technology [ 15 ], while our approach can be much more e cient compared to other assessment methods. Monitoring knowledge acquisition using Web behaviour can be achieved automatically with the users data. With a reliable model, analysis and feedback could be done automatically as well. { Context. Measuring the properties of learning can provide more context to traditional assessment methods. It can be used to interpret and explain the test results, to demonstrate individual strength/weaknesses and to generate appropriate assessments.

The project can also contribute to the adaption of online learning platforms. Using automatically generated data and analysis, feedback can be provided automatically. With the feedback, personalised guidance or even learning contents may be provided which can discover more possibilities. 4.1

Future Work Our next goal is to nd correlates between Web navigation patterns and knowledge acquisition. This will be approached from a data-driven and hypothesisdriven angle. However, in the event that the MOVING platform does not contain enough users to carry out the study, a separate study with recruited participants may be practical to continue the project. Other methods may include acquiring and utilising the interactive data from other MOOC platforms that contains more active users.

Statistical analysis such as regression analysis will be performed between knowledge acquisition and Web navigation behaviour to suggest correlation and causality between di erent metrics and behaviours. However, if no such connections are found, it is possible to investigate di erent metrics with navigational behaviours such as the level of engagement, the e ectiveness of the course and dropout behaviours.

The next step is evaluating the models generated from the statistical analyses. For iterative/exploratory testing purposes, small-scale studies will be designed to evaluate the models generated from the previous stage. For con rmatory testing purposes, the models will be investigated as for how they scale on longitudinal analysis of data.

The nal stage is designing interventions. If knowledge acquisition can be identi ed over time, the remaining questions are, how to support those who might be struggling? How to encourage engagement? How to speed up the learning process? Is there any way of delivering interventions without being too intrusive. 5

Acknowledgments

This work was supported by the EU's Horizon 2020 programme under grant agreement H2020-693092 MOVING (http://moving-project.eu).

He Yu

References

1. Apostolos K. : Multilitteratus Incognitus: Allergic to assessment or measurement? ( 2011 ), http://idstu .blogspot.com/ 2011 /10/allergic-to -assessment-ormeasurement .html

2. Bote-Lorenzo , M.L. , Gomez-Sanchez , E. : Predicting the decrease of engagement indicators in a MOOC . In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference on - LAK '17 . pp. 143 { 147 . ACM Press, New York, New York, USA ( 2017 ). https://doi.org/10.1145/3027385.3027387, http://dl.acm.org/citation.cfm?doid= 3027385 . 3027387

3. Dhawal Shah : By The Numbers: MOOCS in 2017 Class Central ( 2018 ), https://www.class-central.com/report/mooc-stats-2017/

4. Gamage , D. , Whiting , M.E. , Rajapakshe , T. , Thilakarathne , H. , Perera , I. , Fernando , S. : Improving Assessment on MOOCs Through Peer Identi cation and Aligned Incentives . In: Proceedings of the Fourth ( 2017 ) ACM Conference on Learning @ Scale - L@S '17. pp. 315 { 318 . ACM Press, New York, New York, USA ( 2017 ). https://doi.org/10.1145/3051457.3054013

5. Hilbert , D.M. , Redmiles , D.F. : Extracting usability information from user interface events . ACM Computing Surveys 32 ( 4 ), 384 {421 (dec 2000 ). https://doi.org/10.1145/371578.371593

6. Kay , J.S. , Nolan , T.J. , Grello , T.M.: The Distributed Esteemed Endorser Review . In: Proceedings of the Third ( 2016 ) ACM Conference on Learning @ Scale - L@S '16. pp. 157 { 160 . ACM Press, New York, New York, USA ( 2016 ). https://doi.org/10.1145/2876034.2893396

7. Lacave , C. , Molina , A.I. , Cruz-Lemus , J.A. : Learning Analytics to identify dropout factors of Computer Science studies through Bayesian networks . Behaviour & Information Technology pp. 1 { 15 (jun 2018 ). https://doi.org/10.1080/0144929X. 2018 . 1485053 , https://www.tandfonline.com/doi/full/10.1080/0144929X. 2018 .1485053

8. Laura Pappano : Massive Open Online Courses Are Multiplying at a Rapid Pace - The New York Times ( 2012 ), https://www.nytimes.com/ 2012 /11/04/education/edlife/massive-open -onlinecourses-are-multiplying-at-a-rapid-pace .html

9. Prpic , J. , Melton , J. , Taeihagh , A. , Anderson , T. : MOOCs and Crowdsourcing: Massive Courses and Massive Resources (feb 2017 ). https://doi.org/10.5210/fm.v20i12. 6143

10. Ren , Z. , Rangwala , H. , Johri , A.: Predicting Performance on MOOC Assessments using Multi-Regression Models (May 2016 ), http://arxiv.org/abs/1605.02269

11. Rosen , Y. , Rushkin , I. , Ang , A. , Federicks , C. , Tingley , D. , Blink , M.J. : Designing Adaptive Assessments in MOOCs . In: Proceedings of the Fourth ( 2017 ) ACM Conference on Learning @ Scale - L@S '17. pp. 233 { 236 . ACM Press, New York, New York, USA ( 2017 ). https://doi.org/10.1145/3051457.3053993

12. Siemens , G.: What Are Learning Analytics ( 2010 ), http://www.elearnspace.org/blog/2010/08/25/what-are -learning-analytics/

13. Wang , S. , He , F. , Andersen , E.: A Uni ed Framework for Knowledge Assessment and Progression Analysis and Design . In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI '17 . pp. 937 { 948 . ACM Press, New York, New York, USA ( 2017 ). https://doi.org/10.1145/3025453.3025841

14. Wang , Z. , Zhu , J. , Li , X. , Hu , Z. , Zhang, M.: Structured Knowledge Tracing Models for Student Assessment on Coursera . In: Proceedings of the Third ( 2016 ) ACM Conference on Learning @ Scale - L@S '16. pp. 209 { 212 . ACM Press, New York, New York, USA ( 2016 ). https://doi.org/10.1145/2876034.2893416

15. Xiong , Y. , Suen , H.K. : Assessment approaches in massive open online courses: Possibilities, challenges and future directions . International Review of Education 64 ( 2 ), 241 {263 (apr 2018 ). https://doi.org/10.1007/s11159-018-9710-5