Reading skill, an important and complex cognitive ability, is one of the target skills for talent training. Especially with the trend of online learning becoming more and more prevalent, digital reading literacy is a necessary skill for learners. Therefore, many researchers started to focus on study reading process, and study methods are diverse, such as by collecting eye movement data. However, most researches were limited to scope of laboratory, so it is difficult to achieve low-cost, large-scale, non-invasive and objective reading process data collection and analysis. Therefore, this research developed a platform using web camera of laptop and open-source eye tracking library (webgazer.js) to get physiological indicators based on eye tracking data, which can be used to analyze reading process and explore the impact of the reading performance. This research also used inferential statistics and machine learning methods to quantitatively characterize and analyze the relationship between reading behavior and reading performance. This study's conclusions can help make subsequent targeted adjustments and interventions to improve learners' reading performance, and prove the usability for collecting eye-tracking data by platform with web-camera.
Reading skill is an important and complex cognitive ability, and is always regarded as one of the
most significant skills for learning and living in future life. Especially with the prevalence of online
learning, digital reading literacy is an indispensable skill for everyone to study online[
As far as existing research is concerned, eye tracking has been widely used, and was regarded as an
effective tool for analysis of reading process[
In general, the contributions of this research are summarized as follows: (a) We build a platform for the reading process to realize eye tracking based on web-camera, and then explore the relationship between reading behavior indicators and reading performance based on statistical analysis methods and machine learning methods. The results obtained can explore the laws that exist between reading behavior and reading performance. (b) By conclusions obtained in this research with the conclusions of existing research, it can further prove the reliability of the eye tracking platform for the reading process constructed by this research. The platform can use for low-cost, large-scale, and convenient collection of eye tracking data during the reading process in subsequent research, and further explore the relationship between reading behavior and reading performance. The rest of the paper is structured as follows: Section 2 introduces background and related work. And then, Section 3 shows methodology and process of our experiment. Moreover, Section 4 presents our results. Finally, Section 5 concludes and proposes directions for future work.
Reading skill is an important ability for higher education learners, because it is an important
influencing factor for future work and lifelong learning[
Eye tracking technology is currently widely used in the study of complex cognitive processes such
as reading. It is mainly supported by eye-mind-assumption[
From the perspective of data, indicators of eye tracking data are meaningful for the analysis of
complex cognitive activities such as reading. This can be supported by eye-mind hypothesis, which was
present in former content. Based on this theory, we can know that if one person focuses on an area over
certain time, they probably have difficult to understand the information, or was bored with information
in this area. In existing researches, indicators can be divided into two categories, namely, visual
representations such as heat maps, and quantitative representations such as annotation rates[21]. Based
on these data, Area of Interest (AOI) can be identified during the individual cognitive activity. This
means that the learner may have cognitive difficulties or higher interest in the corresponding area[24].
For example, in the process of designing and optimizing the learning platform, the relevant data of eye
tracking can be used as auxiliary decision-making information[
Analyzing reading from the perspective of learning content can be approached from two different
levels. First of all, vocabulary is an important influencing factor and predictive factor for text reading[
Therefore, the main research question of this study is whether the reading behavior reflected by the indicators collected by self-developed reading platform for eye-tracking in the reading process obtained by eye tracking will have a significant impact on the effect of reading? Based on above analysis, we formed two hypotheses: (a) H1: There is a significant correlation between some reading process behavior indicators and reading performance. (b) H2: Some indicators with specific meanings have a significant impact on academic performance`.
This research designed and implemented an experiment to explore., and the details of the
experimental process are shown in the figure 1. This research used eye tracking technology to obtain
learners' reading behavior data during English reading tests. Participants speak English as a second
foreign language. Finally, the subjects were interviewed, which means to know if there are some
difficulties during the test and help verify our conclusion[
The experiment uses a laptop computer with a 46.67-cm (14 in.) diagonal display, a 720pHD camera (about 920,000 pixels), which used for eye-tracking devices during the experiment to realize nonintrusive data collection. And, an eye tracking library, webgazer.js (https://webgazer.cs. brown.edu/), can predict the user's eye-gaze location[32]. During reading process, the system can automatically get the positions of the pdf window on the screen, the current page number, and the distance between the top of the current page and the top of the current pdf window in real time. And then system can judge the eye-gaze position predicted by webgazer.js to fall within which line, and, that is to say, the content read by the user at the current moment can be predicted. Based on these, we develop a software called Readgazer. We used HTML5, Flask back-end framework and MongoDB database to develop such a system, and used python 3.9 and SPSS to analyze data. The average error with the best model in the experiment was at about 130 pixels. In order to combine the reader's eye-gaze position with the text content in the document, we use pdf.js (https://github.com/mozilla/ pdf.js/) to render pdf documents. So, we can get basis to compute eye-tracking indicators.
The reading test selected in this study includes five questions in total (T1, T2, T3, T4 and T5). Since the types and difficulty of the questions are different, the correlation and influence of them will be explored respectively. For the acquired data, SPSS 21.0 is used for Inferential Statistical Analysis, and use algorithm in machine learning to do data mining, and further analyze the relationship between reading behavior and reading performance. Eye-tracking indicators are shown in table 1.
page i/total number of fixations on all pages)
number of fixations on row j of page i) content and questions, each page
page i/total number of fixations on page i) 10 lines (a total of 56 of reading and
4.1.
Correlation and regression analysis results of eye-tracking indicators and reading performances are shown in Table2. In this study, the main of correlation analysis was to filter indicators, which has significant relation with score of each item. Then, logistic regression analysis can use these indicators to identify key indicators, which can have significant relationship with reading performance for each item. For reading performance of T2, there is no significant correlation with all reading behavior indicators, which means that no reading behavior indicator can significantly influence T2 reading performance.
Take T1 as an example to explain the data analysis results in Table2. From table 2, we can find that there are three indicators significantly correlated with the performance of learners on the first question. Based on this, results of logistic regression using the selected indicators indicate that “pagefreq_onpage_rate_total” can predict the learner's correct rate on the first question. Although, p value less than 0.05, and “Hosmer & Lemeshow” value more than 0.05, which means that the logistic analysis result of T1 with responding indicators is statistically significant. Nagelkerke R2 is 0.222, which shows that this logistic regression model can explain 22.2% of change in dependent variable. Percentage accuracy represents that 59.4% results of predict are correct. But, sig value of pagefreq_onpage_rate_total is 0.081 more than 0.05, which means that the prediction of this indicator is not statistically significant. Based on this, we can conclude that the higher the proportion of the participant’s annotation frequency to the overall text, the more concentrated the participant’s attention, and therefore the better the participant’s answering performance on T1.It should be noted that in order to be able to compare the difference in importance between different indicators in the same logistic regression analysis result, the study standardized all indicators by Z-score before performing logistic regression analysis. For example, the logical analysis result of T5 shows that freq_rate_2_8 has a greater influence on T5's reading performance prediction than freq_3_2.
In general, through the correlation analysis and logistic regression analysis of each topic, we can find that: (a) For a relatively simple topic such as T1, as long as the participant can ensure concentration during the reading process There is a great possibility that the answer is correct. (b) For topics such as T3 and T4 that indicate the approximate area of the corresponding article content, the more participants pay attention to the corresponding content or topic of the reading article, it means that the participant has processed more information in that area. The processing process, that is, the subject may have doubts, so the subject's performance on the corresponding question will be worse. (c) For T5 topics that do not indicate the approximate area of the topic corresponding to the content of the article, the higher the participant’s attention to the topic and the corresponding text content in the text, the more information is processed and processed at the corresponding location. There is no proof of a specific text content area. After the correct content can be recognized, the subject will further confirm the text content and the topic. Therefore, the more subject focus on corresponding area, the more correct may be. 4.2.
In order to be able to more clearly characterize the predictive effect of behavior indicators on reading performance, the decision tree and random forest algorithm in machine learning is used in this study to further determine the impact of behavior indicators on the prediction of reading performance results under different circumstances.
When it comes to the data mining of machine learning, behavioral indicators in the reading process can make more accurate decisions for reading performance. The results are summarized in Figure 4, class 0/1 mean not correct/correct answers. Take a branch of T2's decision tree model as an example to explain in detail. Decision tree model of T2 is shown in the Figure 4, and a relatively high accuracy (acc=0.745) is obtained after three-fold cross-validation. As can be seen from the figure, if freq_rate_2_19 less than or equal with 0.004 and freq_rate_1_3 less than 99.0, it can judge that subject may get correct answer for T2. Further, if freq_rate_2_19 more than 0.004, and pagefreq_on_rate_2 less than or equal with 0.166, which means that answer for T2 has a great possibility that it is right correct. Therefore, from the decision tree, we can predict the possible answer of the subjects on the corresponding questions based on the indicators under different threshold conditions.
And then, in order to judge and identify the importance of indicators, random forest models were constructed in this study. The results are shown in the Figure 5. The importance of the index is calculated by using the Leave-One-Out Cross Validation method to construct the training set and the test set, and then obtain the importance of 32 random forest models and the corresponding indexes, and then take the average of the 32 importance, and use this as the basis Sort all the indicators and find the 10 most important indicators corresponding to the missing questions. Among them, the acc value representing the accuracy of the model is marked in Figure 5. From Figure 5, we can further get information as follows. Relatively speaking, the index of the title and the corresponding article content area is more important for prediction of reading performance. Take T4 for example, we can find that freq_rate_2_12 is the most important indicator for prediction of T4. Freq_rate_2_12 is the aera of T2, which means that students foucs on T2 may influence the performance of T4. So, we can further study the reson why cause this result. From this result, we can further get information about the indicators’ importance for each item.
Therefore, the more focus on text content related to topic, the more cognitive attention allocated on it, and that is to say, there may be confusions for participants; Relatively low fixation rate on the topic and text indicates that there is relatively little cognitive processing on the topic and there is no confusion. So, the behavioral data during reading can be used to predict the corresponding reading performance after learning analysis.
Based on the above analysis, we found that the more focus on content or topic, the worse reading performance would be, which is also supported by the existing studies. When it comes to the reason, the higher values of the indicators, such as freq_2_19, represent participants' cognitive processing of reading on the corresponding content, which means that there may be cognitive processing difficulties. Therefore, indicators with larger value associated with poorer reading performance. This is consistent with the conclusion of existing studies. For learners with weak reading skills, they will pay more attention on information processing, which can represent learners' efforts, and can also be indicators to reveal reading difficulties. Therefore, future studies can use our developed reading platform, which can collect eye-tracking data. And then, in order to improve reading performance, it needs to be considered that how to intervene and adjust from key eye-tracking indicators. The significance of this study is that the developed system can well capture the user's reading behavior data during reading process, which can be the basis of studying reading performance improvement, and quantify the influence relationship between behavior indicators and reading performance.
However, the study also has some shortcomings. Firstly, since the system is developed by research team members based on the open-source code and tools, the accuracy in operation is difficult to compare with the current mature system or eye-tracking tools. This system will be modified and improved in the future. In other words, the accurancy of this system need to be improve, which lead the results of this research to have a certain deviation, so we will improve the system. Secondly, due to limited human resources, the number of subjects is small, so there is less data. Therefore, it is difficult to conduct the high-precision machine learning data mining, and the precision of the decision tree and random forest models is poor. The sample size will be further increased in subsequent studies to get more valuable conclusions.
This research was planneed and implemented at East China Normal University, in China. The research is made available under a Research on the Model, Automatic Measurement and Intervention on Academic Well-being in Online Learning (21ZR1419100), which is supported by the 2021 Shanghai Science and Technology Plan. 7. References [20] RAYNER, K., Eye movements in reading and information processing : 20 years of research.
Psychological bulletin, 124(3) (1998) 372-422. doi: 0033-2909/98/$3.00. [21] Tóthová, M. and M. Rusek, Použití eye-trackingu v analýze učebnic pro přírodovědné předměty: přehledová studie. Scientia in educatione, 12(1) (2021) 63-74. doi: https://doi.org/10.14712/18047106.1932. [22] Davies, A., et al., Computational Methods for Analysis of Visual Behavior using Eye-tracking.
Measuring Behavior, (2016). [23] Giorgio, B., et al., Group vs Individual Web Accessibility Evaluations: Effects with Novice
Evaluators. Interacting with Computers, 28(6) (2016) 843-861. doi: 10.1093/iwc/iww006. [24] Scheiter, K. and A. Eitel, The Use of Eye Tracking as a Research and Instructional Tool in Multimedia Learning. In Advances in Educational Technologies and Instructional Design, Germany, IGI Global, 2016. 143-164. doi: 10.4018/978-1-5225-1005-5.ch008. [25] Zehui, Z., An Emotional and Cognitive Recognition Model for Distance learners Based on Intelligent Agent—— The Coupling of Eye Tracking and Expression Recognition Techniques.
Mod. Dist. Educ. Res, (5) (2013) 100-105. doi: 10. 3969 /j. issn. 1009 －5195. 2013. 05. 013. [26] Zehui, Z., L. Ting and M. Zicheng, Distance Learning Supporting Services Based on Virtual Assistant and Its Technical Difficulties. Modern Distance Education Research, (06) (2014) 95103+111. doi: 10.3969/j.issn.1009-5195.2014.06.011. [27] Isnin, I. and N.M. Jaafar, Reading Glossed Passages in English to Improve Comprehension: An Eye Tracking Study. GEMA Online Journal of Language Studies, 21(3) (2021) 125-139. doi: http://doi.org/10.17576/gema-2021-2103-07. [28] Haynes, M., READING IN A SECOND LANGUAGE: MOVING FROM THEORY TO PRACTICE. William Grabe. New York: Cambridge University Press, 2009. Pp. xv + 467. Studies in Second Language Acquisition, 32(4) (2010) 648-650. doi: https://doi.org/10.1017/S0272263110000355. [29] Sulaiman, N., K. Salehuddin and R. Khairudin, Reading English Academic Texts: Evidence from ESL Undergraduates’ Eye Movement Data. 3L The Southeast Asian Journal of English Language Studies, 26(1) (2020) 60-78. doi: http://doi.org/10.17576/3L-2020-2601-05. [30] Hyönä, J., R. Lorch and M. Rinck, Eye Movement Measures to Study Global Text Processing. The Mind's Eye: Cognitive and Applied Aspects of Eye Movement Research, North-Holland, Elsevier Science, 2003. doi: 10.1016/B978-044451020-4/50018-9. [31] Ishimaru, S. and A. Dengel, ARFLED: ability recognition framework for learning and education.
UbiComp '17: Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computer, (2017) 339-343. doi: https://doi.org/10.1145/3123024.3123200. [32] Papoutsaki, A., et al., WebGazer: Scalable Webcam Eye Tracking Using User Interactions.Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence - IJCAI 2016, AAAI Press / International Joint Conferences on Artificial Intelligence press, New York, USA, 2016 3839-3845. URL: http://ijcai-16.org/.