Validation of VARK questionnaire using gaze tracking data

Abstract

-We use gaze data (fixation time on Areas of Interest, AoIs) collected while reading educational materials to validate the VARK (Visual Auditory Reading Kinaesthetic) questionnaire. We analyse the dependencies between four types of AoIs (Title, Text, Graph, Formula) and the VARK scores for sensory modalities using correlation and linear regression analysis. Our results show significant correlations for Formula - Reading, Text - Visual, and Title - Kinaesthetic dependencies. The results of research can be used for objective evaluation of learning style of subjects using gaze tracking technology.

Keywords

• VARK
• learning styles
• gaze tracking
• multimedia

-

I. INTRODUCTION

Learning styles were defined to justify individual preferences and differences in learning and understanding [ 1 ]. Notable models of learning style include Kolb’s experiential learning, which introduces accommodators, convergers, divergers and assimilators [ 2 ]; Mumford’s model, which has activists, reflectors, theorists, and pragmatists [ 3 ]; Barbe et al. model, which considers auditory, visualising, and kinesthetic modalities [ 4 ], and Index of Learning Styles (ILS), which considers, active/reflective, sensing/intuitive, visual/verbal, and sequential/global learning [ 5 ]. Learning styles can be employed for user modelling, developing effective pedagogical guidelines, personalization of learning scenarios and materials, and increasing interactivity of presentation in multimedia-based e-learning systems. The usefulness of the learning styles were proven in various and diverse fields of education such as computer programming [ 6 ] and nursing [ 7 ]. Different tools have been used to evaluate learning styles such as Visual Auditory Reading Kinaesthetic (VARK) [ 8 ], Visual Auditory Kinaesthetic (VAK) [ 9 ] and Learning Style Questionnaire (LSQ) [ 10 ]. However, as the use of questionnaires as a research tool is prone to subjectiveness and difficulty of interpretation, and have been criticized for weak empirical evidence, no correlation with learning outcomes [ 11 ] and the lack of independent research on the model [ 12 ].

The objective evaluation methods were suggested to use electroencephalogram (EEG) [ 13, 14, 15 ] and electrocardiogram (ECG) signals acquired from the learners [ 16 ]. Here we analyse the use of gaze tracking data recorded while learners read learning materials to evaluate their learning styles. The idea in itself is not new as gaze tracking has been used previously in this context [ 17, 18, 19 ] while aiming to detect correlations between assimilation of different types of information and different parameters like learning style. We specifically focus on the validation of the VARK model, which proposed four types of learners: visual, auditory learning, textual and kinaesthetic. Our novelty is what we focus on the validity of the VARK questionnaire in itself and aim to confirm the VARK scores by gaze related © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) characteristics of subjects without analysing the differences in learning style and efficiency.

H3: R subjects prefer the T information.

H4: K subjects prefer the F information.

D. Testing of hypotheses

For testing of hypotheses we use the Pearson correlation:  (xi  X )( yi  Y ) s s x y   here xi , yi are the data values for which the dependency is tested, X,Y are means, sx , sy are standard deviations. The value of r  0 indicates a positive relationship of X and Y, and r  0 indicates a negative relationship.

The significance of the correlation value is calculated using the critical values of t-statistics as follows:    t  r n  2  1  r2 here n is the size of a sample. Given a small sample of n  5 in our case, the statistically significant ( p  0.05 ) correlation value must be at least r  0.86 .

We also construct the linear regression models between the dependent variables (T1, T2, G, F) and the independent variables (V, A, R, K). Linear regression is defined as:

Yi  0  1 Xi  i  here is the value of dependent variable, is the value of the independent variable for the i-th sample, 0 is the free coefficient, 1 is the slope, and is the random error. The sign of slope coefficient defines the direction of dependency (positive or negative), and the absolute value shows the strength of dependency.

The reliability of the linear regression model is evaluated using the significance of the coefficients (all must have p  0.05 ) , and the coefficient of determination r2 :   r 2  exp lained var iation total var iation 

SSR SST 0  r 2  1  

III. EXPERIMENTAL SETTING AND RESULTS

A. Experimental setting

Five participants (one female, four male) were recruited for this study, ages between 23 and 45 with an average of 29.8 years (SD = 8.66). All participants had normal or correctedto-normal vision. Participants were familiar with computers and had previous experience in using the internet. For each subject 7 slides that consisted of title, text, graph and formula were shown. Each slide was shown for 30 seconds interval, and the session took approximately 4 minutes. Subjects were instructed that they should try to memorize as much information as possible because at the end of the slide show a test will be taken. which consists of questions related to all different types of mathematical objects. For instance, to answer which formula, graph or text matches a given statement.

The Tobii 4C eye tracker was used to record eye movements of participants. The eye tracker uses infrared corneal reflection to measure point of gaze with data rates of 90 Hz. A 24 inch screen was used to show the slides. The eye tracker using instructions was mounted just below the visible screen area. The operating distance between the eye tracker and subjects’ eyes was between 70-75 cm. For each subject the eye tracker was re-calibrated using a 5-point calibration to achieve most accurate results. Gaze monitoring system was used to measure the number and duration of fixations in the Areas of Interest (AOIs). The system consists of components listed below (see Fig. 1):

The stimulus was the educational materials from the “Mathematics 1” course delivered to the 1st year Bachelor students at Kaunas University of Technology. The topic of the educational materials was the integral calculus. Structurally arranged as a set of PowerPoint (Microsoft, USA) slides, each slide representing a learning unit had four components: Title, Text, Formula and Graph (see Fig. 2).

Fig. 2. Areas of Interest (AoI) in learning material.

This study examined visual attention and the reading behaviour of the subjects. Each participant took the VARK Questionnaire for the assessment of learning styles. Then the participants we asked to complete a calibration session followed by launching the learning material slides in full screen mode. Following that, participants were asked to read the slides presented at the computer screen. During the experiment, the eye tracker measured the learner's eye movements such as eye fixations and fixation durations. After completing the reading component, a knowledge assessment test was administered to participants on screen. The results of the knowledge evaluation test were not used in this study, as the aim was to motivate the participates to read attentively rather than evaluating their knowledge gained on the subject.

B. Results

The results of gaze time spent on each AoI are summarized in Fig. 3: most time (~35%) was spent on text, while least time (~6%) on the title of the learning material.

Fig. 3. Example of a figure caption. (figure caption)

The summary of the VARK scores are presented in Figure 4. On average, the highest score was assigned to Visual type (9.2), while the lowest score was assigned to Aural type (4.2).

We performed the correlation analysis on the ratio of time spent on the Title (T2), Text (T1), Graph (G) and Formula (F) AoIs vs the Visual (V), Aural (A), Read/Write (R) and Kinesthetic (K) scores from the VARK questionnaire. The results are presented in Fig. 5. We found significant correlations for Title ↔ Kinaesthetic ( r  0.96 ), Text ↔ Visual ( r  0.94 ), and Formula ↔ Read/Write ( r  0.93 ). We did not find any significant correlations for the A modality thus confirming the H2 hypothesis. We could not confirm the H1 hypothesis, however the results show that V subjects strongly do not prefer T information. We also could not confirm the H3 hypothesis, but we found that R subjects prefer F information. We also could not confirm the H3 hypothesis, but the results show that K subjects prefer T information.

We also explored more different types of relationship and analysed the dependencies between the grouped dependent variables (T1+T2, T1+G, T1+F, T2+G, T2+F, G+F) and independent variables (V, A, R and K). The results presented in Fig. 6. The significant correlations were found only for Title + Formula ↔ Kinaesthetic ( r  0.86 ), and Text + Graph ↔ Kinaesthetic ( r  0.86 ).

Four linear regression models were constructed for each of the V, A, R and K modality scores as dependent variables and the Title (T2), Text (T1), Graph (G) and Formula (F) AoIs as independent variables (see a summary presented in Fig. 7). All models are reliable ( p  0.001 for all coefficients and r2  0.99 for all models). When considering the value of slope coefficient, the V modality is mostly influenced by Title (39.5, positively) and Graph (28.8, positively), the A modality is mostly influenced by Title (-74.8, negatively), the R modality is mostly influenced by Formula (94.7, positively) and Title (-65.1, negatively), and the K modality is mostly influenced by Title (71.1, positively).

Fig. 7. Summary of V, A, R and K linear regression models

We also constructed the inverse linear regression models were constructed for the Title (T2), Text (T1), Graph (G) and Formula (F) AoIs as independent variables and the V, A, R and K modality scores as dependent variables (see a summary presented in Fig. 8). In this case, only one model for Title was reliable ( < 0.001 for all coefficients and 2 > 0.99). When considering the value of the slope coefficient, the time spent on Title AoI is mostly influenced by the K modality (0.017, positively), which agrees with the corresponding linear regression model for the K modality presented in Fig. 9.

Finally, we evaluate how much of variance in the data for the sensory modalities is explained by the variance in the AoI (Fig. 9) and vice versa (Fig. 10). We can see that the V modality is most influenced by the Formula (+51%) and Text (-24%) AoIs. The A modality is most influenced by the Text (+35%) and Formula (+31%) AoIs. The R modality is most influenced by the Title (+35%) and Graph (-32%) AoIs. The K modality is most influenced by the Title (+29%) and Formula (+27%) AoIs.

The attention on the Title AoI is most influenced by the V (+38%) and K (+30%) modalities. The attention on the Text AoI is most influenced by the V (-52%) and K (+30%) modalities. The attention on the Graph AoI is most influenced by the K (+57%) and Title (-29%) modalities. The attention on the Formula AoI is most influenced by the V (+75%) and K (+20%) modalities.

Our findings are in line with Al-Wabil et al. [ 21 ], who analysed Index of Learning Styles (ILS) using gaze tracking, also found that verbal learners pay attention to textual content more than multimedia, and visual learners scan the text and direct more attention to multimedia elements than textual content. Hoffler et al. [ 22 ] analysed the Object-Spatial Imagery and Verbal Questionnaire (OSIVQ) and found significant correlations between dwell time and the object and spatial visualizers, while no correlation was found for verbalizers. Our results confirm common knowledge, such as Visual subjects do not like Text but do like Graphs, however also provide interesting insights such as Kinaesthetic subjects liking Titles, which represent a condensed (‘tangible’) form of information, and Visual subjects liking Formulas, which although are a form of mathematical notation, yet share many similarities to the visual representation of information.

D. Threats to validity

A small sample of subjects and biased selection of participants (all subjects have a strong background in computer science) may render the results of our study as less reliable. Furthermore, the factors of stress, emotion and gender have not been accounted for in this study, although our previous research has demonstrated their significant influence on gaze characteristics [ 23, 24, 25 ]. Also note that the types of the AoIs analysed can not be separated strictly: in some cases text and graphs also contained elements of mathematical notations such as the names of variables.

IV. CONCLUSIONS

Our results demonstrate significant positive correlation between the attention on the Title Area of Interest (AoI) and the Kinaesthetic sensory modality ( r  0.96 ), significant negative correlation between the Text AoI and Visual modality ( r  0.94 ), and significant positive correlation between the Formula AoI and the Read/Write modality ( r  0.93 ). The linear regression models show the importance of Titles for the Visual, Aural and Kinaesthetic modalities and the importance of Formula for the Read/Write modality. The inverse linear regression model shows the significant attention of the Visual modality to Titles. The latter is confirmed by the variance analysis, which shows that Visual subjects prefer Formulas and dislike Text, Aural subjects like Text and Formulas, Read/Write subjects like Titles and dislike Graphs, and Kinaesthetic subjects like Titles and Formulas. Our results show that there is a possibility for the VARK questionnaire to be another valid tool to analyze cognitive types of subjects. The gaze tracking data could possibly provide valuable objective information and insights on the cognitive preference of subjects that might possibly supplement the results of the subjective questionnaire. Future work will focus on collecting a larger dataset of gaze tracking data and extending the experiment to a more diverse set of AoIs.

References

[1] Pashler , H. , McDaniel , M. , Rohrer , D. , & Bjork , R.A. ( 2008 ). Learning styles: concepts and evidence . Psychological Science in the Public Interest . 9 ( 3 ): 105 - 119 .

[2] Kolb , D.A. ( 2015 ). Experiential learning: experience as the source of learning and development (2nd ed.). Pearson Education.

[3] Mumford , A. ( 1997 ). Putting learning styles to work. Action learning at work . Aldershot, Hampshire, Brookfield, VT : Gower. 121 - 135 .

[4] Barbe , W.B. , Swassing , R.H. , & Milone , M.N. ( 1979 ). Teaching through modality strengths: concepts practices . Zaner-Bloser.

[5] Felder , R.M. , & Silverman , L.K. ( 1988 ). Learning and Teaching Styles in Engineering Education . Engr. Education , 78 ( 7 ), 674 - 681 .

[6] Abbott , M. R. B. , & Shaw , P. ( 2018 ). Multiple modalities for APA instruction: Addressing diverse learning styles . Teaching and Learning in Nursing , 13 ( 1 ), 63 - 65 . doi: 10 .1016/j.teln. 2017 . 08 .004

[7] Diaz , F.S. , Rubilar , T.P. , Figueroa , C.C. , & Silva , R.M. ( 2018 ). An adaptive E-learning platform with VARK learning styles to support the learning of object orientation . 2nd IEEE World Engineering Education Conference, EDUNINE 2018 , 1 - 6 . doi: 10 .1109/EDUNINE. 2018 .8450990

[8] Prithishkumar , I. J. , & Michael , S. A. ( 2014 ). Understanding your student: Using the VARK model . Journal of Postgraduate Medicine , 60 ( 2 ), 183 - 186 . doi: 10 .4103/ 0022 - 3859 . 132337

[9] Apipah , S. , Kartono , & Isnarto ( 2018 ). An analysis of mathematical connection ability based on student learning style on visualization auditory kinesthetic (VAK) learning model with self-assessment . Journal of Physics: Conference Series , 983 ( 1 ). doi: 10 .1088/ 1742 - 6596/983/1/012138

[10] Kappe , F. R. , Boekholt , L. , den Rooyen , C. , & Van der Flier , H. ( 2009 ). A predictive validity study of the learning style questionnaire (LSQ) using multiple, specific learning criteria . Learning and Individual Differences , 19 ( 4 ), 464 - 467 . doi: 10 .1016/j.lindif. 2009 . 04 .001

[11] Husmann , P. R. , & O'Loughlin , V. D. ( 2019 ). Another nail in the coffin for learning styles? disparities among undergraduate anatomy students' study strategies, class performance, and reported VARK learning styles . Anatomical Sciences Education , 12 ( 1 ), 6 - 19 . doi: 10 .1002/ase.1777

[12] Coffield , F. , Moseley , D. , Hall , E. , & Ecclestone , K. ( 2004 ). Learning styles and pedagogy in post-16 learning: a systematic and critical review . London, England: Learning & Skills Research Centre.

[13] Thepsatitporn , S. , & Pichitpornchai , C. ( 2016 ). Visual event-related potential studies supporting the validity of VARK learning styles' visual and read/write learners . Advances in Physiology Education , 40 ( 2 ), 206 - 212 . doi: 10 .1152/advan.00081.2015

[14] Jawed , S. , Amin , H. U. , Malik , A. S. , & Faye , I. ( 2018 ). Differentiating between visual and non-visual learners using EEG power spectrum entropy . International Conference on Intelligent and Advanced System , ICIAS 2018 , doi:10.1109/ICIAS. 2018 .8540571

[15] Alhasan , K. , Chen , L. , & Chen , F. ( 2018 ). Mining learning styles for personalised elearning . 2018 IEEE SmartWorld, Ubiquitous Intelligence and Computing , Advanced and Trusted Computing , Scalable Computing and Communications, Cloud and Big Data Computing, Internet of People and Smart City Innovations , SmartWorld/UIC/ATC/ScalCom/CBDCom/IoP/SCI 2018 , 1175 - 1180 . doi: 10 .1109/SmartWorld. 2018 .00204

[16] Granero-Molina , J. , Fernández-Sola , C. , López-Domene , E. , Hernández-Padilla , J. M. , Romão Preto , L. S. , & Castro-Sánchez , A. M. ( 2015 ). Effects of web-based electrocardiography simulation on strategies and learning styles . Revista Da Escola De Enfermagem , 49 ( 4 ), 645 - 651 . doi: 10 .1590/S0080-623420150000400016

[17] Mehigan , T. J. , Barry , M. , Kehoe , A. , & Pitt , I. ( 2011 ). Using eye tracking technology to identify visual and verbal learners . IEEE International Conference on Multimedia and Expo , 1 - 6 . doi: 10 .1109/ICME. 2011 .6012036

[18] Koć-Januchta , M. , Höffler , T. , Thoma , G. , Prechtl , H. , & Leutner , D. ( 2017 ). Visualizers versus verbalizers: Effects of cognitive style on learning with texts and pictures - an eye-tracking study . Computers in Human Behavior , 68 , 170 - 179 . doi: 10 .1016/j.chb. 2016 . 11 .028

[19] Winoto , P. , Tang , T. Y., Huang , Z. , & Chen , P. ( 2017 ). “Thinking in pictures?” performance of chinese children with autism on math learning through eye-tracking technology . In: Zaphiris P., Ioannou A . ( eds) Learning and Collaboration Technologies . Technology in Education. LCT 2017. Lecture Notes in Computer Science , vol 10296 . Springer, Cham, 215 - 226 . doi: 10 .1007/978-3- 319 -58515-4_ 17

[20] Yu , X. ( 2015 ). Literature Review of Applying Visual Method to Understand Mathematics . MATEC Web of Conferences , 22 , 1063. doi: 10 .1051/matecconf/20152201063

[21] Al-Wabil , A. , ElGibreen , H., George , R. P. , & Al-Dosary , B. ( 2010 ). Exploring the validity of learning styles as personalization parameters in elearning environments: An eyetracking study . ICCTD 2010 - 2010 2nd International Conference on Computer Technology and Development , 174 - 178 . doi: 10 .1109/ICCTD. 2010 .5646127

[22] Höffler , T. N. , Koć-Januchta , M. , & Leutner , D. ( 2017 ). More evidence for three types of cognitive style: Validating the object-spatial imagery and verbal questionnaire using eye tracking when learning with texts and pictures . Applied Cognitive Psychology, 31 ( 1 ), 109 - 115 . doi: 10 .1002/acp.3300

[23] Vasiljevas , M. , Gedminas , T. , Ševčenko , A. , Jančiukas , M. , Blažauskas , T. , & Damaševičius , R. ( 2016 ). Modelling eye fatigue in gaze spelling task . 2016 IEEE 12th International Conference on Intelligent Computer Communication and Processing , ICCP 2016 , 95 - 102 . doi: 10 .1109/ICCP. 2016 .7737129

[24] Raudonis , V. , Maskeliūnas , R. , Stankevičius , K. , & Damaševičius , R. ( 2017 ). Gender, age, colour, position and stress: How they influence attention at workplace? Computational Science and Its Applications - ICCSA 2017. Lecture Notes in Computer Science , 10408 . Springer, Cham, 248 - 264 . doi: 10 .1007/978-3- 319 -62404-4_ 19

[25] Liaudanskaitė , G. , Saulytė , G. , Jakutavičius , J. , Vaičiukynaitė , E. , Zailskaitė-Jakštė , L. , & Damaševičius , R. ( 2019 ). Analysis of affective and gender factors in image comprehension of visual advertisement . Artificial Intelligence and Algorithms in Intelligent Systems. CSOC2018. Advances in Intelligent Systems and Computing , vol 764 . Springer, Cham, 1 - 11 . doi: 10 .1007/978-3- 319 -91189- 2 _ 1