Non-cognitive factors of learning as predictors of academic performance in tertiary education

Abstract

This paper reports on an application of classi cation and regression models to identify college students at risk of failing in rst year of study. Data was gathered from three student cohorts in the academic years 2010 through 2012 (n=1207). Students were sampled from fourteen academic courses in ve disciplines, and were diverse in their academic backgrounds and abilities. Metrics used included noncognitive psychometric indicators that can be assessed in the early stages after enrolment, speci cally factors of personality, motivation, self regulation and approaches to learning. Models were trained on students from the 2010 and 2011 cohorts, and tested on students from the 2012 cohort. Is was found that classi cation models identifying students at risk of failing had good predictive accuracy (> 79%) on courses that had a signi cant proportion of high risk students (over 30%).

Keywords

• eol>Educational data mining
• learning analytics
• academic performance
• non cognitive factors of learning
• personality
• motivation
• learning style
• learning approach
• self-regulation

1. INTRODUCTION AND LITERATURE RE

VIEW

Learning is a latent variable, typically measured as academic performance in continuous assessment and end of term examinations [ 33 ]. Identifying predictors of academic performance has been the focus of research for many years [ 20, 34 ], and continues as an active research topic [ 6, 8 ], indicating the inherent di culty in generating models of learning [ 29, 46 ]. More recently, the application of data mining to educational settings is emerging as an evolving and growing research discipline [ 40, 43 ]. Educational Data Mining (EDM) aims to better understand students and how they learn through the use of data analytics on educational data [ 42, 10 ]. Much of the published work to date is based on everincreasing volumes of data systematically gathered by education providers, particularly log data from Virtual Learning Environments and Intelligent tutoring systems [ 16, 2 ]. Further work is needed to determine if gathering additional predictors of academic performance can add value to existing models of learning.

Research from educational psychology has identi ed a range of non-cognitive psychometric factors that are directly or indirectly related to academic performance in tertiary education, particularly factors of personality, motivation, self regulation and approaches to learning [ 8, 9, 35, 39, 44, 25 ]. Personality based studies have focused on the Big-5 personality dimensions of conscientiousness, openness, extroversion, stability and agreeableness [ 9, 22, 27 ]. There is broad agreement that conscientiousness is the best personality based predictor of academic performance [ 44 ]. For example, Chamorro et al. [ 9 ] reported a correlation of r=0.37 (p<0.01, n=158) between conscientiousness and academic performance. Correlations between academic performance and openness to new ideas, feelings and imagination are weaker. Chamorro et al. [ 9 ] reported a correlation of r=0.21 (p<0.01, n=158) but lower correlations were reported in other studies (see Table 1) which may be explained by variations in assessment type. Open personalities tend to do better when assessment methods are unconstrained by submission rules and deadlines [ 27 ]. Studies are inconclusive on the predictive validity of other personality factors [ 44 ]. A meta-analysis of 109 studies analysing psychosocial and study skill factors found two factors of motivation, namely self-e cacy (90% CI [0.444,0.548]) and achievement motivation (90% CI [0.353, 0.424]), had the highest correlations with academic performance [ 39 ]. Distinguishing between learning (intrinsic) achievement and performance (extrinsic) achievement goals, Eppler and Harju [ 19 ] found learning goals (r=0.3, p<0.001, n=212) were more strongly correlated with academic performance than performance goals (r=0.13, p> 0.05, n=212). Covington [ 13 ] however argues that setting goals in itself is not enough, as ability to selfregulate learning can be the di erence between achieving, or not achieving, goals set. Self-regulated learning is recognised as a complex concept to de ne as it overlaps with a number of other concepts including personality, self-e cacy and goal setting [ 4 ]. Ning and Downing [ 35 ] reported high correlations between self regulation and academic performance, speci cally self-testing (r=0.48, p<0.001) and monitoring understanding (r= 0.42, p<0.001). On the other hand, Komarraju and Nadler [ 31 ] found e ort management, including persistence, had higher correlation with academic performance (r=0.39, p<0.01) than other factors of self-regulation and found that self-regulation (monitoring and evaluating learning) did not account for any additional variance in academic performance over and above self-e cacy, but study e ort and study time did account for additional variance. Research into approaches to learning has its foundations in the work of Marton & Saljo [ 32 ] who classi ed learners as shallow or deep. Deep learners aim to understand content, while shallow learners aim to memorise content regardless of their level of understanding. Later studies added strategic learners [18, pg. 19], whose priority is to do well, and will adopt either a shallow or deep learning approach depending on the requisites for academic success. Comparing the in uence of approaches to learning on academic performance, Chamorro et al [ 9 ] reported a deep learning approach (r=0.33, p<0.01) had higher correlations with academic performance than a strategic learning approach (r=0.18, p<0.05). Cassidy [ 8 ] on the other hand found correlations with a deep learning approach (r=0.31, p<0.01) were marginally lower than with a strategic learning approach (r=0.32, p<0.01). Di erences found have been explained, in part, by assessment type [ 49 ], highlighting the importance of assessment design in encouraging appropriate learning strategies. Knight, Buckingham Shum and Littleton argued learning measurement should go beyond measures of academic performance [ 29 ], promoting greater focus on learning environment and encouragement of malleable, e ective learning dispositions. Disposition relates to a tendency to behave in a certain way [ 6 ]. An e ective learning disposition describes attributes and behaviour characteristic of a good learner [ 6 ]. A range of non-cognitive psychometric factors have been associated with an e ective learning disposition such as a deep learning approach, ability to self-regulate, setting learning goals, persistence, conscientiousness and subfactors of openness, namely intellectual curiosity, creativity and open-mindednesss [ 6, 29, 47 ]. A lack of correlation between such non-cognitive factors and academic performance is in itself insightful, suggesting assessment design that fails to reward important learning dispositions. It has been argued that e ective learning dispositions are as important as discipline speci c knowledge [ 6, 29 ].

Statistical models have dominated data analysis in educational psychology [ 15 ], particularly correlation and regression [ 25 ]. Relatively high levels of accuracy were reported in regression models of academic performance that included cognitive and non-cognitive factors. For example, ChamorroPremuzic et al [ 9 ] reported a coe cient of determination (R2) of 0.4 when predicting 2nd year GPA (based on essay type examinations) in a regression model that included prior academic ability, personality factors and a deep learning approach. Robbins [ 39 ] reported similar results (R2=0.34) in a meta-analysis of models of cognitive ability, motivation factors and socio-economic status. Models of non-standard students were less accurate, for example Swanberg & Martinsen [ 44 ] reported R2=0.21 in models of older students (age: m=24.8) based on prior academic performance, personality, learning strategy, age and gender. Lower accuracies were also reported in studies not including cognitive ability. Robbins [ 39 ] reported R2=0.27 in a meta-analysis of models of factors of motivation. Komarraju et al. [ 30 ] predicted GPA (R2=0.15) from variables of personality and learning approach, while Bidjerano & Dai [ 4 ] had similar results (R2=0.11) with factors of personality and self-regulation. Linear regression assumes constant variance and linearity between independent and dependent attributes. There is evidence to suggest variance is not constant for some noncognitive factors. For example, De Feyter et al. [ 14 ] found low levels of self-e cacy had a positive, direct e ect on academic performance for neurotic students, and for stable students, average or higher levels of self-e cacy only had a direct e ect on academic performance. In addition, Vancouver & Kendall [ 48 ] found evidence that high levels of self-e cacy can lead to overcon dence regarding exam preparedness, which in turn can have a negative impact on academic performance. Similarly, Poropat [ 38 ] cites evidence of non-linear relationships between factors of personality and academic performance, including conscientiousness and openness. It is therefore pertinent to ask if data mining's empirical modelling approach is more appropriate for models based on non-cognitive factors of learning.

A growing number of educational data mining studies have investigated the role of non-cognitive factors in models of learning [ 6, 41, 36 ]. Bergin [ 3 ] cited an accuracy of 82% using an ensemble model based on prior academic achievement, self-e cacy and study hours, but due to the small sample size (n=58) could not draw reliable conclusions from the ndings. The class label distinguished strong (grade>55%) versus weak (grade<55%) academic performance based on end of term results in a single module. Gray et al. [ 23 ] cited similar accuracies (81%, n=350) with a Support Vector Machine model using cognitive and non cognitive attributes to distinguish high risk (GPA<2.0) from low risk (GPA 2.5) students based on rst year GPA. Model accuracy was contingent on modelling younger students (under 21) and older students (over 21) separately.

The focus of this study was to investigate if non-cognitive factors of learning, measured during rst year student induction, were predictive of academic performance at the end of rst year of study. We evaluated both regression models of GPA and classi cation models that predicted rst year students at risk of failing. Participants were from a diverse student population that included mature students, students with disabilities, and students from disadvantaged socio-economic backgrounds.

2. METHODOLOGY

The following sections report on study participants and the study dataset. Data analysis was conducted following the CRoss Industry Standard for Data Mining (CRISP-DM) using RapidMiner V5.3 and R V3.0.2.

2.1 Description of the study participants

The participants were rst year students at the Institute of Technology Blanchardstown (ITB), Ireland. The admission policy at ITB supports the integration of a diverse student population in terms of age, disability and socio-economic background. Each September 2010 to 2012, all full-time, rst-year students at ITB were invited to participate in the study by completing an online questionnaire administered during rst year student induction. A total of 1,376 (52%) full-time, rst year students completed the online questionnaire. Eliminating students who did not give permission to be included in the study (35) and invalid data (134) resulted in 45% of rst year full time students participating in the study (n=1207).

Participants ranged in age from 18 to 60, with an average age of 23.27; of which, 355 (29%) were mature students (over 23), 713 (59%) were male and 494 (41%) were female. There were 32 (3%) participants registered with a disability. Students were enrolled on fourteen courses across ve academic disciplines, Business (n=402, 33%), Humanities (n=353, 29%), Computing (n=239, 20%), Engineering (n=172, 14%) and Horticulture (n=41, 3%).

Academic performance was measured as GPA, an aggregate score of between 10 and 12 rst year modules, range 0 to 4, and was calculated on rst exam sitting only. The GPA distribution (pro led sample) was compared with the GPA distribution of the full cohort of students for that year (reference sample) using a Kolmogorov-Smirnov nonparametric test. The recorded di erences in the distribution for 2010 (D=0.032, p=0.93), 2011 (D=0.036, p=0.90) and 2012 (D=0.042, p=0.69) were not statistically signi cant. The distribution of GPA was also similar across the three years of study. The largest di erence was between the 2010 and 2012 pro led samples (D=0.063, p=0.37) and was not signi cant. To pass overall, a student must achieve a GPA 2.0 and pass each rst year module. 89% of students with GPA > 2.5 passed all modules indication a low risk group that can progress to year two. 84% of students with a GPA < 2 failed three or more modules, indicating a high risk group falling well short of progression requirements. Of the students in GPA range [2.0, 2.49], 39% passed all modules, 36% failed one module, 18% failed two modules, and 7 % failed more than two modules. This is a less homogenous group in terms of academic pro le, but could be generally regarded as borderline, either progressing on low grades or required to repeat one or two modules in the repeat exam sittings. Figure 1 and Table 2 illustrate GPA distribution by course.

2.2 The Study Dataset

Table 3 lists the psychometric factors included in the dataset, collected using an online questionnaire developed for the study (www.howilearn.ie). With the exception of learning modality, questions were taken from openly available, validated instruments, with some changes to wording to suit the context. Where two questions were similar on the published instrument, only one was included. This choice was made to reduce the overall size of the questionnaire, despite the likely negative impact on internal reliability statistics. Questionnaire validity and internal reliability were assessed using a paper-based questionnaire that included both the revised wording of questions used on the online questionnaire (reduced scale), and the original questions from the published instruments (original scale). The paper questionnaire was administered during scheduled rst year lectures across all academic disciplines. Pearson correlations between scores calculated from the reduced scale, and scores calculated from the original scale, were high for all factors (>=0.9) except intrinsic goal orientation and study time and environment, con rming the validity of the study instrument for those factors. Internal reliability was assessed using Cronbach's alpha. All factors had acceptable reliability (>0.7)1 given the small number of questions per scale (between 3 and 6), with the exception again of intrinsic goal orientation and study time and environment. Learner modality data (Visual, Auditory, Kinaesthetic (VAK) [ 21 ]) was based an instrument developed by the National Leaning Network Assessment Services (NLN) (www.nln.ie). 1While generally a Cronbach alpha of > 0.8 indicates good internal consistency, Cronbach alpha closer to 0.7 can be regarded as acceptable for scales with fewer items [ 12, 45 ]. Prior knowledge of the student available to the college at registration, namely age, gender and prior academic performance, was also available to the study. Access to full time college courses in Ireland is based on academic achievement in the Leaving Certi cate, a set of state exams at the end of secondary school. College places are o ered based on CAO2 points, an aggregate score of grades achieved in a student's top six leaving certi cate subjects, range 0 to 600. Table 4 summarises participant pro le by course.

3. RESULTS

Correlation and regression were used to analyse relationships between study factors and GPA. Subsequent analysis used classi cation techniques to identify students at risk of failing. Unless otherwise stated, models are based age, gender and non-cognitive factors of learning as listed in Table 3. All non-cognitive factors of learning failed the Shapiro Wilk normality test which is common in data relating to education and psychology [ 26 ]. However factors of personality were normally distributed within each discipline except for business. Intrinsic motivation and study e ort were also normally distributed for engineering and computing students. There were further improvements when analysing subgroups by academic course. Factors of personality, self regulation and intrinsic motivation were normally distributed for all courses. With the exception of approaches to learning, learner modality, preference for group work and GPA, other factors were normally distributed for most courses. Table 4 illustrates the number of attributes that di ered signi cantly from a normal distribution by course. Larger groups were more likely to fail tests of normality.

3.1 Correlations with Academic Performance

Correlations between study factors and GPA were assessed using Pearson's product-moment correlation coe cient (PPMCC). As some attributes violated the assumption of normal distribution, signi cance was veri ed with bootstrapped 2CAO refers to the Central Applications O ce with responsibility for processing applications for undergraduate courses in the Higher Education Institutes in Ireland.

Category & Instrument Study Factor Personality: IPIP scales Conscientiousness (5.9 1.5) (ipip.ori.org) [ 22 ] Openness (6.1 1.3) Motivation: Intrinsic Goal Orientation (7.1 1.4) MSLQ [ 37 ] Self E cacy (6.9 1.4)

Extrinsic Goal Orientation (7.8 1.4) Learning approach: Deep Learner (5.4 2.9) R-SPQ-2F [ 5 ] Shallow Learner (1.3 1.9)

Strategic Learner (3.4 2.5) Self-regulation: Self Regulation (5.9 1.4) MSLQ [ 37 ] Study E ort (5.9 1.8)

Study Time & Environment (6.2 2.3) Learner modality: Visual (7.2 2.1) NLN pro ler Auditory(3.3 2.2)

Kinaesthetic(4.5 2.4) Other factors: Preference for group work (6.5 3.4) Age (23.27 7.3)

Male=713 (59%), Female=494 (41%)

Note: All ranges are 0 to 10 apart from age. 95% con dence intervals using the bias corrected and accelerated method [ 7 ] on 1999 bootstrap iterations.

Bootstrap correlation coe cients are given in Table 5. With the exception of learning modality, all non-cognitive factors were signi cantly correlated with GPA. The highest correlations with GPA were found for approaches to learning, speci cally deep learning approach (r=0.23, bootstrap 95% CI[0.18, 0.29]), and study e ort (r=0.19, bootstrap 95% CI [0.13, 0.24] ). Age also had a relatively high correlation with GPA (r=0.25, bootstrap 95% CI [0.19, 0.3]). A shallow learning approach (r=-0.15, bootstrap 95% CI[-0.21, -0.09]) and preference for group work (r=-0.076, bootstrap 95% CI [-0.14, -0.02]) were negatively correlated with GPA. Openness had one of the weakest signi cant correlations with GPA (r=0.08, bootstrap 95% CI [0.03, 0.14]). Correlations were comparable with other studies that included a diverse student population [ 4, 9, 28 ] with the exception of self efcacy (r=0.12, bootstrap 95% CI [0.06, 0.17])) which was lower than expected. This may be re ective of the low entry requirements for some courses.

3.2 Regression models

Regression models predicting GPA from non-cognitive variables were run for the full dataset and for subgroups by disciplines and by course. The coe cient of determination (R2) is reported to facilitate comparison with other studies. However R2 is in uenced by the variability of the underlying independent variables. Consequently Achen [1, pg 58-61] argued that prediction error is a more appropriate tness measure for psychometric data. Therefore absolute error mean and standard deviation is also reported. A regression model for all participants (R2 = 0.14) was comparable with other reported models of non-cognitive factors [ 4, 30 ]. However when modelling students by discipline and by course, there were signi cant di erences in model performance. A chow test [ 11 ] comparing the residual error in a regression model of all participants (full model) with the residual errors of models by discipline (restricted models) showed signi cant di erences between the full and restricted models (F(17,1098)=22.02, p=0). There was also signi cant di erences between models based on a particular discipline (full model) and models of courses within that discipline (restricted models). In computing, signi cant di erences of F(17,205)=2.22 (p=0.005) were found between the full model and the two restricted models. Within engineering, a model combining mechatronics with electronic & computing engineering was not signi cantly di erent from a model of those two courses individually (F(17,79)=0.58, p=0.89), but including either common entry students and/or sustainable electrical & control technology resulted in signi cant di erences between the full and restricted models. Sustainable electrical & control technology was therefore excluded from further consideration because of the small sample size (n=20). Signi cant di erences were also found in models of each of the three humanities courses compared with those courses combined (F(17,302)=2.22, p=0.004). The least signi cant di erences were found in models of business students provided sport management was excluded (F(17, 307)=1.95, p=0.015). Adding sports management further increased the di erence in model residual errors (F(17,334)=8.36, p=0). Table 6 gives model details by course and factors used in each model. Electronic & computer engineering students and mechatronic students were combined.

In general, models based on technical courses had a higher R2 than models for non technical courses. For example, engineering courses, computing (IT) and business with IT all had R2 > 0.3. Absolute error for these courses was in the range [0.63,0.8]. The di erence between the highest absolute error (m=0.8, s=0.563) and the lowest absolute error (m=0.63, s=0.54) was not signi cant (t(15)=1.74, p=0.1). Regression results for International Business was also relatively good (R2=0.27). For the remaining non-technical disciplines R2 was lower (range [0.12,0.17]) but the absolute error was more varied. Early childcare had the lowest absolute error (m=0.37, s=0.34) while general business had the highest absolute error (m=0.9, s=0.53). The di erence was signi cant (t(15)=10.3, p<0.001) and may be explained by the greater distribution of GPA scores in general business. There was little agreement across models on which study 3m=mean, s=standard deviation factors were most predictive of GPA. Approaches to learning and age were signi cant for models of all participants, computing students and engineering students, but motivation and learning strategy were more signi cant for Business with IT. Factors of motivation, learning strategy and approaches to learning were also relevant to models in the humanities courses. All regression models improved when prior academic performance was included in the model. The most signi cant increase was for sports management, R2 increased from 0.16 to 0.30. Business with IT and applied social care also increased by more than 0.1. For all other regression models, R2 increased by between 0.05 and 0.09

3.3 Classification models

Classi cation models were generated using four classi cation algorithms, namely Nave Bayes (NB), Decision Tree (DT), Support Vector Machine (SVM), and k-Nearest Neighbour (k-NN). A binary class label was used based on end of year GPA score, range [ 0-4 ]. The two classes were: high risk students (GPA<2, n=459); and low risk students (GPA 2.5, n=558) giving a dataset of n=1017. Borderline students (2.0

GPA 2.49) have not been considered to date. Gray et al. [ 24 ] found that cross validation over-estimated model accuracy compared to models applied to a di erent student cohort. Therefore models were trained on participants from 2010 and 2011 and tested on participants from 2012. All datasets were balanced by over sampling the minority class, and attributes were scaled to have a mean of 0 and standard deviation of 1. Signi cant attributes were identi ed by nding the optimal threshold for selecting attributes by weight. Attributes were weighted based on uncertainty4 for DT, kNN and Nave Bayes models, and based on SVM weights for SVM models. Table 6 shows the accuracies achieved and factors used in each model. k-NN had the highest accuracy for models of all students (66%). Accuracies for DT (61%), SVM (62%) and Nave Bayes (62%) were similar. The most signi cance attributes by weight were age, deep learning approach and study e ort. Including factors of prior academic performance improved model accuracy marginally to 72%.

Model accuracy improved when modelling each course separately. In general, k-NN had either the highest accuracy, or close to the highest accuracy, for all groups with the exception of two courses, international business and early childcare & education. Nave Bayes had the highest accuracy for both those courses and their attributes of significance were normally distributed. Five courses had accuracies marginally higher than the model for all students, social & community development (70%), applied social care (68%), early childcare & education (69%), creative digital media (67%) and sports management (70%). As illustrated in Table 1, these courses were distinguished by a high average GPA and a low failure rate. Consequently, patterns identifying high risk students may be under represented in these groups. Accuracies for other courses were signi cantly higher ( 79%). For example the di erence between sports management (70%) and the next highest accuracy (Engineering other, 79%) was signi cant (Z=5.86, p<0.001)5. 4Symmetrical uncertainty with respect to the class label. 5Accuracy comparisons were based on the mean accuracy of It could be argued that the smaller sample size of course groups over estimated model accuracy as smaller samples may under represent the complexity of patterns predictive of academic achievement. Therefore 30 samples randomly generated from the full dataset (n=100) were also modelled. Model accuracy for the random samples was normally distributed, with mean=63.12% (s=11%), which was marginally lower than the model of all students (Z=2.68, p=0.017).

There was little agreement across models on which study factors were most predictive of high risk and low risk students. Conscientiousness, study e ort and a shallow learning approach were used most frequently, followed by openness, intrinsic motivation and age. There was no signi cant improvement in model accuracy when prior academic performance was included in each model. For example, the largest increase in accuracy was from 79% to 82% in a model of Engineering students.

4. CONCLUSIONS

Results from this study suggest that models of academic performance, based on non-cognitive psychometric factors measured during rst year student induction, can achieve good predictive accuracy, particularly when individual courses are modelled separately. A deep learning approach, study e ort and age had the highest correlations with GPA across all disciplines. These factors were also signi cant in both the 100 bootstrap samples from each group. regression model and classi cation model of all students. Extrinsic motivation, preference for working alone and self regulation were also signi cant in the regression model, while all factors except extrinsic motivation, preference for working alone and study time were signi cant in a classi cation model of all students. Models of individual courses also differed in the range of factors used. The lack of consensus in identi cation of signi cant factors may be explained by an overlap in the constructs measured by each [ 24 ]. Openness appeared frequently in both classi cation and regression models despite its relatively low correlation with GPA. In general, regression models for students in technical disciplines, such as engineering, computing and business with IT, had a higher coe cient of determination (R2) than models of non technical disciplines. However the coe cient of determination did not re ect prediction error, highlighting the underlying variability in independent variables. For example, early childcare (R2=0.17) and sports management (R2=0.16) had the same R2, but sports management had a higher absolute error (0.64 0.53) than early childcare (0.37 0.34). The di erence was signi cant (t(15)=3.996, p=0.001). Prediction error was re ective of the GPA distribution for each course regardless of discipline.

Classi cation models that distinguished between high and low risk students based on GPA had good accuracy for both technical and non technical disciplines, particularly for courses with a signi cant proportion (>30%) of high risk students. As with regression, models of individual courses outperformed both models of the full dataset and models of random samples taken from the full dataset. This would suggest models trained for speci c courses can outperform models generalising patterns for all students. k-NN, a non-linear classi cation algorithm, gave optimal or near optimal accuracies for most course groups. This may be re ective of non-linear patterns in the dataset.

Including a cognitive factor of prior academic performance did not improve the accuracy of classi cation models signi cantly. On the other hand, Gray et al. [ 23 ] reported that predictive accuracy of models based on cognitive factors only (prior academic performance) increased marginally when non-cognitive factors were included in the model. This would suggest a high overlap in constructs captured by both cognitive and non-cognitive factors of learning.

Model accuracies are based on a heuristic search of attribute subsets. A more exhaustive search is needed to verify optimal attribute subsets. Further work is also required to investigate principal components amongst non-cognitive factors. In addition, results are based on full time students in a traditional classroom setting at one college. Further work is needed to determine if these results generalise to students in other colleges, and other delivery modes.

5. ACKNOWLEDGMENTS

The authors would like to thank Institute of Technology Blanchardstown for their support in facilitating this research, and sta at the National Learning Network for assistance administering questionnaires during student induction.

References

[1] Achen , C. Intrepreting and Using Regression . Number 07-029 in Quantitative Applications in the Social Sciences. Sage Publications , Inc, 1982 .

[2] Baker , R. S. J. D. and Yacef , K. The state of educational data mining in 2009: A review and future visions . Journal of Educational Data Mining , 1 ( 1 ):3{ 17 , 2010 .

[3] Bergin , S. Statistical and machine learning models to predict programming performance . PhD thesis , Computer Science, NUI Maynooth, 2006 .

[4] Bidjerano , T. and Dai , D. Y. The relationship between the big- ve model of personality and self-regulated learning strategies . Learning and Individual Di erences , 17 : 69 { 81 , 2007 .

[5] Biggs , J. , Kember , D. , and Leung , D. The revised two-factor study process questionnaire: R-SPQ-2F . British Journal of Education Psychology , 71 : 133 { 149 , 2001 .

[6] Buckingham Shum , S. and Deakin Crick , R. Learning dispositions and transferable competencies. pedagogy, modelling and learning analytics . In 2nd International Conference on Learning Analytics and Knowledge , pages 92 { 101 , Vancouver, BC, Canada, 2012 .

[7] Carpenter , J. and Bithell , J. Bootstrap con dence intervals - when, which, what? A practical guide for medical statisticians . Statistics in Medicine, 19 : 1141 { 1164 , 2000 .

[8] Cassidy , S. Exploring individual di erences as determining factors in student academic achievement in higher education . Studies in Higher Education , 37 ( 7 ):1{ 18 , 2011 .

[9] Chamorro-Premuzic , T. and Furnham , A. Personality, intelligence and approaches to learning as predictors of academic performance. Personality and Individual Di erences , 44 : 1596 { 1603 , 2008 .

[10] Chatti , M. A. , Dychho , A. L. , Schroeder , U. , and Thus, H. A reference model for learning analytics . International Journal of Technology Enhanced Learning. Special Issue on State of the Art in TEL , pages 318 { 331 , 2012 .

[11] Chow , G. C. Tests of equality between sets of coe cients in two linear regressions . Econometrica , 28 ( 3 ): 591 { 605 , 1960 .

[12] Cooper , A. J. , Smillie , L. D. , and Corr , P. J. A con rmatory factor analysis of the mini-IPIP ve-factor model personality scale . Personality and Individual Di erences , 48 ( 5 ): 688 { 691 , 2010 .

[13] Covington , M. V. Goal theory, motivation, and school achievement: An integrative review . Annual Review of Psychology , 51 : 171 { 200 , 2000 .

[14] De Feyter , T. , Caers , R. , Vigna , C. , and Berings , D. Unraveling the impact of the big ve personality traits on academic performance. The moderating and mediating e ects of self-e cacy and academic motivation . Learning and Individual Di erences , 22 : 439 { 448 , 2012 .

[15] Dekker , G. , Pechenizkiy , M. , and Vleeshouwers , J. Predicting students drop out: a case study . In Barnes, T., Desmarais , M. C. , Romero , C. , and Ventura , S., editors, Proceedings of the 2nd International Conference on Educational Data Mining , pages 41 { 50 , Cordoba , Spain, 2009 .

[16] Drachsler , H. and Greller , W. The pulse of learning analytics. Understandings and expectations from the stakeholders . In 2nd International Conference on Learning Analytics and Knowledge , pages 120 { 129 , Vancouver, BC, Canada, 29 April - 2 May 2012 . ACM.

[17] Du , A. , Boyle , E. , Dunleavy , K. , and Ferguson , J. The relationship between personality, approach to learning and academic performance. Personality and Individual Di erences , 36 : 1907 { 1920 , 2004 .

[18] Entwhistle , N. Contrasting perspectives in learning . In Marton, F., Hounsell , D. , and Entwhistle , N., editors, The Experience of Learning , pages 3 { 22 . Edinburgh: University of Edinburgh, Centre for Teaching, Learning and Assessment, 2005 .

[19] Eppler , M. A. and Harju , B. L. Achievement motivation goals in relation to academic performance in traditional and nontraditional college students . Research in Higher Education , 38 ( 5 ): 557 { 573 , 1997 .

[20] Farsides , T. and Wood eld , R. Individual di erences and undergraduate academic success: The roles of personality, intelligence, and application . Personality and Individual Di erences , 34 : 1225 { 1243 , 2003 .

[21] Fleming , N. D. I'm di erent, not dumb . Modes of presentation (VARK) in the tertiary classroom . Research and Development in Higher Education, Proceedings of the 1995 Annual Conference of the Higher Education and Research Development Society of Australasia , 18 : 308 { 313 , 1995 .

[22] Goldberg , L. R. The development of markers for the big- ve factor structure . Psychological Assessment , 4 ( 1 ): 26 { 42 , 1992 .

[23] Gray , G. , McGuinness , C. , and Owende , P. An investigation of psychometric measures for modelling academic performance in tertiary education . In D'Mello, S. K. , Calvo , R. A. , and Olney , A., editors, Sixth International Conference on Educational Data Mining , pages 240 { 243 , Memphis , Tennessee, July 6-9 2013 .

[24] Gray , G. , McGuinness , C. , and Owende , P. An application of classi cation models to predict learner progression in tertiary education . 4th IEEE International Advanced Computing Conference , pages 549 { 554 , February 2014 .

[25] Gray , G. , McGuinness , C. , Owende , P. , and Carthy , A. A review of psychometric data analysis and applications in modelling of academic achievement in tertiary education . Journal of Learning Analytics , 1 ( 1 ): 75 { 106 , 2014 .

[26] Kang , Y. and Harring , J. R. Reexamining the impact of non-normality in two-group comparison procedures . Journal of Experimental Education , in press.

[27] Kappe , R. and van der Flier , H. Using multiple and speci c criteria to assess the predictive validity of the big ve personality factors on academic performance . Journal of Research in Personality , 44 : 142 { 145 , 2010 .

[28] Kaufman , J. C. , Agars , M. D. , and Lopez-Wagner , M. C. The role of personality and motivation in predicting early college academic success in non-traditional students at a hispanic-serving institution . Learning and Individual Di erences , 18 : 492 { 496 , 2008 .

[29] Knight , S. , Buckingham Shum , S. , and Littleton , K. Epistemology, pedagogy, assessment and learning analytics . In Third Conference on Learning Analytics and Knowledge (LAK 2013 ), pages 75 { 84 , Leuven , Belgium, April 2013 .

[30] Komarraju , M. , Karau , S. J. , Schmeck , R. R. , and Avdic , A. The big ve personality traits, learning styles, and academic achievement . Personality and Individual Di erences , 51 : 472 { 477 , 2011 .

[31] Komarraju , M. and Nadler , D. Self-e cacy and academic achievement. Why do implicit beliefs, goals, and e ort regulation matter? Learning and Individual Di erences , 25 : 67 { 72 , 2013 .

[32] Marton , F. and Sa ljo, R. Approaches to learning . In Marton, F., Hounsell , D. , and Entwhistle , N., editors, The Experience of Learning , pages 36 { 58 . Edinburgh: University of Edinburgh, Centre for Teaching, Learning and Assessment, 2005 .

[33] Mislevy , R. J. , Behrens , J. T. , and Dicerbo , K. E. Design and discovery in educational assessment: Evidence-centered design, psychometrics, and educational data mining . Journal of Educational Data Mining , 4 ( 1 ): 11 { 48 , 2012 .

[34] Moran , M. A. and Crowley , M. J. The leaving certi cate and rst year university performance . Journal of Statistical and Social Enquiry in Ireland, XXIV, part 1 :231{ 266 , 1979 .

[35] Ning , H. K. and Downing , K. The reciprocal relationship between motivation and self-regulation: A longitudinal study on academic performance. Learning and Individual Di erences , 20 : 682 { 686 , 2010 .

[36] Pardos , Z. A. , Baker , R. S. J. D. , San Pedro, M. O. C. A. , Gowda , S. M. , and Gowda , S. M. A ective states and state test. Investigating how a ect throughout the school year predicts end of year learning . In Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK '13) , pages 117 { 124 , Leuven , Belgium, April 2013 . ACM.

[37] Pintrich , P. , Smith , D. , Garcia , T. , and McKeachie , W. A manual for the use of the motivated strategies for learning questionnaire . Technical Report 91-B-004 , The Regents of the University of Michigan, 1991 .

[38] Poropat , A. E. A meta-analysis of the ve-factor model or personality and academic performance . Psychological Bulletin , 135 ( 2 ): 322 { 338 , 2009 .

[39] Robbins , S. B. , Lauver , K. , Le , H. , Davis , D. , and Langley , R. Do psychosocial and study skill factors predict college outcomes? A meta analysis . Psychological Bulletin , 130 ( 2 ): 261 { 288 , 2004 .

[40] Sachin , B. R. and Vijay , S. M. A survey and future vision of data mining in educational eld . In Advanced Computing Communication Technologies (ACCT) , 2012 Second International Conference on, pages 96 { 100 , Jan 2012 .

[41] Shute , V. and Ventura , M. Stealth Assessment . Measuring and Supporting Learning in Video Games . The John D. and Catherine T. MacArthur Foundation Reports on Digital Media and Learning . MIT Press, 2013 .

[42] Siemens , G. Learning analytics. Envisioning a research discipline and a domain of practice . Proceedings of the 2nd International Conference on Learning Analytics and Knowledge , pages 4 {8 , 2012 .

[43] Siemens , G. and Baker , R. S. J. D. Learning analytics and educational data mining. Towards communication and collaboration . Proceedings of the 2nd International Conference on Learning Analytics and Knowledge , pages 252 { 254 , 2012 .

[44] Swanberg , A. B. and Martinsen, . L. Personality, approaches to learning and achievement . Educational Psychology , 30 ( 1 ): 75 { 88 , 2010 .

[45] Tavakol , M. and Dennick , R. Making sense of Cronbach's alpha . International Journal of Medical Education , 2 : 53 { 55 , 2011 .

[46] Tempelaar , D. T. , Cuypers , H., van de Vrie, E., Heck , A. , and van der Kooij, H. Formative assessment and learning analytics . In Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK '13) , pages 205 { 209 , New York, NY, USA, 2013 . ACM.

[47] Tishman , S. , Jay , E. , and Perkins , D. N. Teaching thinking disposition: From transmission to enculturation . Theory into Practice , 32 : 147 { 153 , 1993 .

[48] Vancouver , J. B. and Kendall , L. N. When self-e cacy negatively relates to motivation and performance in a learning context . Journal of Applied Psychology , 91 ( 5 ): 1146 { 53 , 2006 .

[49] Volet , S. E. Cognitive and a ective variables in academic learning: the signi cance of direction and e ort in students' goals . Learning and Instruction , 7 ( 3 ): 235 { 254 , 1996 .