Learning is a latent variable, typically measured as academic performance in continuous assessment and end of term examinations [33]. Identifying predictors of academic performance has been the focus of research for many years [20,34], and continues as an active research topic [6,8], indicating the inherent difficulty in generating models of learning [29,46]. More recently, the application of data mining to educational settings is emerging as an evolving and growing research discipline [40,43]. Educational Data Mining (EDM) aims to better understand students and how they learn through the use of data analytics on educational data [42,10]. Much of the published work to date is based on everincreasing volumes of data systematically gathered by edu-cation providers, particularly log data from Virtual Learning Environments and Intelligent tutoring systems [16,2]. Further work is needed to determine if gathering additional predictors of academic performance can add value to existing models of learning.
Research from educational psychology has identified a range of non-cognitive psychometric factors that are directly or indirectly related to academic performance in tertiary education, particularly factors of personality, motivation, self regulation and approaches to learning [8,9,35,39,44,25].
Personality based studies have focused on the Big-5 personality dimensions of conscientiousness, openness, extroversion, stability and agreeableness [9,22,27]. There is broad agreement that conscientiousness is the best personality based predictor of academic performance [44]. For example, Chamorro et al. [9] reported a correlation of r=0.37 (p<0.01, n=158) between conscientiousness and academic performance. Correlations between academic performance and openness to new ideas, feelings and imagination are weaker. Chamorro et al. [9] reported a correlation of r=0.21 (p<0.01, n=158) but lower correlations were reported in other studies (see Table 1) which may be explained by variations in assessment type. Open personalities tend to do better when assessment methods are unconstrained by submission rules and deadlines [27]. Studies are inconclusive on the predictive validity of other personality factors [44].
A meta-analysis of 109 studies analysing psychosocial and study skill factors found two factors of motivation, namely self-efficacy (90% CI [0.444,0.548]) and achievement motivation (90% CI [0.353, 0.424]), had the highest correlations with academic performance [39]. Distinguishing between learning (intrinsic) achievement and performance (extrinsic) achievement goals, Eppler and Harju [19] found learning goals (r=0.3, p<0.001, n=212) were more strongly correlated with academic performance than performance goals (r=0.13, p> 0.05, n=212). Covington [13] however argues that setting goals in itself is not enough, as ability to selfregulate learning can be the difference between achieving, or not achieving, goals set. Self-regulated learning is recognised as a complex concept to define as it overlaps with a number of other concepts including personality, self-efficacy and goal setting [4]. Ning and Downing [35] reported high correlations between self regulation and academic performance, specifically self-testing (r=0.48, p<0.001) and monitoring understanding (r= 0.42, p<0.001). On the other hand, Komarraju and Nadler [31] found effort management, includ-ing persistence, had higher correlation with academic performance (r=0.39, p<0.01) than other factors of self-regulation and found that self-regulation (monitoring and evaluating learning) did not account for any additional variance in academic performance over and above self-efficacy, but study effort and study time did account for additional variance.
Research into approaches to learning has its foundations in the work of Marton & Säljö [32] who classified learners as shallow or deep. Deep learners aim to understand content, while shallow learners aim to memorise content regardless of their level of understanding. Later studies added strategic learners [18, pg. 19], whose priority is to do well, and will adopt either a shallow or deep learning approach depending on the requisites for academic success. Comparing the influence of approaches to learning on academic performance, Chamorro et al [9] reported a deep learning approach (r=0.33, p<0.01) had higher correlations with academic performance than a strategic learning approach (r=0.18, p<0.05).
Cassidy [8] on the other hand found correlations with a deep learning approach (r=0.31, p<0.01) were marginally lower than with a strategic learning approach (r=0.32, p<0.01). Differences found have been explained, in part, by assessment type [49], highlighting the importance of assessment design in encouraging appropriate learning strategies.
Knight, Buckingham Shum and Littleton argued learning measurement should go beyond measures of academic performance [29], promoting greater focus on learning environment and encouragement of malleable, effective learning dispositions. Disposition relates to a tendency to behave in a certain way [6]. An effective learning disposition describes attributes and behaviour characteristic of a good learner [6]. A range of non-cognitive psychometric factors have been associated with an effective learning disposition such as a deep learning approach, ability to self-regulate, setting learning goals, persistence, conscientiousness and subfactors of openness, namely intellectual curiosity, creativity and open-mindednesss [6,29,47]. A lack of correlation between such non-cognitive factors and academic performance is in itself insightful, suggesting assessment design that fails to reward important learning dispositions. It has been argued that effective learning dispositions are as important as discipline specific knowledge [6,29].
Statistical models have dominated data analysis in educational psychology [15], particularly correlation and regression [25]. Relatively high levels of accuracy were reported in regression models of academic performance that included cognitive and non-cognitive factors. For example, Chamorro-Premuzic et al [9] reported a coefficient of determination (R 2 ) of 0.4 when predicting 2 nd year GPA (based on essay type examinations) in a regression model that included prior academic ability, personality factors and a deep learning approach. Robbins [39] Linear regression assumes constant variance and linearity between independent and dependent attributes. There is evidence to suggest variance is not constant for some noncognitive factors. For example, De Feyter et al. [14] found low levels of self-efficacy had a positive, direct effect on academic performance for neurotic students, and for stable students, average or higher levels of self-efficacy only had a direct effect on academic performance. In addition, Vancouver & Kendall [48] found evidence that high levels of self-efficacy can lead to overconfidence regarding exam preparedness, which in turn can have a negative impact on academic performance. Similarly, Poropat [38] cites evidence of non-linear relationships between factors of personality and academic performance, including conscientiousness and openness. It is therefore pertinent to ask if data mining's empirical modelling approach is more appropriate for models based on non-cognitive factors of learning.
A growing number of educational data mining studies have investigated the role of non-cognitive factors in models of learning [6,41,36]. Bergin [3] cited an accuracy of 82% using an ensemble model based on prior academic achievement, self-efficacy and study hours, but due to the small sample size (n=58) could not draw reliable conclusions from the findings. The class label distinguished strong (grade>55%) versus weak (grade<55%) academic performance based on end of term results in a single module. Gray et al. [23] cited similar accuracies (81%, n=350) with a Support Vector Machine model using cognitive and non cognitive attributes to distinguish high risk (GPA<2.0) from low risk (GPA≥2.5) students based on first year GPA. Model accuracy was contingent on modelling younger students (under 21) and older students (over 21) separately.
The focus of this study was to investigate if non-cognitive factors of learning, measured during first year student induction, were predictive of academic performance at the end of first year of study. We evaluated both regression models of GPA and classification models that predicted first year students at risk of failing. Participants were from a diverse student population that included mature students, students with disabilities, and students from disadvantaged socio-economic backgrounds.
The following sections report on study participants and the study dataset. Data analysis was conducted following the CRoss Industry Standard for Data Mining (CRISP-DM) using RapidMiner V5.3 and R V3.0.2.
The participants were first year students at the Institute of Technology Blanchardstown (ITB), Ireland. The admission policy at ITB supports the integration of a diverse student population in terms of age, disability and socio-economic background. Each September 2010 to 2012, all full-time, first-year students at ITB were invited to participate in the study by completing an online questionnaire administered during first year student induction. A total of 1,376 (52%) full-time, first year students completed the online questionnaire. Eliminating students who did not give permission to be included in the study (35) and invalid data (134) resulted in 45% of first year full time students participating in the study (n=1207).
Participants ranged in age from 18 to 60, with an average age of 23.27; of which, 355 (29%) were mature students (over 23), 713 (59%) were male and 494 (41%) were female. There were 32 (3%) participants registered with a disability. Students were enrolled on fourteen courses across five academic disciplines, Business (n=402, 33%), Humanities (n=353, 29%), Computing (n=239, 20%), Engineering (n=172, 14%) and Horticulture (n=41, 3%).
Academic performance was measured as GPA, an aggregate score of between 10 and 12 first year modules, range 0 to 4, and was calculated on first exam sitting only. The GPA distribution (profiled sample) was compared with the GPA distribution of the full cohort of students for that year (reference sample) using a Kolmogorov-Smirnov nonparametric test. The recorded differences in the distribution for 2010 (D=0.032, p=0.93), 2011 (D=0.036, p=0.90) and 2012 (D=0.042, p=0.69) were not statistically significant. The distribution of GPA was also similar across the three years of study. The largest difference was between the 2010 and 2012 profiled samples (D=0.063, p=0.37) and was not significant. To pass overall, a student must achieve a GPA ≥ 2.0 and pass each first year module. 89% of students with GPA > 2.5 passed all modules indication a low risk group that can progress to year two. 84% of students with a GPA < 2 failed three or more modules, indicating a high risk group falling well short of progression requirements. Of the students in GPA range [2.0, 2.49], 39% passed all modules, 36% failed one module, 18% failed two modules, and 7 % failed more than two modules. This is a less homogenous group in terms of academic profile, but could be generally regarded as borderline, either progressing on low grades or required to repeat one or two modules in the repeat exam sittings. Figure 1 and Table 2 illustrate GPA distribution by course.
Table 3 lists the psychometric factors included in the dataset, collected using an online questionnaire developed for the study (www.howilearn.ie). With the exception of learning modality, questions were taken from openly available, validated instruments, with some changes to wording to suit Where two questions were similar on the published instrument, only one was included. This choice was made to reduce the overall size of the questionnaire, despite the likely negative impact on internal reliability statistics. Questionnaire validity and internal reliability were assessed using a paper-based questionnaire that included both the revised wording of questions used on the online questionnaire (reduced scale), and the original questions from the published instruments (original scale). The paper questionnaire was administered during scheduled first year lectures across all academic disciplines. Pearson correlations between scores calculated from the reduced scale, and scores calculated from the original scale, were high for all factors (>=0.9) except intrinsic goal orientation and study time and environment, confirming the validity of the study instrument for those factors. Internal reliability was assessed using Cronbach's alpha. All factors had acceptable reliability (>0.7) 1 given the small number of questions per scale (between 3 and 6), with the exception again of intrinsic goal orientation and study time and environment. Learner modality data (Visual, Auditory, Kinaesthetic (VAK) [21]) was based an instrument developed by the National Leaning Network Assessment Services (NLN) (www.nln.ie).
Correlation and regression were used to analyse relationships between study factors and GPA. Subsequent analysis used classification techniques to identify students at risk of failing. Unless otherwise stated, models are based age, gender and non-cognitive factors of learning as listed in Table 3.
All non-cognitive factors of learning failed the Shapiro−Wilk normality test which is common in data relating to education and psychology [26]. However factors of personality were normally distributed within each discipline except for business. Intrinsic motivation and study effort were also normally distributed for engineering and computing students. There were further improvements when analysing subgroups by academic course. Factors of personality, self regulation and intrinsic motivation were normally distributed for all courses. With the exception of approaches to learning, learner modality, preference for group work and GPA, other factors were normally distributed for most courses. Table 4 illustrates the number of attributes that differed significantly from a normal distribution by course. Larger groups were more likely to fail tests of normality.
Correlations between study factors and GPA were assessed using Pearson's product-moment correlation coefficient (PP-MCC). As some attributes violated the assumption of normal distribution, significance was verified with bootstrapped [22] Openness (6.1±1.3) Motivation:
Intrinsic Goal Orientation (7.1±1.4) MSLQ [37] Self Efficacy (6.9±1.4) Extrinsic Goal Orientation (7.8±1.4) Learning approach:
Deep Learner (5.4±2.9) R-SPQ-2F [5] Shallow Learner (1.3±1.9) Strategic Learner (3.4±2.5) Self-regulation:
Self Regulation (5.9±1.4) MSLQ [37] Study Effort (5.9±1. 95% confidence intervals using the bias corrected and accelerated method [7] on 1999 bootstrap iterations.
Bootstrap correlation coefficients are given in Table 5. With the exception of learning modality, all non-cognitive factors were significantly correlated with GPA. The highest correlations with GPA were found for approaches to learning, specifically deep learning approach (r=0.23, bootstrap 95% CI[0.18, 0.29]), and study effort (r=0.19, bootstrap 95% CI [0.13, 0.24] ). Age also had a relatively high correlation with GPA (r=0.25, bootstrap 95% CI [0.19, 0.3]). A shallow learning approach (r=-0.15, bootstrap 95% CI[-0.21, -0.09]) and preference for group work (r=-0.076, bootstrap 95% CI [-0.14, -0.02]) were negatively correlated with GPA. Openness had one of the weakest significant correlations with GPA (r=0.08, bootstrap 95% CI [0.03, 0.14]). Correlations were comparable with other studies that included a diverse student population [4,9,28] with the exception of self efficacy (r=0.12, bootstrap 95% CI [0.06, 0.17])) which was lower than expected. This may be reflective of the low entry requirements for some courses.
Regression models predicting GPA non-cognitive variables were run for the full dataset and for subgroups by disciplines and by course. The coefficient of determination (R 2 ) is reported to facilitate comparison with other studies. However R 2 is influenced by the variability of the underlying independent variables. Consequently Achen [1, pg 58-61] argued that prediction error is a more appropriate fitness measure for psychometric data. Therefore absolute error mean and standard deviation is also reported.
A regression model for all participants (R 2 = 0.14) was comparable with other reported models of non-cognitive factors [4,30]. However when modelling students by discipline and by course, there were significant differences in model performance. A chow test [11] comparing the residual error in a regression model of all participants (full model) with the residual errors of models by discipline (restricted models) showed significant differences between the full and restricted models (F(17,1098)=22.02, p=0). There was also significant differences between models based on a particular discipline (full model) and models of courses within that discipline (restricted models). In computing, significant differences of F(17,205)=2.22 (p=0.005) were found between the full model and the two restricted models. Within engineering, a model combining mechatronics with electronic & computing engineering was not significantly different from a model of those two courses individually (F(17,79)=0.58, p=0.89), but including either common entry students and/or sustainable electrical & control technology resulted in significant differences between the full and restricted models. Sustainable electrical & control technology was therefore excluded from further consideration because of the small sample size (n=20). Significant differences were also found in models of each of the three humanities courses compared with those courses combined (F(17,302)=2.22, p=0.004). The least significant differences were found in models of business students provided sport management was excluded (F(17, 307)=1.95, p=0.015). Adding sports management further increased the difference in model residual errors (F(17,334)=8.36, p=0). Table 6 gives model details by course and factors used in each model. Electronic & computer engineering students and mechatronic students were combined.
In general, models based on technical courses had a higher R 2 than models for non technical courses. For example, engineering courses, computing (IT) and business with IT all had R 2 > 0.3. Absolute error for these courses was in the range [0.63,0.8]. The difference between the highest absolute error (m=0.8, s=0.56 3 ) and the lowest absolute error (m=0.63, s=0.54) was not significant (t(15)=1.74, p=0.1). Regression results for International Business was also relatively good (R 2 =0.27). For the remaining non-technical disciplines R 2 was lower (range [0.12,0.17]) but the absolute error was more varied. Early childcare had the lowest absolute error (m=0.37, s=0.34) while general business had the highest absolute error (m=0.9, s=0.53). The difference was significant (t(15)=10.3, p<0.001) and may be explained by the greater distribution of GPA scores in general business.
There was little agreement across models on which study 3 m=mean, s=standard deviation factors were most predictive of GPA. Approaches to learning and age were significant for models of all participants, computing students and engineering students, but motivation and learning strategy were more significant for Business with IT. Factors of motivation, learning strategy and approaches to learning were also relevant to models in the humanities courses. All regression models improved when prior academic performance was included in the model. The most significant increase was for sports management, R 2 increased from 0.16 to 0.30. Business with IT and applied social care also increased by more than 0.1. For all other regression models, R 2 increased by between 0.05 and 0.09
Classification models were generated using four classification algorithms, namely Naïve Bayes (NB), Decision Tree (DT), Support Vector Machine (SVM), and k-Nearest Neighbour (k-NN). A binary class label was used based on end of year GPA score, range [0-4]. The two classes were: high risk students (GPA<2, n=459); and low risk students (GPA≥2.5, n=558) giving a dataset of n=1017. Borderline students (2.0 ≤ GPA ≤ 2.49) have not been considered to date. Gray et al. [24] found that cross validation over-estimated model accuracy compared to models applied to a different student cohort. Therefore models were trained on participants from 2010 and 2011 and tested on participants from 2012. All datasets were balanced by over sampling the minority class, and attributes were scaled to have a mean of 0 and standard deviation of 1. Significant attributes were identified by finding the optimal threshold for selecting attributes by weight. Attributes were weighted based on uncertainty 4 for DT, k-NN and Naïve Bayes models, and based on SVM weights for SVM models. Table 6 shows the accuracies achieved and factors used in each model. k-NN had the highest accuracy for models of all students (66%). Accuracies for DT (61%), SVM (62%) and Naïve Bayes (62%) were similar. The most significance attributes by weight were age, deep learning approach and study effort. Including factors of prior academic performance improved model accuracy marginally to 72%.
Model accuracy improved when modelling each course separately. In general, k-NN had either the highest accuracy, or close to the highest accuracy, for all groups with the exception of two courses, international business and early childcare & education. Naïve Bayes had the highest accuracy for both those courses and their attributes of significance were normally distributed. Five courses had accuracies marginally higher than the model for all students, social & community development (70%), applied social care (68%), early childcare & education (69%), creative digital media (67%) and sports management (70%). As illustrated in Table 1, these courses were distinguished by a high average GPA and a low failure rate. Consequently, patterns identifying high risk students may be under represented in these groups. Accuracies for other courses were significantly higher (≥ 79%). For example the difference between sports management (70%) and the next highest accuracy (Engineering other, 79%) was significant (Z=5.86, p<0.001) 5 . It could be argued that the smaller sample size of course groups over estimated model accuracy as smaller samples may under represent the complexity of patterns predictive of academic achievement. Therefore 30 samples randomly generated from the full dataset (n=100) were also modelled. Model accuracy for the random samples was normally distributed, with mean=63.12% (s=11%), which was marginally lower than the model of all students (Z=2.68, p=0.017).
There was little agreement across models on which study factors were most predictive of high risk and low risk students. Conscientiousness, study effort and a shallow learning approach were used most frequently, followed by openness, intrinsic motivation and age. There was no significant improvement in model accuracy when prior academic performance was included in each model. For example, the largest increase in accuracy was from 79% to 82% in a model of Engineering students.
Results from this study suggest that models of academic performance, based on non-cognitive psychometric factors measured during first year student induction, can achieve good predictive accuracy, particularly when individual courses are modelled separately. A deep learning approach, study effort and age had the highest correlations with GPA across all disciplines. These factors were also significant in both the 100 bootstrap samples from each group. regression model and classification model of all students. Extrinsic motivation, preference for working alone and self regulation were also significant in the regression model, while all factors except extrinsic motivation, preference for working alone and study time were significant in a classification model of all students. Models of individual courses also differed in the range of factors used. The lack of consensus in identification of significant factors may be explained by an overlap in the constructs measured by each [24]. Openness appeared frequently in both classification and regression models despite its relatively low correlation with GPA.
In general, regression models for students in technical disciplines, such as engineering, computing and business with IT, had a higher coefficient of determination (R 2 ) than models of non technical disciplines. However the coefficient of determination did not reflect prediction error, highlighting the underlying variability in independent variables. For example, early childcare (R 2 =0.17) and sports management (R 2 =0.16) had the same R 2 , but sports management had a higher absolute error (0.64±0.53) than early childcare (0.37 ± 0.34). The difference was significant (t(15)=3.996, p=0.001). Prediction error was reflective of the GPA distribution for each course regardless of discipline.
Classification models that distinguished between high and low risk students based on GPA had good accuracy for both technical and non technical disciplines, particularly for courses with a significant proportion (>30%) of high risk students. As with regression, models of individual courses outper-formed both models of the full dataset and models of random samples taken from the full dataset. This would suggest models trained for specific courses can outperform models generalising patterns for all students. k-NN, a non-linear algorithm, gave optimal or near optimal accuracies for most course groups. This may be reflective of non-linear patterns in the dataset.
Including a cognitive factor of prior academic performance did not improve the accuracy of classification models significantly. On the other hand, Gray et al. [23] reported that predictive accuracy of models based on cognitive factors only (prior academic performance) increased marginally when non-cognitive factors were included in the model. This would suggest a high overlap in constructs captured by both cognitive and non-cognitive factors of learning.
Model accuracies are based on a heuristic search of attribute subsets. A more exhaustive search is needed to verify optimal attribute subsets. Further work is also required to investigate principal components amongst non-cognitive factors. In addition, results are based on full time students in a traditional classroom setting at one college. Further work is needed to determine if these results generalise to students in other colleges, and other delivery modes.
The authors would like to thank Institute of Technology Blanchardstown for their support in facilitating this research, and staff at the National Learning Network for assistance administering questionnaires during student induction.