The article explores the problem of missing data to determine the contribution of different components of intelligence and personality factors to student performance. The specificity of the available data is that missed data cannot be considered random. This leads to the instability of the results of filling in the gaps using multiple imputation. Therefore, the task of comparing their performance and choosing the best method for a particular set of data arises. It is suggested to use average dispersion of regression parameter estimates and average statistics reflecting the significance of regression parameters as indicators of method effectiveness. Two methods of multiple imputation for source data the missForest and Amelia were investigated, as well as after selecting the principal components and applying a special procedure of forming limited sets of principal components. The missForest method using the principal components shows the best results and allows identifying informative personal and intellectual predictors of students' academic performance: the level of general, emotional and social intelligence and indicators of introversion and social conformality.
In complex psychological studies of large samples there are problems with data analysis due to missing values of certain indicators in the collected arrays. Such missingness may be caused by both random factors and peculiarities of the study organization. For example, some characteristics are not examined on the entire sample of students, but only on a single group. As a result, a number of variables have missing values for the same objects. This complicates the analysis and filling of the gaps in the data.
Many algorithms for filling in missed data are built on the assumption that the missing values occur at random. Therefore, presence of groups of variables with missing values for the same objects may distort data processing results and reduce efficiency of methods of filling in the missed data. In this regard, it is of interest to compare the effectiveness of different methods of missing data imputation.
As a rule, efficiency of methods of filling of missing values is investigated on model examples for which true values of variables which have been lost in original data are known. For such studies, [1] a number of measures are proposed in the work to assess the quality of filling in the missing data. However, the model examples only partially reflect the observed reality. And the advantages of one method revealed with their help compared to others can be valid only within the framework of these examples and do not apply to the existing specific data set. In the practice of researching student cognitive status, there is a problem with choosing an appropriate method for filling gaps, which would be more effective for a given
data set. Hence, there is a need to establish some measure of effectiveness that can only be calculated from observable data.
This article deals with the practical task of determining the contribution of a number of variables: different components of intelligence and personality traits in the performance of 473 students of Novosibirsk State Technical University. The data set included psychometric indicators of creative abilities, general, emotional and practical intellect, generalized personality factors (extroversion, neurotic and psychoticism) and gender-role stereotypes, but part of the sample lacked the latter indicators.
In this article it is proposed to evaluate the effectiveness of methods to fill gaps based on the stability of regression estimates. The less regression analysis results vary with different options for filling gaps, the more reliable the results the researcher gets because they remain replicable in repeat studies. 2.
The simplest methods of filling in the gaps assume that the gaps are filled separately for each variable, for example, based on the mean. They do not take into account interrelationships between variables and give a significant bias if there are many missing values.
Multiple imputations are more popular [2], but have some disadvantages. They are based on the assumption that data are missing at random (MAR). The latter means that the underlying mechanism of missing data, given the observed data, does not depend on unobservable data.
A sensitivity analysis has been proposed to assess the stability of the results of the multiple imputation with respect to model assumptions (MAR) [3]. It also allows comparing the effectiveness of different multiple imputation methods and quantifying the degree of systematic bias caused by the absence of randomness in the missed data.
Article [4] describes the pitfalls that arise when applying multiple imputation methods: exclusion of a response variable from the imputation procedure, processing non-normally distributed variables, plausibility and violation of the assumption of missed data randomness, computational problems.
In this article, multiple imputation methods that take into account the relationships between all
variables in the dataset have been chosen as methods for filling the missed data, for which effectivity
comparison has been performed. More specifically, the algorithms missForest [
The missForest algorithm uses a random forest trained on observable data matrix values to predict missed values. This is a non-parametric method of filling in the gaps, applicable to different types of variables. The non-parametric method makes no explicit assumptions about the functional form. Instead, it tries to estimate it so that the result appears as close to the data points as possible, but does not seem impractical. It builds a random forest model for each variable. It then uses the model to predict the missing values of the variable with the observed values.
The approach described above gives an estimate of the OOB (out of bag) imputation error and also provides a high level of control over the imputation process. Moreover, it has options to return the OOB separately (for each variable) instead of aggregation across the entire data matrix. This allows for a closer look at how accurately the model fills in the gaps for each variable.
The Amelia algorithm is based on the bootstrap EM algorithm for incomplete data. It allows getting a given number of imputed datasets where the observed sample values are the same, and the unobserved values are drawn from their posterior distribution. The correct results of this method are obtained with the following assumptions: all variables in the dataset have a multivariate normal distribution. Averages and covariances are used to summarize the data.
The missing data are random.
The algorithm works best when the data have a multivariate normal distribution. Otherwise, you need to perform a transformation to get the data close to normal.
If the variables are strongly correlated and have gaps for the same objects, the Amelia method returns a warning message because the results of filling the gaps may be incorrect.
In order to exclude correlation within a group of interrelated cognitive characteristics, it is suggested to use the Principal Components (PC) analysis. Principal component extraction allows you to switch to uncorrected input characteristics. It also allows to increase stability of regression estimates, because it weakens the multicollinearity problem. 3.
The array of analyses included normalized measures of general intellect (which was determined
using the Amthower Structure of Intelligence test) (IQa), emotional intellect (Barchard test) (IQe),
social intelligence (Guilford-Sullivan test) (IQs), creativity (IQc), and practical intelligence (IQp).
Personal traits: extroversion (E), neuroticism (N), psychoticism (P) and social conformance (L) were
determined according to the EPQ questionnaire, femininity (F), masculinity (M), androgynous (A)
according to the Bem questionnaire (for more information on methods of determining intellectual and
personal traits, see Bem's questionnaire). [
The total sample size was 473 observations. However, the number of missing values is quite large. The number of missed data for different variables is shown in Fig. 1.
Although there are more than 100 observations for each variable, deletion of rows containing at least one skip to estimate the regression results in only 13 valid rows remaining. This explains the need to fill in the gaps to evaluate the contribution of each variable to the progress.
Replacing any missing value with the mean of each variable does not yield meaningful results. The regression model built on such data is insignificant at 10% significance level, which does not allow estimating the contribution of variables to the progress. Therefore, to obtain qualitative results, it is necessary to apply multiple imputation methods.
Groups of variables containing missing values for identical objects can be selected: personal characteristics: conformality (L), neuroticism (N), extroversion (E), psychoticism (P); gender stereotypes: masculine (M), feminine (F), anthropogenic (A) and the ratio of masculinity to femininity (K_MF).
For variables of emotional (IQe) and social (IQs) intellectual abilities, about half of the missing values are for the same objects. The same is true for IQa and IQs. For IQp the number of gaps was the highest (almost 200 students), so it was not included in the IQ system and was considered separately. from 30% to 60%. However, a normality test using the Shapiro-Wilk criterion shows that for variables K_MF, L, as well as M and F, the empirical distribution does not deviate from normal at 5% significance level. The empirical distribution of the other indicators is very different from normal. For this reason, we should expect that the Amelia method will give worse results than the non-parametric missForest method. 4.
In order to pass to the independent input variables, for each selected group of cognitive indicators (IQ, gender-role stereotypes, personality characteristics), a principal component analysis was carried out. Rows with at least one missing value were excluded from the analysis. Rotation was used by the varimax method. The resulting loading matrices for the principal components, as well as the proportion of variance explained, are presented in Tables 1-3.
In most cases, the correlation between features is weak; therefore, the share of the explained variance is evenly distributed. Nevertheless, for the group of indicators of gender-role stereotypes (Table 2), it is possible to single out an insignificant component MF4, which explains only 0.5% of the variance of features. Further, it is excluded from the analysis. For the remaining groups of indicators, 4 principal components were used. For them, it is easy to establish a correspondence between the initial characteristics and the principal components based on the maximum absolute values of loadings. They are bold in Tables 1-3.
The values of the principal components were extracted from the constructed loading matrices. For those objects for which there was at least one missing value of the original features, the values of the principal components were also considered missing.
The use of two or more components for one group of features led to the fact that the problem of the presence of variables with gaps for the same objects remained unsolved. To weaken it, it is proposed to form a matrix of input factors based on a limited set of variables included in different groups of cognitive characteristics. The uncorrelatedness of the principal components allows each of them to be included in the regression model separately; this should not lead to a strong change in the parameter estimates. Next, the regression model is estimated for a limited set of variables in which the gaps are filled. Step 1. Form a matrix G of all combinations of numbers from sets,
1,..., K1,1,..., K2,...,1,..., KI (2) one from each set. Each row of the matrix corresponds to a specific combination. The number of columns in the matrix equal I.
Step 2. Put t: = 1.
Step 3. We include in the set of input factors variables of the constant group of factors, the gts-th variable from the s-th group of characteristics, to which the principal component method was applied, for all s = 1, ..., I, where gts is an element of the matrix G.
Step 4. Estimate a linear regression model of the form: r y 0 i xi , (3) i1 where y is academic performance, x1,..., xr - is the set of variables selected in step 3.
Step 5. While
I t Ks , (4) s1 t:=t+1, go to step 3, otherwise the end of the algorithm.
The estimation results are generalized to the initial set of k features (in our case, k = 12) by averaging the estimates for each variable obtained for all possible sets of input factors. For our case, there will be 48 such sets. As a result, we get the average value of the assessment of the contribution of the i-th feature for a given j-th variant of filling in the gaps. The variance is calculated at the same time mean j ˆi , varj ˆi , estimates of the contribution of the i-th feature, obtained for a given j-th variant of filling in the gaps.
Since the considered multiple imputation algorithms produce random results, it is of interest, first of all, the stability of the regression estimates constructed from data with filled gaps. Let the algorithms be applied to the same data set m times, then we get m sets of regression estimates. They are used to calculate the variance of the regression estimates. The Mean Variance for all estimates will be an indicator characterizing the stability of the estimates. (5) (6) where var ˆi is the variance of the estimate ˆi over all random imputations, k is the number of elements of vector except the intersection. When using the algorithm for generating sets of independent variables 1 m var ˆi varj ˆi . (9) m j1
However, the magnitude of the variance depends on the scale; therefore, comparison of the
variation in estimates constructed from different datasets may not always be meaningful. Therefore, it
is additionally proposed to use the indicator proposed in the work [
1 k S t ˆi , (10)
k i1 Where
MV 1 k
var ˆi , k i1 ˆ ˆ0 ,ˆ1,...,ˆk t(ˆi ) mean ˆi
sd ˆi sd ˆi var ˆi is the analogue of t-statistics, mean ˆi is the estimate ˆi averaged over all random imputations, is the standard deviation of the estimate ˆi over all random imputations. When using the algorithm for generating sets of independent variables 1 m mean ˆi mean j ˆi . (13) m j1
This statistic characterizes the quality of estimates. It grows with increasing absolute values of estimates and decreasing their deviation. Thus, the S-statistic is an averaged t-statistic, and can be interpreted in a similar way.
Table 4 presents indicators of the quality of the regression estimation, calculated for 500 random imputations. It is worth noting that in terms of computational performance, missForest is significantly slower than Amelia. However, missForest significantly outperforms Amelia in terms of stability. (12)
missForest
The use of the method of principal components can significantly reduce the average variance of estimates, this decrease is especially significant for the Amelia method (4.5 times). However, the Sstatistic does not increase significantly as a result. The use of limited sets of principal components from different groups of characteristics improved S-statistics for regression estimates constructed after filling in the gaps with the Amelia method.
The best result was achieved using the missForest method using principal components. At the same time, the application of the proposed procedure for the formation of limited sets of principal components made it possible to improve the results of the Amelia method.
characteristics to student performance of cognitive and personal
Let us compare the results of applying the principal component analysis and of using raw data in terms of the obtained regression estimates and their significance (Tables 5-6).
In general, for parameters with high values t(ˆi ) (in particular, t(ˆi ) 2 ) in Table 6, there are no significant discrepancies in terms of the direction of influence (negative or positive) of psychological characteristics on student performance.
Noteworthy is the more significant influence on the performance of indicators of personal
characteristics (LH and MF, Table 6) than intellectual abilities (IQ). Apparently, the use of multiple
imputation for filling the gaps leads to the fact that the effectiveness of the implementation of a
behavioral response (passing exams), first of all, according to the psychobiological cognitive-adaptive
model of personality [
The opposite contribution to the performance of general and social intelligence (respectively, IQa
and IQs in Table 6) obtained according to the MissForest method, with the noted positive role of its
emotional component (IQe), may reflect the dominant influence of personality characteristics as
“personal intelligence” [
In this paper, some performance indicators are proposed for missing value imputation. They allow you to choose an appropriate method for dealing with missing data that provides more stable results of estimating the impact of intellectual and personal properties on the academic performance.
The close relationship of intellectual and personal properties in the analysis of the effectiveness of
educational activities is shown in other studies (for example, [
Consequently, despite some value of the developed procedures for the formation of sets of input factors based on the principal component analysis and the proposed approach to determining the effectiveness of methods for filling in missing psychometric data in the particular problem we have considered, further studies of the effectiveness of solving this problem are required depending on the dimension and nature of the collected arrays.
The research is supported by the Ministry of Science and Higher Education of Russian Federation (project No. FSUN-2020-0009).
Bergold S., Steinmayr R. Personality and intelligence interact in the prediction of academic achievement // J. Intell. 2018. V. 6. N 27. e6020027.
Crameri A. et al. Sensitivity analysis in multiple imputation in effectiveness studies of psychotherapy //Frontiers in psychology. – 2015. – Т. 6. – Art. 1042.
Fonteyne L., Duyck W., De Fruyt F. Program specific prediction of academic achievement on the basis of cognitive and non-cognitive factors // Learn. Individ. Differ. 2017. V. 56. P. 34–48. Gil-Olarte Márquez P, Palomera Martín R, Brackett MA. Relating emotional intelligence to social competence and academic achievement in high school students. Psicothema. 2006;18 Suppl:118-123.