Questionnaire surveys are useful for many areas of science, in particular social sciences. Such surveys are often the prime means of gathering data directly from participants, however, they are prone to missing data, which could be caused by many reasons: (i) an error by survey administrators, (ii) participants not responding to certain questions, (iii) acts of nature and, (iv) etc. In order to keep the full survey sample, researchers must often use imputation to deal with the missing data problem. Methods for imputation can sometimes ofer reasonable estimates for the missing data, however, in the case of the survey: (i) imputation can add high noise to the data, (ii) imputation becomes unreliable when more than 40% of the data is missing. This work attempts to address these issues by evaluating if the usage of matrix completion methods stemming from collaborative filtering (CF) in recommender systems can yield more accurate imputations of survey data. The rationale for the usage of these methods is (i) the similarity between the problem framing, methods and data representation used in CF and survey imputation; (ii) the efectiveness of CF-based methods in recommender systems. We use data from the World Values Survey, a valuable dataset in social science of high volume and veracity, to compare (i) one simple approach to imputation, (ii) two established imputation approaches (iii) two CF matrix completion techniques. The results show that our chosen CF matrix completion techniques perform overall comparable, but not better than existing imputation techniques for the case of survey imputation. The matrix completion techniques, however, might prove useful in niche situations, such as in the imputation of non-ordinal question answers. The right technique for imputation often depends on the problem, these results beckon the consideration of CF-based techniques in future research on survey imputation.
For many areas of science, in particular social sciences, questionnaires are an essential tool for
gathering data. The process of collecting data through questionnaires, called a survey [
A common approach to dealing with missing values is to delete all entries which contain
them [
Existing imputation techniques have advantages, such as, allowing the user to operate on the
full data, however, they can have the issues of: (i) introduction of high noise to the dat1a0[]
and, (ii) in the survey case, inefectiveness when more than 40% of the data is missing1[
These similarities in problem framing, are most noticeable in collaborative filtering (CF) for recommender systems. CF operates on a user-to-item ratings matrix that stores the opinion of human users about given items, usually expressed as a scalar value called a rating (ex. 1-5 Likert scale, where 1 is a very negative and 5 is a very positive ratting). Due to the large volume of items in such systems, users are usually familiar with only a fraction of the items, consequently, much of the entries in the ratings matrix are empty 1[], i.e. missing. The recommendation is then done by filling these missing entries using solely data from this matrix, through a process called matrix completion 1[], items with high predicted ratings, i.e. opinion, are then recommended to the user. If we represent the questionnaire data as a matrix, where rows represent participants, and columns represent questions, the problem of filling missing data is now similar to the problem of matrix completion.
This paper focuses on the comparison between matrix completion techniques and classical
survey imputation techniques, in the task of filling in missing answers in the World Values
Survey [
A number of studies have utilized matrix completion and collaborative filtering outside of the
ifeld of recommender systems. Some of the fields which have used these techniques include
medicine[
Two works examine the use of matrix completion in a broad imputation scenario: (i) Wang et al. produce an ensemble-based imputation method, which includes an item-to-item collaborative technique in the ensemble, they show that their ensemble method outperforms k-nearest neighbors (KNN) imputation, on common datasets from the UCI (University of California Irvine) data repository, however, do not evaluate the performance of the item to item collaborative technique on its own; (ii) Chi and Li3[] examines the use of low-rank matrix completion for the general role of imputation, they use synthetic data to show that low-rank matrix completion techniques can operate under the statistical assumptions for missing data, utilized in imputation.
In the case of survey imputation, the use of matrix completion is also highlighted in some
cases. Vozalis et. al.[17] test the usage of a user-based collaborative filtering technique in
the imputation of a small transportation survey consisting of univariate question answers on
the Likert (1-5) scale. They report a MAE (Mean Absolute Error) of 0.846 for this technique
when imputing data with 20% missing answers. Similarly, Oliveira et al.1[
Although there has been research using matrix completion on survey data, to the best of our knowledge, there have been no attempts to compare the efectiveness of matrix completion techniques and classical survey imputation techniques. In this work, we fill this gap by directly comparing both approaches on the scenario of World Values Survey data.
For the purpose of comparing the efectiveness of matrix completion and classical imputation techniques in the case of missing survey data, we utilize data from the World Values Survey (WVS). The WVS is an international research program devoted to the scientific and academic study of social, political, economic, religious, and cultural values of people in the worl6d].[ In our testing, we use a subset of the data from the WVS’s 7th wave (7th iteration) of the survey, conducted across 57 countries in the years 2017-2021. This subset used in our testing contains the answers of 84638 participants to 274 survey questions, covering topics such as: (i) ethical values (ii) social values and perceptions (iii) political values, (iv) stances on various social and political questions, (v) etc.
The questions used in the WVS are closed questions, meaning that the participants respond using a list of provided answers rather than articulating the answers themselves. Responses are recorded as a number which denotes the participant’s choice from the list. The ranges of the numbers used to record responses in the WVS are from 1 to the number of answers, e.g. for ifve answers the answer range is 1 to 5. Among the questions in our subset, we find 8 question answer ranges: ”1-2”, ”1-3”, ”1-4”, ”1-5”, ”1-7”, ”1-8”, ”1-10” and ”1-11”. Based on their range, the questions answers in our subset can be divided into three categories: (i) Dichotomous questions with binary (e.g. Yes/No) answers, questions in the ”1-2” range fall into this category, (ii) Nominal-Polytomous - questions with a set of more than two answers with no inherent ordering, in our subset questions on the ”1-3” range fall into this category, and (iii) OrdinalPolytomous, questions with a set of more than two answers which in themselves contain an ordering, in our subset all other ranges (”1-4” and up) fall into this category.
For reasons mentioned in the methodology section of this paper, we also retain data on the participant’s country of origin in our testing subset.
The flow of our methodology from the data preparation step to the final comparisons between approaches is presented in Figure1.
The data utilized in our testing is described in Section 3. Among the three types of survey question answers, observed in section 3, i.e. (i) Dichotomous, (ii) Nominal-Polytomous, and (iii) Ordinal-Polytomous questions, we find two imputation tasks, namely, a regression task and a classification task. Ordinal-polytomous answers are handled using regression, while classification is used to handle answers to dichotomous and nominal-polytomous questions.
In both tasks, we compare the efectiveness of matrix completion and classical imputation approaches on the testing data for the specific task. The approaches remain the same in both tasks, only they are adjusted to fit the problem (classification or regression).
Three classical imputation approaches are considered: (i) simple imputation, which serves as a baseline, it imputes the mean value in the regression case and the mode value in the classification case; (ii) k-nearest neighbors (KNN) imputation, a hot-deck approach, in the regression case it uses weighted mean resolution to impute from the neighborhood, while mode resolution is used in the classification case; (iii) model based imputation, which performs initial simple imputation then imputes utilizing one regressor per feature, it uses linear regression with initial mean imputation for the regression task and a bayesian ridge regressor with initial mode imputation in the classification task.
The classical imputation approaches are compared to two matrix completion techniques: (i) item-to-item CF, which, similarly to KNN, uses weighted mean resolution among similar items in the regression task and mode resolution among similar items in the classification case; (ii) non-negative matrix factorization, refined by a Decision Tree regressor and classifier in the regression and classification tasks respectively.
The non-negative matrix factorization is refined by a decision tree in the following way. Let be a column of the original matrix and let’ be the estimation of in the resulting matrix from matrix factorization, for each pair of and ’ we use the available data in to train a Decision Tree which predicts from ’ and use this model to predict the remaining missing answers in from ’ .
To compare the five approaches described above, for each task, we simulate varying degrees of missingness in the data, from 10% to 50%, and evaluate their performance in imputing the data. For regressors, we evaluate Mean Absolute Error (MAE) and Mean Squared error (MSE), while classifiers are evaluated with their accuracy, precision, recall, and F1 scores. The simulation and evaluation is done through an augmented cross-validation technique. A comparison between ordinary and augmented cross-validation is given in Figu2re.
To cater our imputation to the data, hence producing more robust results, we perform all
imputations per country separately. We impute per country since, if the survey data contains
clusters, such as those born of demographics, better imputation results are achieved if data is
imputed for each cluster separately [
Dichotomous and Nominal-Polytomous
Questions Regression
Ordinal-Polytomous
Questions and efective approach is imputing answers for each country separately [18].
5.1.
ifnally across all magnitudes of missingness tested (from 10% to 50%). For the task of regression, all values are scaled from to the 1 to 3 scale.
We keep this scale in our final results to make sense of average error in the
multivariate scenario. The best-performing imputer is marked in bold.
The results for Accuracy, F1 Score, Precision, and Recall in the classification task are given in Table 2. Similarly, as in the regression case, the evaluation statistics presented are the average across all questions that fall under this task, as well as over all countries and magnitudes of missingness tested. The best-performing technique for each evaluation statistic is marked in bold.
From Table 2 we can see that the mode per country is a powerful predictor in the case of the classification task. This implies that the data is unbalanced, hence, the F1 score, Precision, and Recall are better indicators of the performance in this imputation task. Since the F1 score is a balanced measure between Precision and Recall, we will use it as the prime metric for comparison in the case of classification.
The results show that our chosen CF matrix completion techniques perform overall comparable, but not better than existing imputation techniques for the case of survey imputation. The matrix completion techniques, however, might prove useful in niche situations highlighted in the results. Item-to-item collaborative filtering performs comparable to the KNN technique in both imputation tasks, only failing to match it on high ratios of missing data in the classification case. On the other hand, item-to-item fails to compare to model-based imputation in the regression, however, performs better than it in the classification task. Moreover, the results show that the matrix factorization technique ofers poor performance in terms of MSE in the regression case, failing to match both existing imputation techniques, however, in the case of classification it outperforms all techniques tested with its F1 performance on unbalanced data.
In comparison with our related work, we achieve similar results to Vozalis et. al1.7[] for MAE in terms of matrix completion, his MAE of 0.846 on univariate 1 to 5 data is comparable to our 0.40 MAE on the scale of 1 to 3, achieved under multivariate data. This raises the question of whether the scale afects the matrix completion techniques, collaborative filtering techniques in recommender systems usually operate on ratings all on the same scale. Can alterations of these techniques to fit multivariate data, be more beneficial in future work in survey imputation?
We also note that the nature of the data might afect the results, for example, the modelbased imputer performs initial mean imputation before building its models, therefore the high performance of the model-based imputer in the regression task may be due to the power of mean imputation in our data. Future work might compare matrix completion and classical imputation techniques on a larger range of survey data.
Future work on this subject should also consider these techniques in diferent scenarios, as well as, examine the efects that these techniques have on the statistical inference. Moreover, our study included only simple techniques for matrix completion, CF techniques are vast and varied, and other techniques might succeed where we have failed, the considerations of such techniques in the study of imputation may also prove fruitful in future work.