In this article we performed an online experiment to test the role of the user's prior experience with recommender systems in his/her engagement with a new recommendation technology. Our research model is based on the technology acceptance model (TAM) with two new constructs corresponding to the simplicity of the graphical user interface and the skills the user believe he/she has to use recommender systems. Our experiments con rmed the hypothesis that prior user experience plays an important role in which factors a ect the user engagement with a new system.
According to the Technology Acceptance Model (TAM)
[
Figure 1 shows our research model. We measured the Skills construct with questions regarding to the ability of users to use the system, the awareness of the user selfpreferences and the ability to evaluate the recommendations provided. The PEOU construct was measured by the ease of interaction, ease of use and ease of learning. The PU construct was assessed by the attractiveness, satisfaction and accuracy of the recommendations, the suitability to the users mood and tastes, the accuracy of the underlying technology, the understanding of the user preferences and receiving recommendations as good as those that the users could receive from a friend. The SimpleGUI construct was assessed by the absence of confusing links and irrelevant features and the ease of the registration process. Finally, the engagement was assessed by the intention to own the recommended items and the intention to return to the system. Our research hypothesis is that the paths of the model are di erent for users with high experience than for users with low experience. 3.
We asked students and researchers in di erent areas from two di erent universities in Argentina to use a new recommender system1 and to ll a post treatment survey with questions associated to the variables of our model. Each question could be answered in a Likert-5 scale with 1 corresponding to \strongly disagree" and 5 corresponding to \strongly agree". The resulting dataset2 consisted of 153 cases, collected from December 2013 to June 2014. Most users (69.9%) were male and 30.1% were female; 73.2% were in the 20-29 age group, 20.3% in the range 30-39 and 6.5% over 40 years old. 1http://pelisquemegustan.inductia.com/join 2http://marcelo.armentano.isistan.unicen.edu.ar/datasets 3.1
We performed Principal Component Analysis with
Varimax rotation to extract factors from the observed variables.
We found that variables loaded signi cantly only on one of
ve di erent factors extracted (eigenvalues values greater
than 1), demonstrating su cient discriminant validity.
Together, they account for 73.48% of the variability in the
original variables, with a Kaiser-Meyer-Olkin measure of
sampling adequacy of 0.85 suggesting that ve latent
factors are representative. On the other hand, Bartlett's test
of sphericity was highly signi cant (p<0.001) and therefore
factor analysis is appropriate. The factors extracted also
demonstrated su cient convergent validity, as their loadings
were all above the recommended minimum threshold of 0.45
for samples size of 150 [
Regression Evaluation for the constructs was performed
using AMOS, resulting all estimates signi cant at p<0.001
level. We performed model t, obtaining the following
indicators: cmin/df: 0.328, CFI: 1.00, RMSEA: 0.00, PCLOSE:
0.88. The model is then within the acceptable range of
tting according to the guideline thresholds proposed in [
To test for convergent validity we computed the Average
Variance Extracted (AVE). For all factors, the AVE was over
0.5, which is the recommended threshold [
Finally, we conducted categorical and metric invariance tests dividing the dataset according to the users' previous experience. The model t of the unconstrained measurement models (with groups loaded separately) had adequate t: cmin/df: 1.466, CFI: 0.939, RMSEA: 0.39, PCLOSE: 0.999. These values indicate that the model has con gural invariance. After constraining the models to be equal, we found the chi-square di erence test to be non-signi cant (p>0.05). Thus, our measurement model meets criteria for metric invariance across the experience level in the use of recommender systems. 3.3
Regression Evaluation was performed using AMOS and all regression weights resulted signi cant when considering the complete set of users involved in the experiment. To test our categorical moderation hypothesis, we produced the critical ratios for the di erences in regression weights between both groups. From these critical ratios we computed p-values to determine the signi cance of the di erences. The results are summarized in Table 1 and discussed in Section 4. 4.
From our experiments, we found that there is no significant di erence among groups for the e ects of SimpleGUI on PU and PU on Engagement. On the other hand, for those users with low experience in the use of recommender systems the positive e ect of Skills on PEOU, PEOU on PU and PEOU on Engagement is not signi cant while the e ect is stronger and signi cant for users with high experience. Finally, the e ect of SimpleGUI on Engagement is stronger and signi cant for users with low experience while it is not signi cant for users with high experience.
All these ndings can help developers to focus their attention on factors that encourage the use of recommender systems.