Context-aware recommender systems help users nd their desired content in a reasonable time, by exploiting the pieces of information that describe the situation in which users will consume the items. One of the remaining issues in such systems is determining which contextual information is relevant and which is not. This is an issue since the irrelevant contextual information can degrade the recommendation quality and it is simply unnecessary to spend resources on the acquisition of the irrelevant data. In this article we compare two approaches: the relevancy assessment from the user survey and the relevancy detection with statistical testing on the rating data. With these approaches we want to see if it is possible for users to predict which context in uences their decisions and which approach leads to better detection of the relevant contextual information.
Incorporating contextual information in recommender
system (RS) has been a popular research topic over the past
decade. Contextual information is de ned as information
that can be used to describe the situation and the
environment of the entities involved in such a system [
in a speci c service? As the authors in [
It is not always easy to predict which pieces of
contextual information are important for a speci c service. There
are many pieces of contextual information that can in
uence users' decisions in a more (location, social, working
day/weekend) or less (weather, temperature) intuitive way.
The authors in [
We can therefore identify two di erent approaches to the determination of context relevancy: the assessment from user-survey and the detection from the rating data. In the rest of the text we will simply refer to these two approaches as the assessment and the detection.
The assessment and the detection approaches are very different in terms of requirements and approach. The assessment does not require any real rating data, while for the detection we need a substantial number of ratings with associated context. The assessment could therefore be a valuable tool for determining relevant pieces of contextual information before data acquisition, or in other words, during the phase of designing a CARS. Furthermore, while the detection is made on real situation data, assessment is obtained from hypothetical situations described in survey questions. Apart from the fact that users do not necessarily know what really in uences their decisions, quality of the assessment could also be in uenced by users' ability to conceptualize a hypothetical situation. Finally, the assessment is intrusive and requires users to spend their time on an additional task. As we found out, only a minority of users in our system, that we will describe later in the article, were willing to participate in the survey as they did not see the immediate bene t from it (in the contrast with rating items to improve their pro le). The detection, on the other hand, is done without the need for any additional e ort on users' part.
In Table 1 we list pros and cons of each method.
The assessment and the detection di er in the aspects of when each could be used, what information is needed and whether they rely on real or hypothetical situation. However, to the best of our knowledge, several other questions remain that we try to answer in this study.
How well do the outcomes from these approaches match? Which approaches is better in determining relevant context?
Are users aware of what in uences their decisions? 1.3
In this study we compare two approaches for relevant context determination: (i) the assessment from users' survey and (ii) the detection from rating data. The determination of contextual information relevancy for the contextualized recommendations is a binary decision. The piece of contextual information is either relevant (i.e., it contributes to explaining the variance of user's decision/rating) or irrelevant (i.e., it does not contribute to explaining the variance of user's decision/rating).
We will use the detection and the assessment as two di erent methods to classify each piece of contextual information in one of the two classes: relevant and irrelevant. Since the ground truth is unknown (i.e., we do not know which piece of contextual information is actually helpful), we will use the contextualized matrix-factorization algorithm, with each piece of contextual information separately, to determine the in uence of each piece on the rating prediction. The idea is that the relevant pieces of contextual information will lead to better results than the irrelevant ones. The success of the contextualized rating prediction will be evaluated by the root mean square error (RMSE) measure. Once the RMSE for each piece of contextual information is achieved the detection and the assessment can each be evaluated based on the number of times a piece of contextual information determined as relevant performed better than the irrelevant one. Finally, we inspect how well do the results from these approaches match, how well do they perform and which approach is better. 2.
In this section we provide the description of the data used in the study and the methods used to answer the questions established in the problem statement. 2.1
In order to be able to compare the methods for determining context relevancy, we needed a context-rich RS database. Unfortunately, commonly used databases such as Moviepilot and Yahoo! Music, contain only that context which can be derived from the timestamps. Other information available in these databases is general user information that describes users (age, sex, etc.), does not vary for a xed user, and thus cannot be used as a contextual information.
Since we were interested in inspecting more di erent
contextual variables we decided to create a database
containing several potential pieces of contextual information. Since
users have a tendency to rate items di erently in real and
supposed contexts [
We created an online application for rating movies which
users are employing to track the movies they watched and
obtain the recommendations (www.ldos.si/recommender.
html). Users are instructed to log into the system after
watching a movie, enter a rating for a movie and ll in a
simple questionnaire created to explicitly acquire the
contextual information describing the situation during the
consumption. Users are instructed to provide the rating and
contextual information immediately after the consumption,
so that we can make sure that the ratings are not in uenced
by any other factors (e.g., discussing the movie with others,
observing the average movie score on the Internet, etc.)
between the consumption and rating. The users' goal for rating
movies is to improve their pro les, express themselves and
help others, according to [
We collected 1611 ratings from 89 users to 946 items.
Average users' age is 27. Users are from 6 countries and 16
di erent cities. The maximum number of ratings per user is
220 and the minimum is one. The contextual variables that
we collected are listed in Table 2. The decision which pieces
of contextual information to acquire was made according to
the de nition in [
The relevancy of each contextual variable in the LDOSCoMoDa database was tested by hypothesis testing to determine the association between each contextual variable and the ratings. The null hypothesis of the test was that the contextual variable and ratings are independent. The alternative hypothesis states that they are dependent. If we successfully reject the null hypothesis we conclude that the contextual variable and the ratings are dependent and thus that piece of contextual information is relevant.
Since all the variables in the LDOS-CoMoDa database
are categorical we decided to use the Freeman-Halton test,
which is the Fisher's exact test extended to n m
contingency tables [
In order to acquire users' opinion on which contextual information is relevant we created an online survey. It contained 12 questions, one for each contextual information in the LDOS-CoMoDa database. All questions were presented in the same manner. For example, for the day type context: "Do you think you would rate/select a movie di erently if you watched it: on a working day, weekend or holiday?". Available answers for each subject to select for each question were: No, Probably not, Maybe, Probably yes and Yes. All pieces of contextual information were explained and the questions presented in the participants' mother tongues.
The survey was answered by 72 subjects, from which 27 were also users in the LDOS-CoMoDa database. 2.4
Once the survey data was acquired we needed to assess which piece of contextual information is relevant and which is irrelevant, from the subjects' opinions. Since the LDOSCoMoDa database is still small the detection could not be achieved for each user individually. Therefore, both the detection and the assessment were done for the entire population of users, so that the results can be compared. For each contextual information the assessment score was calculated as: s = Pi5=1 !ini, where ni is the amount of answers i and !i is the weight appointed to the answer i, where i goes from
The weights were determined according to the following rules: 1. Answer "Maybe" was set as neutral. Weight for "Maybe" was set as !3 = 0. 2. Answer "Yes" and "No", have more weight than "Probably yes" and "Probably no", respectively. Therefore we decided that !5=!4 = 2 and !1=!2 = 2. 3. Answers "Yes" and "No", and answers "Probably yes" and "Probably no" are exact opposites. !5=!1 = 1 and !4=!2 = 1
If the calculated score is s 0 we assess that the piece of contextual information is relevant, otherwise it is irrelevant. 2.5
The problem of determining how well the results from
both approaches match, is basically the problem of
determining the inter-annotator agreement. In this case we have
two annotators: the assessment and the detection; ve
categorical classes form survey answers as the annotations: from
No to Yes; and twelve pieces of contextual information to
annotate. For the task we used the Cohen's coe cient,
which is a statistical measure of inter-annotator agreement
for categorical items [
We will use the same approach to test how similar the assessment from two groups of survey subjects is, one being only those subjects that are users in LDOS CoMoDa database, and the other being those subjects that are not users in the database. We will test this to see if there is a di erence in opinions between the users that are already using a CARS and the ones that are not. 2.6
In order to evaluate how well each method determined
relevant and irrelevant pieces of contextual information we
calculated context-dependent ratings predictions. Predictions
were made by the matrix factorization as a
collaborativeltering algorithm described and used in [
+ bi + bu + q~iT p~u
where r^ (u; i) is the predicted rating from a user u for the
item i, is a global ratings' bias, bu is a user's bias, bi is
an item's bias, q~i is an item's latent feature vector, and p~u
is a user's latent feature vector. r^, , bu and bi are scalars,
and q~i and p~u are vectors. The contextual variable in the
following equations will be denoted by c. We calculated the
users' and items' feature vectors using the gradient descent
method [
Context was incorporated in the matrix factorization in two ways, by contextualizing users' biases: r^ (u; i; c) =
+ bi + bu(c) + q~iT p~u; and users' latent features: r^ (u; i; c) =
+ bi + bu + q~iT p~u(c); separately. We decided not to contextualize the item's biases due to the small number of ratings per item in LDOSCoMoDa database and since the context ltering of items would increase sparsity in the ratings per items and degrade the results signi cantly. We contextualized users' biases and feature vectors separately to inspect how relevant and irrelevant context in uences biases and feature vectors.
We used the root mean square error (RMSE) as the evaluation measure for the predicted ratings. 2.7
In order to determine which approach is better, we need to have a measure which tells us how good each approach is. This can be measured by comparing the list of relevant and irrelevant contextual information, given by each method, with the results of rating predictions for each contextual information.
We assume that rating prediction that utilizes the relevant context will result in better predictions, i.e., lower RMSE, than the one that utilizes the irrelevant one. This means that for each pair (c(r); c(i)) we should have (c(r)) < (c(i)), where c(r) is the piece of contextual information detected as relevant, c(i) is the piece of contextual information detected as irrelevant and (c) is the root mean square error achieved with the matrix factorization utilizing the context c.
With this assumption in mind we now count the number of times the piece of contextual information determined as relevant lead to worse results than the irrelevant one by the equation:
n(r) n(i) = X X 'ij ; where n(r) and n(i) are the number of pieces of contextual information detected as relevant and irrelevant respectively and 'ij = ( 0 ; 1 ; (ci(r)) (ci(r)) < (c(ji)) (c(ji)) Finally the measure of performance of each method can be calculated as: = 1 n(r)n(i) ; 2 [0; 1]; where n(r)n(i) is the number of all possible pairs (c(r); c(i)). In the best case, when (c(r)) < (c(i)) for every pair (c(r); c(i)), = 0; in the worst case, when (c(r)) (c(i)) for every pair (c(r); c(i)), = n(r)n(i). Note that when calculating 'ij we could use relative RMSE di erence instead of 1 when (ci(r)) (c(ji)), however, since in this study we are interested in the binary decision on the contextual information relevancy, we ignore the degree of di erence between the RMSE scores. 3.
On the survey data, using the assessment method described in Section 2.4 we assessed which contextual information is relevant and which is irrelevant. Similarly, on the data from the LDOS-CoMoDa database, using the detection method described in Section 2.2, we detected which contextual information is relevant and which is irrelevant. The assessment and the detection results are presented in Table 3.
Between the assessment and the detection method the calculated Cohen's coe cient was = 0:118 which is characterized as slight agreement. Between the users and the non users groups of subjects the calculated coe cient was = 0:833 which is an almost-perfect agreement.
Figure 1 shows the average RMSE from both approaches i.e., contextualized users' biases and contextualized users' latent-feature vectors, for all the collected contextual variables. Rating for the items in the dataset are from one to ve.
Performance of each method was evaluated by the method described in Section 2.7. For the assessment we obtained a = 0:629 and for the detection d = 0:969.
The results presented in the previous section can help us answer the questions proposed in the problem statement (Section 1.2).
How well do the outcomes from these approaches
match? There was only a slight agreement between the
assessment and the detection approach in determining
relevant and irrelevant contextual information ( = 0:118). The
di erence between the assessment and the detection can also
be seen in Table 3. This is due to the fact that the
detection is made on the real ratings data and the assessment
depends on the users' ability to imagine a hypothetical
situation. This result agrees with the conclusions in [
Which approaches is better in determining relevant context? If we compare a = 0:629 and d = 0:969 we can conclude that for the determination of the relevant and the irrelevant context, the detection from the rating data performs better for the rating prediction task than the assessment from the users' opinion. In other words, using those pieces of contextual information that were detected as relevant will lead to better rating prediction than when using those that were assessed as relevant.
Are users aware of what in uences their decisions? Almost-perfect agreement ( = 0:833) between the survey subjects that were the users in our CARS and those that were not suggests that there is a sort of an overall population opinion on what context could in uence their decisions regarding movies. This also means that there is no di erence between the users that are familiar with using CARS and those that are not. However, only a slight agreement between the assessment and the detection, and the fact that the detection performs better, tells us that users are not entirely aware of what really in uences their decisions (i.e., ratings) in a movie domain. This is important since the determination of context relevancy from the survey data can lead (as in this case), to using harmful pieces of information, and ignoring the relevant ones.
For the detection, assessment and rating prediction, in this article, we used each contextual information independently. We will inspect these e ects on the multiple, combined context models in the future work. 3.2
In this study we compared two approaches for determining relevant and irrelevant pieces of contextual information in a movie RS: the assessment from the users' survey and the detection from the rating data. We used a real rating data, that we collected in the LDOS-CoMoDa database, and a survey data to test how well these approaches match and which performs better for the rating prediction task. To evaluate each approach we used a contextualized matrix-factorization algorithm. The results showed that there is a di erence between the outputs of these two approaches and that the detection performs better. This points to the fact that the users are not necessarily aware of what in uences their decisions in a movie domain. Still, the assessment could be a valuable approach since it can be used a priori, i.e., before any rating data is collected. However, once the rating data is acquired, the detection should be employed since it will provide better insight into which piece of contextual information is relevant and should be used to improve the recommendations. Our future work consists of inspecting other statistical methods for the detection of relevant context, for di erent variable types. We are also interested in inspecting the detection on the user level, i.e., for each user separately.