=Paper=
{{Paper
|id=None
|storemode=property
|title=Large scale evaluation of multi-mode recommender system using predicted contexts with mobile phone users
|pdfUrl=https://ceur-ws.org/Vol-791/paper6.pdf
|volume=Vol-791
}}
==Large scale evaluation of multi-mode recommender system using predicted contexts with mobile phone users==
https://ceur-ws.org/Vol-791/paper6.pdf
Large Scale Evaluation of Multi-Mode Recommender
System Using Predicted Contexts with Mobile Phone Users
Takashi Shiraki Chihiro Ito Takeo Ohno
Information and Media Processing Information and Media Processing Service Platforms Research
Laboratories, NEC Corporation Laboratories, NEC Corporation Laboratories, NEC Corporation
t-shiraki@bu.jp.nec.com c-ito@az.jp.nec.com takeo@aj.jp.nec.com
1753 Shimonumabe, Nakahara-ku, Kawasaki, Kanagawa, 211-8666, Japan
ABSTRACT had a root mean square error (RMSE) 10% better than that of the
Context-aware recommender systems can improve user Netflix‟s legacy algorithm Cinematch.
satisfaction with recommended information when user preferences However, many challenges still remain in predicting various user
change depending on user contexts (e.g. location, time, and needs, which change depending on context. Cyberguide [1],
weather). However, the effect of each context on various user GUIDE [3], and COMPASS [11] are mobile tour guide systems
preferences has yet to be fully elucidated. In addition, few that provide information such as tourist resorts and exhibits using
examples address this challenge in large-scale real-world the time of day and user current locations as contexts. Magitti [2]
experiments. Therefore, relationships between mobile phone users‟ is a mobile leisure guide system that predicts a user‟s current and
contexts and their preferences have been evaluated. Our system future activity. It is based on extensive fieldwork in which an
calculates the intensity of various user preferences (e.g. favorite online survey and many interviews were conducted. Reference [9]
area and type of content) and recommends content accordingly. It also predicts a user‟s current activity from past / current / future
was applied to a restaurant recommendation service with 2,762 contexts of all users. Though all these systems take into account
mobile phone users over a two-month period. The evaluation contexts that influence users‟ interests, these contexts have rarely
results indicate quantitative evidence that user preferences depend been evaluated precisely in large-scale experiments, because of
on context information. In particular, location information the difficulty in obtaining both user contexts (e.g. location) and
strongly correlates with users‟ interest in recommending different user feedback in the real world.
types of restaurants. Finally, predicted context data is shown to be
more effective for recommendation than raw data. To overcome this, we have developed a Multi-Mode
Recommender System (MMRS), which can adapt to various
Categories and Subject Descriptors contextual conditions and evaluate relationships between contexts
and user interests automatically. We applied it to a trial
H.3.3 [Information Storage and Retrieval]: Information Search recommendation service of about 28,000 restaurants for 2,762
and Retrieval – Information filtering, Relevance feedback, mobile phone users with the largest Japanese mobile
Retrieval models, Selection process. communications operator NTT DOCOMO. As a result, we
clarified the following issues based on quantitative evaluations in
General Terms a large-scale experiment.
Algorithms, Experimentation, Human Factors
- Users‟ favorite items depend on user contexts.
Keywords - The MMRS can find effective profile/context.
Recommender System, Context-awareness, Mobile Systems
- Predicted contexts can be more effective for recommendation.
1. INTRODUCTION 2. MMRS
Due to the recent information overload, recommendation schemes
Figure 1 shows the overall architecture of the MMRS. The
that adapt information to changing user needs are becoming more
important. Recommender systems are increasingly being used for MMRS consists of four components: model learner, user-mode
services on the web and on devices (e.g. video recorders). For estimator, sub-recommender systems (SRSes), and item selector.
Model learner constructs user-mode learning models that are
example, Amazon.com, iTunes, and TiVo mainly recommend
calculated from past user behavioral logs. The user-mode
books, music, and TV programs, respectively. Recommender
systems score items using user profiles, content data, user estimator predicts the current user-mode from the user-mode
feedback such as purchase logs, and user contexts. learning models when a user requests recommendations. Each
SRS makes an item list by using each algorithm such as
Many personalized recommender systems using profiles, content content-based filtering and collaborative filtering. The item
data, and feedbacks have already been proposed. They are suitable selector retrieves results from item lists of SRSes. We describe
for services for providing users with items that match their those algorithms in detail in the subsections 2.1-2.4.
preferences. Amazon.com recommends products by using The MMRS has four types of input: profile data, context data,
item-based collaborative filtering, which finds similar items from feedback logs, and item data. Profile data are static-feature user
user feedback [6]. The Netflix Grand Prize solution [5, 10, 13] data, such as age, gender, and whether she/he smokes. Context
---- data are dynamic-feature user data, such as time, location, and
CARS-2011, October 23, 2011, Chicago, Illinois, USA. pulse rate. Feedback logs are user positive feedback data in
Copyright is held by the author(s)/owner(s). recommendation services, such as purchases, clicks, and
bookmarks. Item data are content data that should be retrieved in To eliminate zeros from the denominator in Equation (3), we use
recommendation services. Laplace smoothing [7], which simply adds one to each count
The MMRS has two types of output: an item list and user-modes. , as an
An item list is set of recommended items that the MMRS retrieves initial condition. The model learner calculates for all
as results. User-modes are defined as user interests in selecting and F for all on a regular basis.
information such as location of users‟ favorite contents, user
preference, and user‟s favorite method for retrieving items. 2.2 User-mode estimator
User-mode estimator is used for online-processing that predicts
current user-mode intensities from user-mode learning models,
profile data, and current context data. When the MMRS receives a
query from a user, the user-mode estimator obtains probabilities in
Equation (1) for a profile/context combination of the user.
Figure 2 shows a screenshot of the user-mode intensities. In the
case of user-mode “Preference,” the equalizer bars in the screen
indicate that user-mode estimator predicts “Gourmet meal,” which
here means a relatively expensive meal, as the most interesting for
the user. The user can also move the equalizer bars to control
recommendation results interactively.
Figure 1. Overall architecture of MMRS
2.1 Model learner
The model learner is used for offline-processing that produces
user-mode learning model from profile data, past context data,
feedback logs, and item data. User-mode learning model is the set
of intensities of user-modes for all profile/context combinations.
The intensity is based on the probability of category of
user-mode in each profile/context combination. It is given by
User-mode of user-mode category
Profile/context of user profile/context category
Figure 2. Screenshot showing user-modes
.
We use the Naïve Bayes algorithm to calculate Equation (1). 2.3 Sub-recommender system (SRS)
Under the Naïve Bayes Assumption between profile/context sets, Each SRS produces an item list from user data and item data.
we have Recommendation service providers can use any recommender
systems as SRSes. Recommended item lists of SRSes generally
differ from each other. The MMRS can evaluate each SRS‟s
effect in context-aware recommendation. In our previous study
[12], we evaluated the effect of five ranking methods
The Naïve Bayes algorithm has the limitation in handling the (content-based / profile-based / item-to-item collaborative filtering
dependency among profiles/contexts. However, we apply it to the / user-to-user collaborative filtering / profile-item matching) in
MMRS because it has the following merits for the conditional personalized recommendation that did not use context data. We
independence assumption between profile/context sets. found that users‟ favorite methods depended on the person. For
example, men in their twenties did not prefer ranking methods
- Space complexity can be reduced from using user profile data as much as females and older males.
to where 2.4 Item selector
- We can evaluate each profile/context set in a fair manner Item selector is used for online-processing that produces a result
- It is easy to use in distributed systems item list from the user-mode intensities predicted by user-mode
Additionally, in Equation (2) is calculated using estimator and item lists of SRSes. User-mode estimator sets the
probability in accordance with Equation (1) for each SRS. When a
frequency , which is defined as the amount of user
user requests recommendation, item selector randomly selects a
feedback in condition during the experiment period, as below, SRS in accordance with the SRS‟s probability and collects one
item at a time repeatedly from its SRSes until the item selector
obtains the requested number of items (Figure 3).
Table 1. User profile and context information (H=6)
Category Description
Age×Gender “Age”( -29 / 30s / 40s / 50- ): four patterns
“Gender” (male/female): two patterns
Drinker Whether user checks “like alcohol” in initial
input data or not. (drinker / non-drinker)
Weather (sunny/cloudy/rainy or snowy) is obtained by a
weather forecast service
Day-type× “Day-type” estimated from user behavioral logs
Time-of-day× (workday/day off): two patterns
Figure 3. Item selection method in item selector User-attribute “Time-of-day:” (0-4 / 4-8 / 8-12 / 12-16 / 16-20 /
3. EXPERIMENT 20-24): six patterns
“User-attribute” means user‟s current behavior
In this section, we explain how we applied the MMRS to a predicted from user‟s historical behavioral
restaurant recommendation service. Its conditions are as follows. pattern. (home / office / second office / staying in
- Users: 2762 mobile phone users who registered online private place/ commute to work / moving for
work / return home / moving to private place):
- Items: about 28000 restaurants in Tokyo and six prefectures
eight patterns
around Tokyo
- Recommendation requests: 65,445 times Current user The user area in Tokyo and six prefectures
area around Tokyo (e.g. Ginza, Roppongi, and Chiba).
- Experimental period: 64 days This is directly obtained using a GPS system or
Therefore, we use two types of positive feedback data. the base stations of the mobile phone network.
- Browse: users click to see detailed restaurant information
Next user Predicted area the user will go next, which is
- Bookmark: users bookmark to save restaurant information
area derived from the analysis of each user‟s
3.1 Profile and context data behavioral patterns.
In this experiment, we chose profile/context sets with different 3.2 User-mode
properties (Table 1). “Age,” “Gender,” and “Drinker” are profile The MMRS recommended restaurants after predicting
data (static-feature user data) obtained from service registration “User-modes” (Table 2) by using the profile/context information
information. The others are context data (dynamic-feature user as a result. In this experiment, we set three user-modes:
data). “Restaurant area,” “Preference,” and “Ranking method.”
“Day-type,” “User-attribute,” and “Next user area” in Table 1, are “Restaurant area” is important, especially for mobile phone users.
predicted contexts by user behavioral pattern analyses as follows. We also consider “Preference” (types of restaurants) as the main
The user behavioral patterns are composed of “stop places” and factor for selecting items. We are also interested in “Ranking
“trip routes.” Stop places are defined as the places where users method” because the MMRS blends results of SRSes.
had stayed within a 500-meters radius for more than 30 minutes.
Table 2. User-mode information (I=3)
Trip routes are defined as the paths between two stop places.
These are generated from location logs of a user‟s mobile phone. Category Description
Before predicting the contexts, we estimate each stop place Restaurant area The area the user prefers in Tokyo and six
between a user‟s home, office (office / second office), and private prefectures around Tokyo .
(except the home) by comparing hours stayed at all stop places, Preference The type of meal the user prefers.
after which we predict the contexts.
Casual meal With price under ¥4,999
“Day-type” tells us whether the day is a workday or day off for
the user. For example, Tuesday is a workday for a user if the user Gourmet meal With price over ¥5,000
has often been in the office on that day. “User-attribute” shows
Alcohol Bars and pubs
user‟s current behavior from eight patterns. The attributes of the
stop place or trip route determine where the user is. “Next user Café For soft drinks and
area” means the user‟s next destination. We obtained this from light meals
trip times, hour, and origin-destination history in the user Entertainment e.g. Karaoke, darts
behavioral patterns.
Ranking The method for ranking that provides
“Age×Gender” and “Day-type×Time-of-day×User-attribute” method recommended items.
denote multiple profile/context sets. We combined “age” and
“gender” because we considered that the difference in preferences Global ranking Determined by popularity
between women and men in their twenties is greater than that
between women and men of other generations. The predicted Personalized This scores items in
contexts “Day-type” and “User-attribute” depend on ranking accordance with how
“Time-of-day” because behavioral pattern analyses of them relate closely they match the
to time. user‟s stated preferences.
The MMRS had three steps in this experiment. First, it selected - Location data > Time data
one “Restaurant area” ( ) by the highest value of Equation (1). - Next user area (predicted context) > Current user area
Next, it estimated probability of “Preference” ( ) by We compared the value of each user-mode in Table 3. For all
context sets, the intensity of three user-modes is as follows.
- Restaurant area > Preference > Ranking method.
Finally, it estimated probability of “Ranking method” ( ) by We considered “Preference” to be the most important user-mode
because it is closest to user interests with regards to restaurants.
Figure 4 shows the drilled-down data of the relationship between
Equations (4) and (5) are easy extensions of Equation (1). “Preference” and “Next user area.” We chose this because “Next
user area” has the highest coefficient (0.354) in “Preference” in
Item selector stochastically selects items from SRSes. We can Table 3. As shown in Figure 4, the MMRS predicted that many
flexibly change user-modes and profile/context sets explicitly that users going to Shibuya/Roppongi (popular nightlife districts in
are important selection criteria based on services. Tokyo) would be interested in “Gourmet” restaurants. Figure 4
also indicates that users going to an urban area like to browse for
4. RESULTS AND DISCUSSIONS high-end restaurants (“Gourmet”) and cafés while those going to a
We elucidated the effect of each context on user-modes. As listed suburban/rural area browse for cheaper restaurants and alcohol.
in Table 3, we obtained the intensities of the relationships In the case of only profile data, we chose the relationship between
between user-modes and profile/context sets from the Cramer‟s “Preference” and “Age×Gender” because “Age×Gender” has a
coefficient association values [4]. Value 0 means no dependency higher coefficient (0.083) than “Drinker” in Table 3. Figure 5
between user-modes and profile/context data, and value 1 means shows the drilled-down data of it. As shown in Figure 5, women
that the user-mode is fully predictable from the profile/context. like to select “Cafés” and men like to select “Casual meals”
First, we compared the values of each profile/context set in (inexpensive restaurants). In addition, men become more
Table 3. We found that context data, except “Weather,” affected interested in “Casual meals” as they age, and women over 50 very
user-modes more than “Age×Gender” and “Drinker” did. are interested in “Gourmet meals” (expensive restaurants). This
Therefore, location context strongly correlates with user-modes. may be due to the fact that Japanese women in their 50‟s tend to
The results indicate that location information is more useful in have a lot of discretionary money.
predicting user-modes. The values of “Next user area” were In this way the model learner can clarify the relationship between
higher than those of “Current user area” in all user-modes. The any user-modes and any profile/context data. The user-mode
results indicate that context data computed by analyzing user‟s estimator predicts current user-mode intensities, and the item
behavioral logs can improve context-aware recommendations. selector can provide recommendation items that are automatically
- Context data (except “Weather”) > Profile data adapted to user interests using the intensities.
Table 3. Cramer's coefficient of association between user-modes and profile/context sets
Profile Context
Time Location
Age×Gender Drinker Weather Day-type × Time-of-day Current Next
×User-attribute user area user area
Day- Time- User-
Age Gender
type of-day attribute
Restaurant 0.142 0.144
0.185 0.064 0.643 0.716
area 0.158 0.188 0.077 0.053 0.144
0.083 0.129
Preference 0.069 0.048 0.209 0.354
0.062 0.103 0.031 0.026 0.086
Ranking 0.043 0.067
0.020 0.009 0.082 0.197
method 0.019 0.014 0.002 0.009 0.028
Figure 4. Relationship between user-mode “Preference” and context “Next user area” (0.354)
6. ACKNOWLEDGMENTS
The research presented in this paper is part of the Information
Grand Voyage Project funded by the Japanese Ministry of
Economy, Trade and Industry. NTT DOCOMO produced “My
Life Assist Service” [8] for the project, and we used their data.
We thank Isao Kobayashi of NTT DOCOMO, Kyota Kanno,
Kenshi Nishimura, Tsunehisa Kawamata, Nobuyuki Saji, and
Yasuhiro Miyao of NEC for their helpful comments. We also
thank all members that contributed to system implementation.
7. REFERENCES
[1] Abowd, G., Atkeson, C., Hong. J., Long, S., Kooper, R. and
Pinkerton, M. Cyberguide: a mobile context-aware tour
guide. Wirel. Netw. 3, 5 (1997), 421-433.
Figure 5. Relationship between user-mode [2] Bellotti, V., Begole, B., Chi, Ed H., Ducheneaut, N., Fang, J.,
“Preference” and profile “Age×Gender” (0.083) et al., Activity-based serendipitous recommendations with
the Magitti mobile leisure guide, In Proc. of the twenty-sixth
Finally, we discuss the relationship between the machine learning annual SIGCHI conference on Human factors in computing
mechanism of the MMRS system and user satisfaction by systems (CHI „08) , pp. 1157-1166, 2008.
observing the click rate, which is defined as the number of clicked [3] Cheverst, K., Davies, N., Mitchell, K., et al., Developing a
items divided by the number of retrieved items. Each plot in context-aware electronic tourist guide: some issues and
Figure 6 is the click rate on one day, and the line is the linear experiences. Proc. CHI‟00. ACM Press, NY, 2000, p. 17-24
regression line. It also shows that the click rate gradually
increased during the experiment. There are several possible [4] Cramer, E. M. et al., Some Symmetric, Invariant Measures of
factors for this increase. Multivariate Association, Psychometrika, 44, 43-54. (1979).
・ Machine learning makes MMRS more effective [5] Koren, Y., The BellKor Solution to the Netflix Grand Prize,
(2009).
・ The ratio of active users who give a lot of clicks increases http://www.netflixprize.com/assets/GrandPrize2009_BPC_B
because some inactive users quit using this services ellKor.pdf
[6] Linden, G., Smith, B. and York, J., Amazon.com
Recommendations: Item-to-Item Collaborative Filtering,
IEEE Internet Computing. Jan./Feb. 2003.
[7] Manning, C.D., Raghavan P. and Schütze H., Introduction to
Information Retrieval, Cambridge University Press, p. 240.
[8] METI of Japan, Information Grand Voyage Project: My Life
Assist Service, 2009-2010.
http://www.meti.go.jp/policy/it_policy/daikoukai/igvp/content
s_en/activity09/ms09/list/personal/ntt-docomo-inc-1.html
[9] Oku, K., et al., A Recommendation System Considering
Figure 6. Click Rates Users‟ Past / Current / Future Contexts, CARS2010.
5. CONCLUSIONS [10] Piotte, M. and Chabbert, M., The Pragmatic Theory Solution
We proposed a context-aware recommender system that retrieves to the Netflix Grand Prize, (2009).
items and user-modes. We elucidated the effect of each context on http://www.netflixprize.com/assets/GrandPrize2009_BPC_Pr
various user preferences. We applied it to a large-scale restaurant agmaticTheory.pdf
recommendation service with 2,762 mobile phone users over a
[11] van Setten, M., Pokraev, S., Koolwaaij J., Context-Aware
two-month period. From the results, we found that context
Recommendations in the Mobile Tourist Application
information is more effective for making recommendations than
COMPASS. In Adaptive Hypermedia 2004, vol. 3137 of
profile information. This indicates that recommender systems
LNCS, p.235-244, 2004.
using context input data can effectively improve user satisfaction.
Furthermore, location data is fairly useful for predicting user [12] Shiraki, T., Kanno, K., Nishimura, K. and Kawamata, T.,
intentions. User-modes depend on “Next user area” more than Evaluation Methods for a Multi-Mode Recommendation
“Current user area”. The results indicate that predicted context System, 71th IPSJ Conference 2009 (in Japanese).
data computed by analyzing user‟s behavioral patterns can http://www.ipsj.or.jp/01kyotsu/award/taikai_yushu/71award_
improve recommendations. Finally we discussed the relationship paper/2C_4.pdf
between the machine learning mechanism of the MMRS and user
satisfaction. We found that the click rate gradually increases. [13] Töscher, A., Jahrer M. and Bell, R., The BigChaos Solution
However, there may be several factors for this increase. We will to the Netflix Grand Prize, (2009).
clarify the relationship between the MMRS and satisfaction with http://www.netflixprize.com/assets/GrandPrize2009_BPC_Bi
comparative experiments in future work. gChaos.pdf