=Paper=
{{Paper
|id=None
|storemode=property
|title=Towards a Context-Aware Photo Recommender System
|pdfUrl=https://ceur-ws.org/Vol-889/paper4.pdf
|volume=Vol-889
}}
==Towards a Context-Aware Photo Recommender System==
https://ceur-ws.org/Vol-889/paper4.pdf
Towards a Context-Aware Photo Recommender System
Fabrício D. A. Lemos1 2, Rafael A. F Carmo 3, Windson Viana1 3, Rossana M. C. Andrade1 2 *
1
Group of Computer Networks, Software Engineering and Systems (GREat)
2
Master and Doctorate in Computer Science (MDCC)
3
UFC Virtual Institute (IUFCV)
Federal University of Ceará (UFC)
{fabriciolemos, rossana}@great.ufc.br
{rafael, windson}@virtual.ufc.br
ABSTRACT improvements and home sensors technologies allow a better
The main challenge of recommender systems is to be able to user’s context characterization and this information can be useful
identify and recommend items that have a greater chance of to improve recommendations.
meeting the interests of their users, which generally have a very In this scenario, this paper presents the Mobile Media to You
subjective and heterogeneous nature. It is imperative, then, that (MMedia2U), a photo recommender system that suggests images
recommender systems, from the identification of each user's previously annotated with contextual information. In order to
profile, could recommend personalized items. However, the user’s execute the recommendation, MMedia2U explores the current
profile is not enough for the system to be able to completely context of the user acquired by his/her mobile device. This
identify the user’s interests. The use of the system in a different domain is interesting since there are available a large number of
context from the usual may cause an unsatisfactory result for the images on sharing sites such as Flickr1 and Picasa Web2. Many of
recommendation, requiring it to be adapted to a new context. This these pictures are captured by mobile devices that store the
paper presents the MMedia2U, a prototype of a mobile photo location, date, time, and other contextual information which can
recommender system that exploits the user’s context and the be explored for recommendation.
context when the photo was created as a means to improve the
recommendation. Three context dimensions area exploited: 2. Context-Aware Recommender Systems
spatial, social and temporal. We describe the similarity measures
used for each dimension and the results of the system evaluation
(CARS)
With the dissemination of ubiquitous computing concepts,
by 13 users following a Gold Standard approach.
context-awareness has become a very important research field. Its
Keywords ideas have been used to increase efficiency and usability of
Recommender Systems, Context-Awareness, Ubiquitous Information Systems, particularly, those systems accessed from
Computing. mobile devices [5]. Context-awareness in recommender system
has been motivated from research, which recognizes the
1. INTRODUCTION dependence of user long-term needs on time, location, and any
One of the most important challenges in Information Systems is information about the physical environment surrounding the user
information overload. Recommender Systems try to cope with this [6][10]. Context-awareness introduces an additional level of
problem by helping people in retrieving information (ex: videos, personalization since it takes into account the influence of the
TV programs, routes, images, people, etc.) that may match their external environment of the user on his/her appreciation of the
preferences and intentions. Recommender Systems try to identify products or items. Recommender systems can take benefits from
items, from the information corpus, that have a greater chance to the context-awareness by considering not only the characteristics
meet the wishes of its users [4][6]. However, the characterization intrinsic to each item and user, but also the characteristics of their
of user’s preferences and intentions is a complex task that, as rule, current situations (both user and item). For example, gathering
needs user intervention to be fulfilled correctly. Another issue of context, a restaurant recommender system will be able to adapt its
Recommender Systems is related to user's context. The use of the recommendations for restaurants that are next to the user, open
system in a different context than usual may cause an and that have available seats for the amount of people who are
unsatisfactory result for the recommendation, since preferences with the user (e.g., his/her family).
and intentions can be influenced by the user's context (location, In the last years, some researchers have showed the feasibility of
trajectory, time, activity, etc.). Context-awareness refers exactly to implementing CARS (e.g., news, movies, music, and services).
the capacity that a system has to detect user’s situation and guide Adomavicius et al. [6], for instance, implement a recommender
the system behaviour accordingly [5]. Nowadays, mobile devices system of movies that takes into account the user context (e.g., if
Permission to make digital or hard copies of all or part of this work for the user is going to watch the movie at home or in the movie
personal or classroom use is granted without fee provided that copies are theatre) and it has attested improvements in the precision and
not made or distributed for profit or commercial advantage and that recall of recommendations. The authors also propose a
copies bear this notice and the full citation on the first page. To copy classification of context-aware systems into two categories. The
otherwise, or republish, to post on servers or to redistribute to lists, first category contains systems that use contextual information as
requires prior specific permission and/or a fee.
CARS-2012, September 9, 2012, Dublin, Ireland.
Copyright is held by the authors. 1
www.flickr.com/
2
http://picasaweb.google.com/lh/explore
*Sponsored by CNPq under Productivity Scholarship DT-2
criterion for filtering items. For instance, COMPASS [1] is a a context model, in which are established the elements that
tourism guide that recommends Points of Interest (POI) taking compose its description and how it should be represented (e.g.,
into account the current context of the user. Kaialeido Photo [2], using ontologies, XML, objects). Fig 1 shows our context model
in turn, performs contextual annotation of images and allows the represented as OWL-DL ontology. Our model has four
user to filter the albums according to preset categories (e.g., an dimensions: spatial (location and points of interest), social (e.g.,
event where the photo was created). In this system, the user has to personal information and activity being performed), temporal
explicitly specify the filters to be used. Columbus [11] is simpler; (date and time) and computational (mobile device).
it is a mobile application that displays georeferenced photos taken
near the user’s location.
The second category comprises systems that use the contextual
information at the time that the user evaluates an item. In addition
to the ratings and characteristics of items and users, systems of
this category also take into account contextual information (e.g.,
location) during recommendation. The work described by
Adomavicius et al. [6] is an example of a recommender system in
this category. The system records time that a user watched a
movie, stores both the score given by the user to the film and the
context in which the movie was watched (e.g., at home, with his
wife). Thus, for example, a film that was well rated by users in a
given context, are more likely to be recommended to a user who is
in a similar context. In the domain of multimedia content, the
system C2_Music [7] incorporates contextual information on
music recommendation. In general, the behaviour of this system is
similar to the movie recommender system, in which the song Fig 1- Our Context Model.
evaluation is enriched with the context in which it was heard (e.g.,
In MMedia2U, these dimensions were explored in the acquisition
day of week, weather conditions).
of knowledge about users and photos. These dimensions are
Such systems depend, however, on a historical database already exploited in context-aware management of photos [3].
describing which items were evaluated by others users (or the user They have been proved to be relevant in organizing and finding
himself/herself) in similar contexts to the current user’s context. personal photos, which is an indicator that can also be exploited in
Another problem occurs when a new item is added to the recommender systems of this type of multimedia document. In
collection. As this item has not yet been used, it will be difficult to MMedia2U, the location attribute is extracted from the user’s
recommend it in accordance with recommendation techniques mobile device (e.g., GPS). Other attributes such as place
such as collaborative filtering (so-called cold start problem). The description (e.g., shopping, beach, etc.) can be derived from freely
work described in this paper differs from the studies available web services such as GeoNames3 e WikiMapia.4 Date
aforementioned since it takes into account both the context of and time considered are the time of use of the system. In the
users and the context in which the items were created. The current version, the activity needs to be informed by the user and
hypothesis is that photos taken in a given context c may be of can be chosen from among the options presented or reported
interest to users who are in similar contexts to c. Then, we do not manually. Examples of activities are: sports, festivals, and
need, in a first moment, of a historical database of landscapes.
recommendation evaluations. MMedia2U uses a knowledge-based
recommendation method trying to avoid the cold start problem of 3.2 Similarity measure
collaborative filtering [8]. Similarity measure is used in the system in order to retrieve those
photos created in contexts more similar to the user's current
3. MOBILE MEDIA TO YOU (MMedia2U) context. The algorithm developed is an adaptation of traditional
In the system presented in this paper, users receive knowledge-based techniques [13], which uses the user context as
recommendations of photos created in contexts similar to current indicative of their preferences and the context of items (i.e.,
users’ context. This similarity computes three contextual photos) as a representation of its features. In our system, the
dimensions (spatial, social, and temporal). The system has as context of items is the context in which the photos were created.
target two types of users. The first type are those who are in an The similarity is calculated between the context of the user U and
unusual context (e.g., visiting a tourist sight for the first time) and the context of an item I using the following formula:
they can enjoy the pictures recommended to have references to
activities or new places to explore. The second group contains
users that have already been in this similar context and that the (1)
recommended photos may give a new vision and perspective of In Formula 1, the similarity is calculated without the need of
the situation they find themselves. Rost et al.[11] have noted that training data. In this case, c is an attribute belonging to the
georeferenced images can influence in a positive and playful way dimensions of the context model (e.g., location); wc is the weight
the exploration of space by these two categories of users. of influence of attribute c (e.g., location has a weight of 50%) and
3.1 Context Modelling simc is the similarity function for attribute c. Those pictures that
A fundamental part in the development of a context-aware system have the highest value of similarity are the ones recommended to
is the definition of what information should compose the
“context”, since elements that describe the contextual information 3
http://www.geonames.org/
depend on the system tasks, and on the system capacity to observe 4
this information. This definition is associated with the creation of http://wikimapia.org/
user U. The function simc is particular to each type of context and provided by our recommender system, and it informs the user's
application domain. current context. MMedia2U server receives the request, and
Each context model dimension must have a method to calculate its performs an enrichment of user context data. The metadata stored
similarity. The location similarity, for instance, can be calculated in the repository of photos are scanned and compared with the
by measuring the distance between the place where the picture current user’s context. In step 5, MMedia2U computes a photo
was taken and the current user’s location. The similarity for ranking according to the results of similarity measures. The
activity is calculated by comparing the activity or occasion that ranking contains the photos URLs and their metadata.
the user is found and the activity or occasion in which the image 3.4 Photo Corpus
was generated. Some of the activities mapped in image shoots and MMedia2U has a repository of photos for recommendation. These
their similarities are shown in Table 1. photos need to be associated with contextual information in order
Shopping Party Leisure Sports to be compared with the current context of the mobile users. The
photos should have as metadata the location where they were
Shopping 1 0 0 0 taken and the activity of the photo’s author at the time of their
Party 0 1 0,5 0 creation. At first, we expect to use photos from Web 2.0
applications, such as Flickr and Picasa Web. However, we find
Leisure 0 0,5 1 0,5 many errors in the metadata of these photos (e.g., time, location).
In order to evaluate the recommendation method without
Sports 0 0 0,5 1
annotation errors, we built a repository of photos from images of
Tab. 1 – Activity Similarity Picasa Web. Manually, we corrected and increased the metadata
Formula 2 is used with numeric context attributes. The similarity returned by the Picasa Web Service. Then, we incorporated the
is defined by how close two values are. new metadata into the photo file by using IPTC and EXIF
headers. Examples of enrichment are the inference of the activity
V c (U )− V c (I ) from the photos description, and the day of week according to the
sim c (U,I )= 1−
max (c )− min (c ) shot date. We hope the evolution of multimedia content
(2) management systems, such as CoMMeDiA [3], will reduce the
In Formula 2, Vc(U) and Vc(I) represent, respectively, the values effort to enrich this kind of image metadata.
of context c for the user and the item; max(c) and min(c)
represent, respectively, the maximum and minimum values for the
3.5 MMedia2U Mobile Application
compared attribute of context c (e.g., for c = hour of the day, The mobile application was developed for the Android5 platform,
min(c) = 0, and max(c) = 12). The similarity between dates can compatible with devices that have version 2.2 or higher. Fig. 3
compare the various attributes related to the moment the photo shows an execution flow of the mobile application.
was shot and the moment the user is found. Some attributes
compared were: the hour of the day, day of the week and month of
the year. The date attributes were compared individually to
analyze the influence of each one (e.g., Has the similarity between
hour of the day a greater influence on the choice of the user than
the similarity of the day of the week?). The similarity between the
months of the year can be calculated by Formula 2, adapting it to
cyclic values (e.g., the distance between January and December is
1, instead of 11).
3.3 System Architecture Fig 3. Execution flow of the mobile application.
We designed the MMedia2U following a client-server architecture The user chooses the "activity/interest," then, the system captures
for mobile computing that is based on RESTful Web Services. the current context (location and date/time), and it sends to the
This design allows mobile devices, even those with low server. This, in turn, returns a list of recommended images (third
processing capabilities, make use of our recommender service. Fig screen). If the user selects a photo, he can see its position and the
2 presents the execution flow of the system. distance between him and the place where the photo was captured.
Clicking on the picture located on the map, its content is displayed
in full screen, and its metadata can be also viewed.
4. EXPERIMENTS
Evaluating a recommendation system is a hard task, due to the
property that an item’s relevancy has a strong personal nature and
is complex to be measured. This difficulty is enhanced when exist
a lack of historical evaluation data, which makes large-scale
studies very costly and difficult to be run. In the case of CARS,
the complexity is even bigger, since we need to range the possible
Fig 2. Execution flow of the MMedia2U recommendation. contexts of real situations (i.e., places, daily situations, etc.). As
In step 1, a mobile application is responsible for gathering user’s we did not have a historical data about recommended photos, we
context. Some types of information (e.g., location) can be created a Gold Standard, which consisted of photos evaluated by
acquired from sensors (e.g., GPS). Other types rely on information
passed by the user (e.g., current activity). In the second step, the
5
mobile application accesses, from a HTTP call, the Web-Service http://developer.android.com/
users in a certain context. The objective was both to use the Gold algorithm, the average precision was 0.28. This precision is
Standard to compare the performance of our recommendation and relatively high since some users have chosen more than 30 photos
to use it as historical data. for a specific context (e.g., a user in particular has selected half
While building this Gold Standard, it was asked for a group of 13 part of the corpus).
users to evaluate photos from 8 different contexts, each one The last two rows of the table show the average precision when
consisting of a stage of evaluation. In each stage, one context using the calculation of similarity of only one of the contextual
(e.g., shopping in some stores on the seaside of the city of dimensions. Combination I, which got the best results, was the
Fortaleza6 during the evening) was presented to the user, who had one we assign twice the importance of Activity in relation to
to visualize a set of photos and choose those that seemed to be Location and four times in relation to temporal attributes.
more appealing for him/her, taking into consideration the context Comparing the results obtained without training (Combination I
he/she was suggested. The photos chosen by the users were and Equal Weights) in relation to random experiments, one can
included in the Gold Standard and provided the historical base in see that it is possible to have much higher precision than random
which the recommendation of MMedia2U was evaluated. The choices (agreeing with the hypothesis H-II and H-I). Moreover,
degree of success on recommendations was then evaluated by the the gain over the random method was not big when considering
ratio of chosen photos that are in the Gold Standard (e.g., if in a only one contextual dimension (e.g., only location or only
given combination of user and context, 10 photos are activity), leading us to believe that a model of full context is
recommended and 6 of these are in the Gold Standard of the same essential for a good context-aware recommendation (hypothesis
combination, then the recommendation precision is 0.6). H-III).
In each stage of evaluation, a mean of 100 photos were visualized Fig 4 shows the F measure (harmonic mean) analysis for the six
by the users. 20 were taken in similar contexts to the one showed recommendation algorithms. Equal weights and Combination I
to the user and 80 were different in some dimensions of the had the better results for Top 3. Regarding the Top 5, Top 10 and
context (e.g., same activity but very distant location). Five of 13 Top 20 lists, Combination I and Least Square were the most
users didn’t know the place chosen as the location for the users effective. For instance, Combination I (0,410 for Top 20) was two
(the seaside of Fortaleza). Ten users were Computer Science times better than Random algorithm (0,201).
graduate students aging from 23 to 28. All of them use mobile
phones every day. For some of the users, we have presented 8 We used the Student t-test in the precision values for the Top 3
photo collections, while for others we have only presented a obtained by the algorithms compared to the results obtained by
subset of it, performing 66 simulations. random selection of photos. This showed that the combined use of
contextual attributes is significantly better than randomization
4.1 QUANTITATIVE RESULTS (95% degree of confidence, probability < 0.0001). In addition, the
The objective of the experiment was to evaluate the following test showed that the use of only one attribute (activity or location,
hypotheses: for example) is not significantly better than the random method.
(H-I) It is possible to make satisfactory recommendations of Finally, the comparison with the results of Combination I and
georeferenced photos without prior knowledge of the user profile, Least Squares resulted in performance differences not statistically
considering only its current context; significant, which may not indicate the need for training to
improve the precision of this kind of CARS.
(H-II) The context in which the photos were taken is relevant in
making recommendations; and Top 3 Top 5 Top 10 Top 20
(H-III) The usage of a context model considering various Equal weights 0.56 0.54 0.45 0.42
contextual dimensions may lead to an improved recommendation Least Squares 0.54 0.56 0.51 0.44
comparing to the result of one which uses only one context
attribute (e.g., location). Combination 1 0.56 0.56 0.51 0.47
In order to verify these hypotheses, first, the algorithm was run Random 0.29 0.28 0.26 0.25
with different weights for each dimension without a previous Localization 0.28 0.30 0.30 0.29
training data. Another implementation used the weights obtained
Activity 0.35 0.33 0.33 0.34
by training the algorithm using a 7-fold method (same way of
adjusting the parameters) [5]. Weights adjustment in the training Table 2. Mean values of obtained precisions.
data was performed by linear regression using the least squares
method. We compared the precision changing the amount of
recommended photos. For this analysis, we also used random Activity
choices as base statement since we were unable to find other Localization
photo recommendation algorithms that use context information. Top 20
Table 2 shows the average precision of our recommendation
Random
Top 10
algorithm in relation to four sizes of recommendation lists (Top 3, Combination I
Top 5, Top 10 and Top 20). The average precision of the Top 5
algorithm without training, assigning equal weights to all Least Squares
similarity measures, was 0.54 for the Top 5 (5 recommended Top 3
Equal weights
items). Using the weights obtained by the least squares, the
precision of the Top 5 was 0.55. Recommending pictures at 0,0000 0,5000
random, without the ranking generated by the recommendation
Fig 4- Mean values of f-measure
6
http://en.wikipedia.org/wiki/Fortaleza
4.2 USERS QUESTIONNAIRE ACKNOWLEDGEMENTS
At each stage of the experiments, a questionnaire was applied to This work is a partial result of the UbiStructure project supported
the users so that relevant factors were investigated in the by CNPq (MCT/CNPq 14/2011 - Universal) under grant number
implementation of CARS for photos. One of the factors to be 481417/2011-7.
investigated was the relevance of a mobile system for photo
recommendation and, in the case of existence of such system, if it 6. REFERENCES
would be interesting to recommend photos taken into account the [1] Van Setten, M., Pokraev, S. and Koolwaaij, J. 2004. Context-
current user’s context. He/she was asked whether, in this specific aware recommendations in the mobile tourist application
context, the user would like to receive recommendations of photos compass. In Adaptive Hypermedia and Adaptative Web-
taken in a similar context. 74% of users answered “yes” for this Based Systems, 235–244
question. When asked whether the recommendation of pictures [2] Xia, S., Gong, X., Wang, W., Tian, Y., Yang, X., and Ma, J.
generated in similar contexts would be interesting, the level of 2010. Context-Aware Image Annotation and Retrieval on
interest was 100%. Another point investigated was the relative Mobile Device. In Second International Conference on
weight of each contextual dimension. Each user was asked what Multimedia and Information Technology, 111-114.
contextual dimension, including location, activity, date and time;
[3] Viana, W., Miron, A., Moisuc, B., Gensel, J., Villanova-
they would prefer to be taken into consideration to build the set of
Oliver M. and Martin, H. 2011. Towards the semantic and
photos. Eight of the users said they think the proximity of the
context-aware management of mobile multimedia. In
location of the photo is the most important factor to increase their
Multimedia Tools and Applications, v. 53, 391-429
interest. Five of the users responded that the activity associated
with the picture portrayed by the activity, which he/she is playing [4] Jannach, D., Zanker, M., Felfernig, A. and Friedrich, G.
at the moment is the most important factor. No users found the 2010. Recommender Systems, an Introduction. In Cambridge
similarity between date and time the most important factor. University Press, first edition
5. CONCLUSION AND FUTURE WORK [5] Fabiana G. Marinho, Rossana M.C. Andrade, Cláudia
Werner, Windson Viana, Marcio E.F. Maia, Lincoln S.
This work presented MMedia2U, CARS for contextual photos.
Rocha, Eldânae Teixeira, João B. Ferreira Filho, Valéria L.L.
During the system development, we specify a context model to
Dantas, Fabrício Lima, Saulo Aguiar, MobiLine: A Nested
cope this application domain and we adapted a knowledge-based
Software Product Line for the domain of mobile and context-
technique to incorporate the context information. MMedia2U
aware applications, Science of Computer Programming,
allows recommendation of photos even those that have never been
Available online 18 May 2012, ISSN 0167-6423,
evaluated by users. The recommendation is performed only from
10.1016/j.scico.2012.04.009.
the context in which the photos were created. This allows users,
without a history of use, to receive recommendations based on [6] Adomavicius, G. and Tuzhilin, A. 2010. Context-Aware
their current context. The recommendation mechanism was Recommender Systems. In Recommender Systems
validated by the construction of a Gold Standard. The average Handbook: A Complete Guide for Research Scientists and
precision achieved by the algorithm allowed us to conclude that, Practitioners, Springer Press, first edition
for the data used, context-awareness can bring gains in the photo [7] Lee, J. S. and Lee, J. C. 2007. Context awareness by case-
recommendation compared to a random list. It is important to note based reasoning in a music recommendation system. In
that even weight combinations without training phases (the Comb Proceedings of the 4th international conference on
I and equal weights) achieved satisfactory results. This strategy Ubiquitous computing systems
can be used to reduce the drawback of cold start problem. In
addition, the user’s survey suggests that systems of this nature are [8] Burke, R. 2000. Knowledge-Based Recommender Systems.
interesting to users. In Encyclopedia of Library and Information Systems, no. 32,
180-200
In this moment, we cannot generalize our performance results
(dependence of users and the images corpus). However, it serves [9] Albanese, M., d'Acierno, A., Moscato, V., Persia, F. and
as a good indication of the quality of the prototype Picariello, A. 2011. A Multimedia Semantic Recommender
recommendation. Analysis of these results was limited to the size System in Cultural Heritage Applications. In Semantic
of the corpus (655 photos, with 335 from two tourist areas of Computing (ICSC), 2011 Fifth IEEE International
Fortaleza, city in Brazil). We think the increase of the database Conference on , pp.403-410, 18-21
could decrease the precision of the algorithm or present [10] Boutemedjet , S. and Ziou , D. 2010. Using Images in
concentration of photos in certain places. In such cases, new Context Aware Recommender Systems, In Proc of 1st Int'l
search filters and clustering algorithms should be used to solve Workshop on Adaptation, Personalization and
these challenges. Nevertheless, the results are already encouraging Recommendation in the Social-semantic Web
the construction of context-aware photo corpus of city sights (e.g.,
[11] Rost, M., Cramer, H., and Holmquist, L. E. 2011. Mobile
points of interest in host cities of 2014 World Cup).
exploration of geotagged photographs. In Personal and
As future work, we want to increase the image corpus by using an Ubiquitous Computing
evolution of the CoMMeDiA system [3]. We aim to ensure the
[12] Siersdorfer, S. and Sizov, S. 2009. Social recommender
accuracy of contextual information, and take benefits from the
systems for web 2.0 folksonomies. In Proceedings of the
automatic context acquisition. We also expect to evaluate
20th ACM conference on Hypertext and hypermedia (HT '09)
MMedia2U with users (tourist or otherwise) in a real mobile
situation. In a new version of the system, we would like to [13] Mcsherry, D. 2003. Similarity and Compromise. In
integrate clustering algorithms in the rank results, and allow users Proceedings of the Fifth International Conference on Case-
to add words of interest in order to refine the recommendations. Based Reasoning.