The main challenge of recommender systems is to be able to identify and recommend items that have a greater chance of meeting the interests of their users, which generally have a very subjective and heterogeneous nature. It is imperative, then, that recommender systems, from the identification of each user's profile, could recommend personalized items. However, the user's profile is not enough for the system to be able to completely identify the user's interests. The use of the system in a different context from the usual may cause an unsatisfactory result for the recommendation, requiring it to be adapted to a new context. This paper presents the MMedia2U, a prototype of a mobile photo recommender system that exploits the user's context and the context when the photo was created as a means to improve the recommendation. Three context dimensions area exploited: spatial, social and temporal. We describe the similarity measures used for each dimension and the results of the system evaluation by 13 users following a Gold Standard approach.
Systems, Context-Awareness, Ubiquitous
One of the most important challenges in Information Systems is
information overload. Recommender Systems try to cope with this
problem by helping people in retrieving information (ex: videos,
TV programs, routes, images, people, etc.) that may match their
preferences and intentions. Recommender Systems try to identify
items, from the information corpus, that have a greater chance to
meet the wishes of its users [
CARS-2012, September 9, 2012, Dublin, Ireland.
Copyright is held by the authors. improvements and home sensors technologies allow a better user’s context characterization and this information can be useful to improve recommendations.
In this scenario, this paper presents the Mobile Media to You (MMedia2U), a photo recommender system that suggests images previously annotated with contextual information. In order to execute the recommendation, MMedia2U explores the current context of the user acquired by his/her mobile device. This domain is interesting since there are available a large number of images on sharing sites such as Flickr1 and Picasa Web2. Many of these pictures are captured by mobile devices that store the location, date, time, and other contextual information which can be explored for recommendation.
With the dissemination of ubiquitous computing concepts,
context-awareness has become a very important research field. Its
ideas have been used to increase efficiency and usability of
Information Systems, particularly, those systems accessed from
mobile devices [
In the last years, some researchers have showed the feasibility of
implementing CARS (e.g., news, movies, music, and services).
Adomavicius et al. [
The second category comprises systems that use the contextual
information at the time that the user evaluates an item. In addition
to the ratings and characteristics of items and users, systems of
this category also take into account contextual information (e.g.,
location) during recommendation. The work described by
Adomavicius et al. [
Such systems depend, however, on a historical database
describing which items were evaluated by others users (or the user
himself/herself) in similar contexts to the current user’s context.
Another problem occurs when a new item is added to the
collection. As this item has not yet been used, it will be difficult to
recommend it in accordance with recommendation techniques
such as collaborative filtering (so-called cold start problem). The
work described in this paper differs from the studies
aforementioned since it takes into account both the context of
users and the context in which the items were created. The
hypothesis is that photos taken in a given context c may be of
interest to users who are in similar contexts to c. Then, we do not
need, in a first moment, of a historical database of
recommendation evaluations. MMedia2U uses a knowledge-based
recommendation method trying to avoid the cold start problem of
collaborative filtering [
In the system presented in this paper, users receive
recommendations of photos created in contexts similar to current
users’ context. This similarity computes three contextual
dimensions (spatial, social, and temporal). The system has as
target two types of users. The first type are those who are in an
unusual context (e.g., visiting a tourist sight for the first time) and
they can enjoy the pictures recommended to have references to
activities or new places to explore. The second group contains
users that have already been in this similar context and that the
recommended photos may give a new vision and perspective of
the situation they find themselves. Rost et al.[
A fundamental part in the development of a context-aware system is the definition of what information should compose the “context”, since elements that describe the contextual information depend on the system tasks, and on the system capacity to observe this information. This definition is associated with the creation of a context model, in which are established the elements that compose its description and how it should be represented (e.g., using ontologies, XML, objects). Fig 1 shows our context model represented as OWL-DL ontology. Our model has four dimensions: spatial (location and points of interest), social (e.g., personal information and activity being performed), temporal (date and time) and computational (mobile device).
Fig 1- Our Context Model.
In MMedia2U, these dimensions were explored in the acquisition
of knowledge about users and photos. These dimensions are
already exploited in context-aware management of photos [
Similarity measure is used in the system in order to retrieve those
photos created in contexts more similar to the user's current
context. The algorithm developed is an adaptation of traditional
knowledge-based techniques [
user U. The function simc is particular to each type of context and application domain.
Each context model dimension must have a method to calculate its similarity. The location similarity, for instance, can be calculated by measuring the distance between the place where the picture was taken and the current user’s location. The similarity for activity is calculated by comparing the activity or occasion that the user is found and the activity or occasion in which the image was generated. Some of the activities mapped in image shoots and their similarities are shown in Table 1.
0 0 1 0,5 (2)
1 0 0 0 0 1 0 0,5 0 0,5 1 0,5 Formula 2 is used with numeric context attributes. The similarity is defined by how close two values are.
simc (U,I )= 1−
V c (U )− V c (I ) max (c )− min (c ) In Formula 2, Vc(U) and Vc(I) represent, respectively, the values of context c for the user and the item; max(c) and min(c) represent, respectively, the maximum and minimum values for the compared attribute of context c (e.g., for c = hour of the day, min(c) = 0, and max(c) = 12). The similarity between dates can compare the various attributes related to the moment the photo was shot and the moment the user is found. Some attributes compared were: the hour of the day, day of the week and month of the year. The date attributes were compared individually to analyze the influence of each one (e.g., Has the similarity between hour of the day a greater influence on the choice of the user than the similarity of the day of the week?). The similarity between the months of the year can be calculated by Formula 2, adapting it to cyclic values (e.g., the distance between January and December is 1, instead of 11).
We designed the MMedia2U following a client-server architecture for mobile computing that is based on RESTful Web Services. This design allows mobile devices, even those with low processing capabilities, make use of our recommender service. Fig 2 presents the execution flow of the system.
Fig 2. Execution flow of the MMedia2U recommendation. In step 1, a mobile application is responsible for gathering user’s context. Some types of information (e.g., location) can be acquired from sensors (e.g., GPS). Other types rely on information passed by the user (e.g., current activity). In the second step, the mobile application accesses, from a HTTP call, the Web-Service provided by our recommender system, and it informs the user's current context. MMedia2U server receives the request, and performs an enrichment of user context data. The metadata stored in the repository of photos are scanned and compared with the current user’s context. In step 5, MMedia2U computes a photo ranking according to the results of similarity measures. The ranking contains the photos URLs and their metadata.
MMedia2U has a repository of photos for recommendation. These
photos need to be associated with contextual information in order
to be compared with the current context of the mobile users. The
photos should have as metadata the location where they were
taken and the activity of the photo’s author at the time of their
creation. At first, we expect to use photos from Web 2.0
applications, such as Flickr and Picasa Web. However, we find
many errors in the metadata of these photos (e.g., time, location).
In order to evaluate the recommendation method without
annotation errors, we built a repository of photos from images of
Picasa Web. Manually, we corrected and increased the metadata
returned by the Picasa Web Service. Then, we incorporated the
new metadata into the photo file by using IPTC and EXIF
headers. Examples of enrichment are the inference of the activity
from the photos description, and the day of week according to the
shot date. We hope the evolution of multimedia content
management systems, such as CoMMeDiA [
The mobile application was developed for the Android5 platform, compatible with devices that have version 2.2 or higher. Fig. 3 shows an execution flow of the mobile application.
Fig 3. Execution flow of the mobile application.
The user chooses the "activity/interest," then, the system captures the current context (location and date/time), and it sends to the server. This, in turn, returns a list of recommended images (third screen). If the user selects a photo, he can see its position and the distance between him and the place where the photo was captured. Clicking on the picture located on the map, its content is displayed in full screen, and its metadata can be also viewed.
Evaluating a recommendation system is a hard task, due to the property that an item’s relevancy has a strong personal nature and is complex to be measured. This difficulty is enhanced when exist a lack of historical evaluation data, which makes large-scale studies very costly and difficult to be run. In the case of CARS, the complexity is even bigger, since we need to range the possible contexts of real situations (i.e., places, daily situations, etc.). As we did not have a historical data about recommended photos, we created a Gold Standard, which consisted of photos evaluated by
users in a certain context. The objective was both to use the Gold Standard to compare the performance of our recommendation and to use it as historical data.
While building this Gold Standard, it was asked for a group of 13 users to evaluate photos from 8 different contexts, each one consisting of a stage of evaluation. In each stage, one context (e.g., shopping in some stores on the seaside of the city of Fortaleza6 during the evening) was presented to the user, who had to visualize a set of photos and choose those that seemed to be more appealing for him/her, taking into consideration the context he/she was suggested. The photos chosen by the users were included in the Gold Standard and provided the historical base in which the recommendation of MMedia2U was evaluated. The degree of success on recommendations was then evaluated by the ratio of chosen photos that are in the Gold Standard (e.g., if in a given combination of user and context, 10 photos are recommended and 6 of these are in the Gold Standard of the same combination, then the recommendation precision is 0.6). In each stage of evaluation, a mean of 100 photos were visualized by the users. 20 were taken in similar contexts to the one showed to the user and 80 were different in some dimensions of the context (e.g., same activity but very distant location). Five of 13 users didn’t know the place chosen as the location for the users (the seaside of Fortaleza). Ten users were Computer Science graduate students aging from 23 to 28. All of them use mobile phones every day. For some of the users, we have presented 8 photo collections, while for others we have only presented a subset of it, performing 66 simulations.
The objective of the experiment was to evaluate the following hypotheses: (H-I) It is possible to make satisfactory recommendations of georeferenced photos without prior knowledge of the user profile, considering only its current context; (H-II) The context in which the photos were taken is relevant in making recommendations; and (H-III) The usage of a context model considering various contextual dimensions may lead to an improved recommendation comparing to the result of one which uses only one context attribute (e.g., location).
In order to verify these hypotheses, first, the algorithm was run
with different weights for each dimension without a previous
training data. Another implementation used the weights obtained
by training the algorithm using a 7-fold method (same way of
adjusting the parameters) [
The last two rows of the table show the average precision when using the calculation of similarity of only one of the contextual dimensions. Combination I, which got the best results, was the one we assign twice the importance of Activity in relation to Location and four times in relation to temporal attributes. Comparing the results obtained without training (Combination I and Equal Weights) in relation to random experiments, one can see that it is possible to have much higher precision than random choices (agreeing with the hypothesis H-II and H-I). Moreover, the gain over the random method was not big when considering only one contextual dimension (e.g., only location or only activity), leading us to believe that a model of full context is essential for a good context-aware recommendation (hypothesis H-III).
Fig 4 shows the F measure (harmonic mean) analysis for the six recommendation algorithms. Equal weights and Combination I had the better results for Top 3. Regarding the Top 5, Top 10 and Top 20 lists, Combination I and Least Square were the most effective. For instance, Combination I (0,410 for Top 20) was two times better than Random algorithm (0,201).
We used the Student t-test in the precision values for the Top 3 obtained by the algorithms compared to the results obtained by random selection of photos. This showed that the combined use of contextual attributes is significantly better than randomization (95% degree of confidence, probability < 0.0001). In addition, the test showed that the use of only one attribute (activity or location, for example) is not significantly better than the random method. Finally, the comparison with the results of Combination I and Least Squares resulted in performance differences not statistically significant, which may not indicate the need for training to improve the precision of this kind of CARS.
Top 3
Top 5
Top 10
Top 20 0.42 0.44 0.47 0.25 0.29 0.34 Top 20 Top 10 Top 5 Top 3
Activity Localization
Random Combination I Least Squares Equal weights 0,0000 0,5000 Fig 4- Mean values of f-measure
At each stage of the experiments, a questionnaire was applied to the users so that relevant factors were investigated in the implementation of CARS for photos. One of the factors to be investigated was the relevance of a mobile system for photo recommendation and, in the case of existence of such system, if it would be interesting to recommend photos taken into account the current user’s context. He/she was asked whether, in this specific context, the user would like to receive recommendations of photos taken in a similar context. 74% of users answered “yes” for this question. When asked whether the recommendation of pictures generated in similar contexts would be interesting, the level of interest was 100%. Another point investigated was the relative weight of each contextual dimension. Each user was asked what contextual dimension, including location, activity, date and time; they would prefer to be taken into consideration to build the set of photos. Eight of the users said they think the proximity of the location of the photo is the most important factor to increase their interest. Five of the users responded that the activity associated with the picture portrayed by the activity, which he/she is playing at the moment is the most important factor. No users found the similarity between date and time the most important factor.
This work presented MMedia2U, CARS for contextual photos. During the system development, we specify a context model to cope this application domain and we adapted a knowledge-based technique to incorporate the context information. MMedia2U allows recommendation of photos even those that have never been evaluated by users. The recommendation is performed only from the context in which the photos were created. This allows users, without a history of use, to receive recommendations based on their current context. The recommendation mechanism was validated by the construction of a Gold Standard. The average precision achieved by the algorithm allowed us to conclude that, for the data used, context-awareness can bring gains in the photo recommendation compared to a random list. It is important to note that even weight combinations without training phases (the Comb I and equal weights) achieved satisfactory results. This strategy can be used to reduce the drawback of cold start problem. In addition, the user’s survey suggests that systems of this nature are interesting to users.
In this moment, we cannot generalize our performance results (dependence of users and the images corpus). However, it serves as a good indication of the quality of the prototype recommendation. Analysis of these results was limited to the size of the corpus (655 photos, with 335 from two tourist areas of Fortaleza, city in Brazil). We think the increase of the database could decrease the precision of the algorithm or present concentration of photos in certain places. In such cases, new search filters and clustering algorithms should be used to solve these challenges. Nevertheless, the results are already encouraging the construction of context-aware photo corpus of city sights (e.g., points of interest in host cities of 2014 World Cup).
As future work, we want to increase the image corpus by using an
evolution of the CoMMeDiA system [
This work is a partial result of the UbiStructure project supported by CNPq (MCT/CNPq 14/2011 - Universal) under grant number 481417/2011-7.