We discuss the issue of evaluating our context-based personalized mobile search approach with a methodology based on a combination of two evaluation approaches: context simulation and user study. Our personalized approach aims at exploiting some context-aware user profiles through a personalized score to re-rank initial search results obtained from a standard search system. We use Yahoo!'s open search web services platform BOSS 1 as a baseline. The context simulation allows us to simulate user locations and their related user interests. The user study involves real users who give their relevance judgments to the top 20 documents returned by yahoo and by our approach through an assessment tool 2 available on the web platform OSIRIM . The experimental results show the effectiveness of our personalized approach according to the proposed evaluation protocol.
H.3.3 [Information Search and Retrieval]: Relevance
feedback
(location, time and interests), such systems are faced to a
new challenge for IR, that is how those contextual data can
enhance user satisfaction. Another important issue is how to
evaluate the strategies and techniques involved in these new
systems. It is commonly accepted that the traditional
evaluation methodologies used in TREC, CLEF and INEX
campaigns are not always suitable for considering the
contextual dimensions in the information access process. Indeed,
laboratory-based or system oriented evaluation is challenged
by the presence of contextual dimensions such as user profile
or environment which significantly impact on the relevance
judgments or usefulness ratings made by the end user [
As an initial approach, yet allowing meaningful observations, we present here, the evaluation protocol aiming to evaluate empirically the performance of a novel context-based personalized mobile search system. For this purpose, we compare the performance of retrieval: without personalization and with personalization. We compare our approach to the results obtained from yahoo BOSS web search service, which did not implement itself any personalization capability. This paper discusses the methodology adopted and presents the results obtained. We first briefly survey IR evaluation methodologies in mobile contexts (Sec. 2). We then presents our approach for mobile search personalization, and introduce our contextual IR evaluation protocol (Sect. 3). Finally, we conclude and give perspectives for future works.
Context-awareness in mobile IR focuses on context models
including user profiles and environmental data (time,
location, near persons, device and networks). The
state-of-theart highlights that significative theoretical and
technological progress has been achieved in this area over the last
few years, encouraged by the growing interest to co-located
human-human communications and large scale location-based
applications ([
In this section, we first introduce our context-based personalized approach for mobile search, we then present our evaluation protocol devoted for our proposed approach.
Our context-aware approach to personalize search results
for mobile users [
Ks (t) =
1 |Drs| d∈Drs wtd .
(1)
Ks is matched with each concept cj of the ODP3 ontology
→
represented by single term vector cj using the cosine
similarity measure. The scores of the obtained concepts are
propagated over the semantic links as explained in [
In the absence of a standard evaluation framework, a formal evaluation of contextualization techniques may require a significant amount of extra feedback from users in order to measure how much better a retrieval system can perform with the proposed techniques than without them. In this case, the standard evaluation measures from the IR field require the availability of manual content ratings with respect to query relevance and specific user preference (i.e., constrained to the context of his search). For this aim we build a testbed consisting of a search space corpus, a set of queries, and a set of hypothetic context situations. A user study was conducted, participants were asked to provide ratings, in a blind test, for two retrieval scenarios: 1) top 20 documents returned by Yahoo BOSS, 2) top 20 documents returned by our personalized approach. In the following, we describe our experimental data sets and our evaluation protocol.
Since the contextualization techniques are applied as the time goes, we have defined a set of six short use cases as part of the evaluation setup. Each use case is composed of a set of queries within a given geographical context, and a narrative describing the relevance of a document regarding a query and a geographical context. We have simulated a set of six geographical contexts defined by a location type (zoo, music store, cinema, library, garden and museum). We have created a set of totally 25 different queries, 5 queries belonging to each geographical context. Since mobile search queries are known to be short (and thus ambiguous), our queries are generally short (query length ≤ 3) and some of them are consequently ambiguous (eg. jaguar ) and are tested within different geographical contexts (eg. the query ”water lilies” is tested within the two contexts ”garden” and ”museum”), totalizing a number of 30 queries within the six contexts. Our goal was to verify whether the consideration of geographical contexts and user profiles can enhance the performance of the search engine to respond to such ambiguous queries. Table 1 gives an example of the use case of the context museum.
The document collection consists of a set of about 3750 web pages retrieved from the web by yahoo BOSS as response to our set of queries. It is built by collecting the 150 first retrieved documents per query.
The user profiles are integrated in the evaluation strategy according to a simulation algorithm that generates them using hypothetic user interactions for each query. They are constructed based on a manual judgments of the <query, narrative, document> tuples for all the document in the collection. These, so built profiles, simulate user click-through data.
Our experimental design consists of evaluating the
effectiveness of our personalized approach when using the user profile
in the IR model over a sequence of user contexts. In the
absence of an initial score of the document results list of yahoo
BOSS, the re-ranking procedure is done based only in the
personalized score (ie. γ = 0 in equation 2). The evaluation
scenario is based on the k-fold cross validation like in [
In order to evaluate the performance of our proposed approach, a user study is conducted to compare the 20 top ranking output of our approach and of Yahoo BOSS. Using an assessment tool available on the web platform OSIRIM, six users who participated to the experiment were asked to judge each tuple <query, document, narrative> within the 20 top ranking output of both our approach and of Yahoo BOSS. Participants were unaware of the system they judge. Relevance judgments have been made using a three level relevance scale: relevant, partially relevant, or not relevant.
We evaluate the effectiveness of the personalized search over the six use cases and we compare the obtained results to the initial ones from Yahoo BOSS. To better estimate the quality of the search results at the top of the ranked list (since mobile users are unlikely to scroll long lists of retrieved items), we estimate the DCG@10 for all the queries. Figure 1 compares the effectiveness obtained by the initial yahoo search lists and the re-ranked ones obtained by our approach over all the queries. We observe that in general, our approach enhances the initial DCG@10 obtained by the standard search and improve the quality of the top search results lists. We have also computed the percentage of improvement of personalized search comparatively to the standard search computed at different cut-off points P@5, P@10, P@15 and P@20 averaged over all the queries. Results are presented in Table 2. Results prove that personalized search achieves higher retrieval precision of almost the queries in the six simulated contexts. Best performance are achieved by the personalized search in terms of average precision at different cut-off points achieving an improvement of 87,50% at P@5, 63,56% at P@10, 53,49% at P@15 and 50,92% at P@20 comparatively to Yahoo BOSS. However, precision improvement varies between queries, Figure 2 gives an example of this improvement variation between the queries of the context museum. This is probably due to the difference between the degree of ambiguity of the queries, which can not be explained only by the difference in query length. In fact, it depends also on the contents of the documents present in the collection.
In this paper we have presented our evaluation protocol of a context-aware personalization approach for mobile search. It is based on a combination of context simulation and user study. More precisely, we exploit context simulation to create user contexts and profiles in one hand. On the other hand, we exploit Yahoo’s BOSS web search service and real user judgments, through a user study, to evaluate the search effectiveness of our approach comparatively to a standard search. We evaluated our approach according to the promuseum
woman with a parasol M17 M23 M24 M25 M21 posed evaluation protocol and show that it is effective. In future work, we plan to extend this protocol by using real user data provided from a search engine log file. Extending the protocol aims at testing the effectiveness of the personalized search based on real mobile search contexts and click-through data available in the log file.
The authors acknowledge the support of the project QUAERO, directed by OSEO agency, France, and thank PhD students at IRIT for their participation in the experiment.