=Paper=
{{Paper
|id=Vol-512/paper-1
|storemode=property
|title=Demonstration of Improved Search Result Relevancy Using Real-Time Implicit Relevance Feedback
|pdfUrl=https://ceur-ws.org/Vol-512/paper01.pdf
|volume=Vol-512
|dblpUrl=https://dblp.org/rec/conf/sigir/CramerWH09
}}
==Demonstration of Improved Search Result Relevancy Using Real-Time Implicit Relevance Feedback==
https://ceur-ws.org/Vol-512/paper01.pdf
Demonstration of Improved Search Result Relevancy
Using Real-Time Implicit Relevance Feedback
David Hardtke Mike Wertheim Mark Cramer
Surf Canyon Surf Canyon Surf Canyon
Incorporated Incorporated Incorporated
274 14th St. 274 14th St. 274 14th St.
Oakland, CA 94612 Oakland, CA 94612 Oakland, CA 94612
hardtke@surfcanyon.com mikew@surfcanyon.com mcramer@surfcanyon.com
ABSTRACT to infer the relevance or non-relevance of documents. Many
Surf Canyon has developed real-time implicit personaliza- different user behavior signals can contribute to a proba-
tion technology for web search and implemented the tech- bilistic evaluation of document relevance. Explicit docu-
nology in a browser extension that can dynamically mod- ment relevance determinations are more accurate, but im-
ify search engine results pages (Google, Yahoo!, and Live plicit relevance determinations are more easily obtained as
Search). A combination of explicit (queries, reformulations) they require no additional user effort.
and implicit (clickthroughs, skips, page reads, etc.) user
signals are used to construct a model of instantaneous user 2. IMPLICIT SIGNALS AND USER INFOR-
intent. This user intent model is combined with the ini- MATION NEED
tial search result rankings in order to present recommended
With the large, open nature of the World Wide Web it is
search results to the user as well as to reorder subsequent
very difficult to evaluate the quality of search engine algo-
search engine results pages after the initial page. This pa-
rithms using explicit human evaluators. Hence, there have
per will use data from the first three months of Surf Canyon
been numerous investigations into using implicit user sig-
usage to show that a user intent model built from implicit
nals for evaluation and optimization of search engine quality.
user signals can dramatically improve the relevancy of search
Several studies have investigated the extent to which a click-
results.
through on a specific search engine result can be interpreted
as a user indication of document relevancy (for a review see
Keywords [3]). The primary issue involving clickthrough data is that
Implicit Relevance Feedback, Personalization, Adaptive Search users are most likely to click on higher ranked documents
System because they tend to read the SERP (search engine results
page) from top to bottom. Additionally, users trust that
a search engine places the most relevant documents at the
1. INTRODUCTION highest positions on the SERP.
It has long since been demonstrated that explicit relevance Joachims et al used eye tracking studies combined with
feedback can improve both precision and recall in informa- manual relevance judgements to investigate the accuracy of
tion retrieval[1]. An initial query is used to retrieve a set of clickthrough data for implicit relevance feedback [4]. They
documents. The user is then asked to manually rate a sub- conclude that clickthrough data can be used to accurately
set of the documents as relevant or not relevant. The terms determine relative document relevancies. If, for instance,
appearing in the relevant document are then added to the a user clicks on a search result after skipping other search
initial query to produce a new query. Additionally, non- results, subsequent evaluation by human judges show that
relevant documents can be used to remove or de-emphasize in ∼80% of cases the clicked document is more relevant to
terms for the reformulated query. This process can be re- the query than the documents that were skipped.
peated iteratively, but it was found that after a few iterations In addition to clickthroughs, other user behaviors can be
very few new relevant documents are found [2]. related to document relevancy. Fox et al. used a browser
Explicit relevance feedback as described above requires ac- add-in to track user behavior for a volunteer sample of of-
tive user participation. An alternative method that does not fice workers[5]. In addition to tracking their search and web
require specific user participation is pseudo relevance feed- usage, the browser add-in would prompt the user for spe-
back. In this scheme, the top N documents from the initial cific relevance evaluations for pages they had visited. Using
query are assumed to be relevant. The important terms in the observed user behavior and subsequent relevance evalu-
these documents are then used to expand the original query. ations, they were able to correlate implicit user signals with
Implicit Relevance Feedback aims to improve the precision explicit user evaluations and determine what user signals
and recall of information retrieval by utilizing user actions are most likely to indicate document relevance. For pages
clicked by the user, the user indicated that they were either
satisfied or partially satisfied with the document nearly 70%
of the time. In the study, two other variables were found
SIGIR ’09, July 19-23, 2009, Boston, USA. to be most important for predicting user satisfaction with
Copyright is held by the author/owner(s). a result page visit. The first was the duration of time that
the user spent away from the SERP before returning – if 4. TECHNOLOGICAL DETAILS
the user was away from the SERP for a short period of time Surf Canyon’s technology can be used as both a tradi-
they tended to be dissatisfied with the document. The other tional web search engine and as a browser extension that dy-
important variable for predicting user satisfaction was the namically modifies the search results page from commercial
“Exit type” – users that closed the browser on a result page search engines (currently Google, Yahoo!, and Live Search).
tended to be satisfied with that result page. The impor- The underlying algorithms in the two cases are mostly iden-
tant outcome of this and other studies is that implicit user tical. As the data presented was gathered using the browser
behavior can be used instead of explicit user feedback to extension, we will describe that here.
determine the user’s information need. Surf Canyon’s browser extension was publicly launched
on February 19, 2008. From that point forward visitors to
the Surf Canyon website2 were invited to download a small
3. IMPLICIT REAL-TIME PERSONALIZA- piece of free software that is installed in their browser. The
TION software works with both Internet Explorer and Firefox. Al-
As discussed in the previous section, it has been shown though the implementation differs for the two browsers, the
that implicit user behavior can often infer satisfaction with functionality is identical.
visited results pages. The goal of the Surf Canyon technol- Internet Explorer leads in all current studies of web browser
ogy is to use implicit user behavior to predict which unseen market share with March 2008 market share estimated be-
documents in a collection are most relevant to the user and tween 60% and 90%. Among users of the Surf Canyon
to recommend these documents to the user. browser extension, however, about 75% use Firefox. Among
Shen, Tan, and Zhai1 have investigated context-sensitive users who merely visit the extension download page, the
adaptive information retrieval systems [6]. They use both breakdown by browser type is nearly 50/50. Part of the
clickthrough information and query history information to skew towards Firefox in both website visitors and users of the
update the retrieval and ranking algorithm. A TREC collec- product can be attributed to the fact that marketing of the
tion was used since manual relevancy judgements are avail- product has been mainly via technology blogs. Readers of
able. They built an adaptive search interface to this collec- technology blogs are more likely to use operating systems for
tion, and had 3 volunteers conduct searches on 30 relatively which Internet Explorer is not available (e.g. Mac, Linux).
difficult TREC topics. The users could query, re-query, ex- Additionally, we speculate that Firefox may be more preva-
amine document summaries, and examine documents. To lent among readers of technology blogs. The difference be-
quantify the retrieval algorithms, they used Mean Average tween the fraction of visitors to the site using Firefox (∼50%)
Precision (MAP) or Precision at 20 documents. As these and the fraction of people who install and use the product
were difficult TREC topics, users submitted multiple queries using Firefox (∼75%) is likely due to the more widespread
for each topic. They found that including query history acceptance towards browser extensions in the Firefox com-
produced a marginal improvement in MAP, while use of munity. The Firefox browser was specifically designed to
clickthrough information produced dramatic increases (up have minimal core functionality augmented by browser add-
to nearly 100%) in MAP. ons submitted by the developer community. The technolo-
Shen et al. also built an experimental adaptive search in- gies used to implement Internet Explorer browser extensions
terface called UCAIR (User-Centered Adaptive Information are also often used to distribute malware so there may be a
Retrieval) [7]. Their client-side search agent has the capabil- higher level of distrust among IE users.
ity of automatic query reformulation and active reranking of Once the browser extension is installed, the user never
unseen search results based on a context driven user model. needs to visit the company web site again to use the prod-
They evaluated their system by asking 6 graduate students uct. The user enters a Google, Yahoo!, or Live Search web
to work on TREC topic distillation tasks. At the end of search query just as they would for any search (using either
each topic, the volunteers were asked to manually evaluate the search bar built into the browser or by navigating to
the relevance of 30 top ranked search results displayed by the the URL of the search engine). After the initial query, the
system. The top results shown are mixed between Google search engine results page is returned exactly as it would be
rankings and UCAIR rankings (some results overlap), and were Surf Canyon not installed (for most users who have not
the evaluators could not distinguish the two. UCAIR rank- specified otherwise, the default number of search results is
ings show a 20% increase in precision for the top 20 results. 10). Two minor modifications are made to the SERP. Small
The Surf Canyon browser extension represents the first bull’s eyes are placed next to the title hyperlink for each
attempt to integrate implicit relevance feedback directly into search result (see Figure 1). Also, the numbered links to
the major commercial search engines. Hence, we are able to subsequent search engine results pages at the bottom of the
evaluate this technology outside of controlled studies. From SERP are replaced by a single “More Results” link.
a research perspective, this is the first study to investigate The client side browser extension is used to communicate
this technology in the context of normal searches by normal with the central Surf Canyon servers and to dynamically
users. The drawback is that we have no chance to collect update the search engine results page. The personalization
a posteori relevancy judgements from the searchers or to algorithms currently reside on the Surf Canyon servers. This
conduct surveys to evaluate the user experience. We can, client-server architecture is used primarily to facilitate op-
however, quickly collect large amounts of user data in order timization of the algorithm and to support active research
to evaluate the technology. studies. Since web search patterns vary widely by user, the
best way to evaluate personalized search algorithms is to
1 vary the algorithms on the same set of users while main-
Shen, Tan, and Zhai are co-authors on one Surf Canyon
patent application but were not actively involved in the work
2
presented here http://www.surfcanyon.com
implicit relevance feedback - Google Search http://www.google.com/search?q=implicit+relevance+feedback&ie=ut...
Web Images Maps News Shopping Gmail more ▼ Sign in
Google Advanced Search
implicit relevance feedback Search Preferences
Reset recommendations
Web Results 1 - 10 of about 1,180,000 for implicit relevance feedback. (0.04 seconds)
Relevance feedback - Wikipedia, the free encyclopedia
The idea behind relevance feedback is to take the results that are initially ... Implicit
feedback is inferred from user behavior, such as noting which ...
en.wikipedia.org/wiki/Relevance_feedback - 19k - Cached - Similar pages
Implicit Relevance Feedback from Eye Movements (ResearchIndex)
We explore the use of eye movements as a source of implicit relevance feedback
information. We construct a controlled information retrieval experiment where ...
citeseer.ist.psu.edu/730378.html - 20k - Cached - Similar pages
Click data as implicit relevance feedback in web search
In this article, we address three issues related to using click data as implicit relevance
feedback: (1) How click data beyond the search results page might ...
portal.acm.org/citation.cfm?id=1224561.1224720 - Similar pages
Surf Canyon recommends 3 search results:
Using Implicit Relevance Feedback in a Web (ResearchIndex)
The explosive growth of information on the World Wide Web demands effective intelligent
search and filtering methods. Consequently, techniques have been ...
citeseer.ist.psu.edu/572595.html - 20k - Cached - Similar pages
More results from citeseer.ist.psu.edu »
Implicit relevance feedback in interactive music (from page 2)
This paper presents methods for correlating a human performer and a synthetic
accompaniment based on Implicit Relevance Feedback (IRF) using Graugaard's ...
portal.acm.org/citation.cfm?id=1164845 - Similar pages
More results from portal.acm.org »
Scalable Relevance Feedback Using Click-Through Data for Web Image ... (from page 2)
File Format: PDF/Adobe Acrobat - View as HTML
In this paper, we have presented a scalable relevance feedback. mechanism for web
image retrieval. Click-through data is used as. implicit relevance ...
research.microsoft.com/users/leizhang/Paper/ACMMM06-Cheng.pdf - Similar pages
More results from research.microsoft.com »
[PPT] LBSC 796/INFM 718R: Week 8 Relevance Feedback
1 of 3 03/20/2008 10:14 AM
Figure 1: A screenshot of the Google search result page with Surf Canyon installed. The third link was
selected by the user, leading to three recommended search results.
taining an identical user interface. With the client-server generated immediately below this search result.
architecture, the implicit relevance feedback algorithms can At the bottom of the 10 organic search results, there is a
be modified without alerting the user to any changes. Noth- link to get “More Results”. If the user requests the next page
ing fundamental prevents the technology from becoming ex- of results, all results shown on the second and subsequent
clusively client side. pages are determined using Surf Canyon’s instantaneous rel-
In addition to the ten results displayed by the search en- evancy algorithm. Unlike the default search engine behavior,
gine to the user, a larger set of results (typically 200) for subsequent pages of results are added to the existing page.
the same query is gathered by the server. With few excep- After selecting “More Results” links 1-20 are displayed in the
tions, the top 10 links in the larger result set are identical browser, with link 11 focused at the top of the window (the
to the results displayed by the search engine. While the user needs to scroll up to see links 1-10).
user reads the search result page, the back-end servers parse
the larger result set and prepare to respond to user actions. 5. ANALYSIS OF USER BEHAVIOR
Each user action on the search result page is sent to the
Most previous studies of Interactive Information Retrieval
back-end server (note that we are only using the user’s ac-
systems have used post-search user surveys to evaluate the
tions on the SERP for personalization and do not follow the
efficacy of the systems. These studies also tended to re-
user after they leave the SERP). For certain actions (select
cruit test subjects and use closed collections and/or spe-
a link, select a Surf Canyon bull’s eye, ask for more results)
cific research topics. The data presented here was collected
the back end server sends recommended search results to
from an anonymous (but not necessarily representative) set
the browser. The Surf Canyon real-time implicit personal-
of web surfers during the course of their interactions with
ization algorithm incorporates both the initial rank of the
the three leading search engines (Google, Yahoo, and Live
result and personalized instantaneous relevancies. The im-
Search). The majority of searches were conducted using
plicit feedback signals used to calculate the real-time search
Google. Where possible, we have analyzed the user data
result ranks are cumulative across all recent related queries
independently for each of the search engines and have not
by that user. The algorithm does not, however, utilize any
found any cases where the conclusions drawn from this study
long-term user profiling or collaborative filtering. The pre-
would differ depending on the user’s choice of search en-
cise details of the Surf Canyon algorithm are proprietary
gine. The total number of unique search queries analyzed
and are not important for the evaluation of the technology
was ∼700,000.
presented below. If an undisplayed result from the larger set
Since the users in this study were acquired primarily from
of results is deemed by Surf Canyon’s algorithm to be more
technology web blogs, their search behavior can be expected
relevant than other results displayed below the last selected
to be significantly different than the average web surfer.
link, it is shown as an indented recommendation below the
Thus, we cannot evaluate the real-time personalization tech-
last selected link.
nology by comparing to previous studies of web user be-
The resulting page is shown in Figure 1. Here, the user
havior. Also, since we have changed the appearance of the
entered a query for “implicit relevance feedback” on Google3 .
SERP and also dynamically modify the SERP, any metrics
Google returned 10 organic search results (only three of
calculated from our data cannot be directly compared to
which are displayed in Figure 1) of the 1,180,000 documents
historical data due to the different user interface.
in their web index that satisfy the query. The user then
Surf Canyon only shows recommendations after a bull’s
selected the third organic search result, a paper from an
eye or search result is selected. It is therefore interesting
ACM conference entitled “Click data as implicit relevance
to investigate how many actions a user makes for a given
feedback in web search”. Based on the implicit user signals
query as this tells us how frequently implicit personalization
(which include interactions with this SERP, recent similar
within the same query can be of benefit. Jansen and Spink
queries, and interactions with those results pages) the Surf
[8] found from a meta-analysis of search engine log studies
Canyon algorithm recommends three search results. These
that user interaction with the search engine results pages is
links were initially given a higher initial rank (> 10) by
decreasing. In 1997, 71% of searchers viewed beyond the first
the Google algorithm in response to the query “implicit rel-
page of search results. In 2002 only 27% of searchers looked
evance feedback”. The real-time personalization algorithm
past the first page of search results. There is a paucity of
has determined, however, that the three recommended links
data on the number of web pages visited per search. Jansen
are more pertinent to this user’s information need at this
and Spink [9] reported the mean number of web pages vis-
particular time than the results displayed by Google with
ited per query to be 2.5 for AllTheWeb searches in 2001,
initial ranks 4-10.
but they exclude queries where no pages were visited in this
Recommendations are also generated when a user clicks
estimate. Analysis of the AOL query logs from 2006 [10]
on the small bull’s eyes next to the link title. We assume
gives a mean number of web pages viewed per unique query
that a selection of a bull’s eye indicates that the linked doc-
of 0.97. For the current data sample, the mean number of
ument is similar to but not precisely what the user is looking
search results visited is 0.56. The comparatively low num-
for. For the analysis below, up to three recommendations
ber of search results that were selected in the current study
are generated for each link selection or bull’s eye selection.
has multiple partial explanations. The search results page
Unless the user specifically removes recommended search re-
now contains multiple additional links (news, videos) that
sults by clicking on the bull’s eye or by clicking the close box,
are not counted in this study. Additionally, the information
they remain displayed on the page. Recommendations can
that the user is looking for is often on the SERP (e.g. a
nest up to three levels deep – if the user clicks on the first
search for a restaurant often produces the map, phone num-
recommended result then up to three recommendations are
ber, and address). Search engines have replaced bookmarks
and direct URL typing for re-visiting web sites. For such
3
http://www.google.com navigational searches the user will have either one or zero
Fraction of Queries (%) the fact that users do not often click on more than one search
result as discussed above. The important point, however, is
60
that the Surf Canyon implicit relevance feedback technol-
50 ogy increases the click frequency by ∼80% compared to the
links presented without any real-time user-intent modelling.
40
The relative increase in clickthrough rate is constant (within
30 statistical errors) for all display positions even though the
absolute clickthrough rates rapidly drop as funciton of dis-
20
play position.
10
0 NONE 1 2 3 4 5+
Click Probability (%)
3 w/ Implicit Feedback
Number Of Search Results Selected 2.5 w/o Implicit Feedback
2
Figure 2: Distribution of total number of selections
per query. 1.5
1
0.5
clicks depending on whether the specific web page is listed
on the SERP. Additionally, it may be that the current sam- 0 1 2 3
ple of users is biased towards searchers who are less likely to
click on links. Recommended Link Position
Figure 2 shows the distribution of the total number of
selections per query. 62% of all queries lead to the selection Figure 3: Probability (%) that a recommended
of zero search results. Since Surf Canyon does nothing until search result will be clicked as a function of display
after the first selection, this number is intrinsic to the current position relative to the last selected search result.
users interacting with these particular search engines. A The red circles are for recommendations selected
recent study by Downey, Dumais and Horvitz also showed using Surf Canyon’s instantaneous relevancy algo-
that after a query the user’s next action is to re-query or end rithm, while the black triangles are for the random
the search session about half the time [11]. In our study, only control sample that does not incorporate relevance
12% of queries lead to more than one user selection. A goal feedback.
of implicit real-time personalization would be to decrease
direct query reformulation and to increase the number of Figure 4 shows the per query distribution of initial search
informational queries that lead to multiple selections. The result ranks for all selected search links in the current data
current data sample is insufficient to study whether this goal sample. The top 10 links are selected most frequently. Search
has been achieved. results beyond 10 are all displayed using Surf Canyon’s al-
In order to evaluate the implicit personalization technol- gorithm (either through a bull’s eye selection, a link selec-
ogy developed by Surf Canyon we chose to compare the ac- tion, or when the user selects more results). For the re-
tions of the same set of users with and without the implicit sults displayed by Surf Canyon (initial ranks > 10), the
personalization technology enabled. Our baseline control selection frequency follows a power-law distribution with
sample was created by randomly replacing recommended P (IR) = 38% ∗ IR−1.8 , where IR is the initial rank.
search results with random search results selected from among As Surf Canyon’s algorithm favors links with higher initial
the results with initial ranks 11-200. These “Random Rec- rank, the click frequency distribution does not fully reflect
ommendations” were only shown for 5% of the cases where the relevancy of the links as a function of initial rank. Fig-
recommendations were generated. The position (1, 2, or 3) ure 5 shows the probability that a shown recommendation
in the recommendation list was also random. These ran- is clicked as a function of the initial rank. This is only
dom recommendations were not necessarily poor, as they do for recommendations shown in the first position below the
come from the list of results generated by the search engine last selected link. After using Surf Canyon’s instantaneous
in response to the query. relevancy algorithm, this probability shows at most a weak
Figure 3 shows the click frequency for Surf Canyon rec- dependence on the initial rank of the search result. The dot-
ommendations as a function of the position of the recom- ted link shows the result of a linear regression to the data,
mendation relative to the last selected search result. Posi- P (IR) = 3.2 − (0.0025 ± 0.00101) ∗ IR. When sufficient data
tion 1 is immediately below the last selected search result. is available we will repeat the same analysis for “Random
Also shown are the click frequencies for “Random Recom- Recommendations” as that will give us a user-interface in-
mendations” placed at the same positions. In both cases, dependent estimate of the relative relevance for deep links
the frequency is relative to the total number of recommen- in the search result set before the application of the implicit
dations shown at that position. The increase in click rate feedback algorithms.
(∼60%) is constant within statistical uncertainties for all For the second and subsequent results pages, the browser
recommended link positions. Note that the recommenda- extension has complete control over all displayed search re-
tions are generated each time a user selects a link and are sults. For a short period of time we produced search re-
considered to be shown even if the user does not return to the sults pages that mixed Surf Canyon’s top ranked results
SERP. The low absolute click rates (3% or less) are due to with results having the top initial ranks from the search
engine. This procedure was proposed by Joachims as a way
to use clickthrough data to determine relative user prefer-
ence between two search engine retrieval algorithms [12].
Each time a user requests “More Results”, two lists are gen-
erated. The first list (SC) contains the remaining search
results as ranked by the Surf Canyon’s instantaneous rele-
vancy algorithm. The second list (IR) contains the same set
of results ranked by their initial display rank from the search
Click Frequency (%)
10 engine. The list of results shown to the user is such that the
Google w/ Surf Canyon top kSC and kIR results are displayed from each list, with
1 |kSC − kIR | < 1. Whenever kSC = kIR the next search re-
sult is taken from one of the lists chosen at random. Thus,
10
-1 the topmost search result on the second page will reflect
Surf Canyon’s ranking half the time and the initial search
10
-2 result order half the time. By mixing the search results
this way, the user will see, on average, an equal number of
-3 search results from each ranking algorithm in each position
10
0 50 100 150 200 on the page. The users have no way of determining which
Initial SERP Rank algorithm produced each search result. If the users select
more search results from one ranking algorithm compared
to the other ranking algorithm it demonstrates an absolute
Figure 4: Frequency per non-repeated search query user preference for the retrieval function that led to more
for link selection as a function of initial search result selections.
rank. Figure 6 shows the ratio of link clicks for the two retrieval
functions. IR is the retrieval function based on the result
rank returned from the search engine. SC is the retrieval
function incorporating Surf Canyon’s implicit relevance feed-
back technology. The ratio is plotted as a function of the
number of links selected previously for that query. Previ-
ously selected links are generally considered to be positive
content feedback. If, on the other had, no links were selected
then the algorithm bases its decision exclusively on negative
feedback indications (skipped links) and on the user intent
model that may have been developed for similar recent re-
lated queries.
Link Selection Ratio [SC/IR]
Rec. Link Click Prob. (%)
3.5 1.4
3 1.3
2.5 1.2
2 1.1
1.5 1
1 0.9
0.5
0.8 NONE 1 2-4 5+
0 20 40 60 80 100 120 140 160 180 200 # Previous Search Results Selected
Initial SERP Rank
Figure 6: Ratio of click frequency for second and
Figure 5: Probability that a displayed recommended subsequent search results page links ordered by
link is selected as a function of the initial search re- Surf Canyon’s Implicit Relevance Feedback algo-
sult rank. This data only include links from the first rithm (SC) compared to links ordered by the initial
position immediately below the last selected search search engine result rank (IR).
result.
We observe that, independent of the number of previous
user link selections in the same query, the number of clicks on
links from the relevance feedback algorithm is higher than
links displayed because of their higher initial rank. This
demonstrates an absolute user preference for the ranking al-
gorithm that utilizes implicit relevance feedback. Remark-
ably, the significant user preference for search results re- feedback. In SIGIR ’05, 2005.
trieved using the implicit feedback algorithm is also appar- [7] Xuehua Shen, Bin Tan, and ChengXiang Zhai.
ent when the user had zero positive clickthrough actions on Implicit user modelling for personalized search. In
the first 10 results. After skipping the first 10 results and CIKM ’05, 2005.
asking for a subsequent set of search links, the users are [8] B. Jansen and A. Spink. How are we searching the
∼35% more likely to click on the top ranked Surf Canyon world wide web?: a comparison of nine search engine
result compared to result # 11 from Google. Clearly, the transaction logs. Information Processing and
searcher is not so interested in search results produced by Management, 42(1):248–263, 2006.
the identical algorithm that produced the 10 skipped links [9] B. Jansen and A. Spink. An analysis of web documents
and an update of the user intent model for this query is retrieved and viewed. In The 4th International
appropriate. Conference on Internet Computing, pages 65–69, 2003.
[10] G. Pass, A. Chowdhury, and C. Torgeson. A picture of
6. CONCLUSIONS AND FUTURE DIREC- search. In The First International Conference on
TIONS Scalable Information Systems, 2006.
Surf Canyon is an interactive information retrieval system [11] D. Downey, S. Dumais, and E. Horvitz. Studies of web
that dynamically modifies the SERP from major search en- search with common and rare queries. In SIGIR ’07,
gines based on implicit relevance feedback. This was built 2007.
with the goal of relieving the growing user frustration with [12] T. Joachims. Unbiased evaluation of retrieval quality
the search experience and to help searchers “find what they using clickthrough data. In SIGIR Workshop on
need right now”. The system presents recommended search Mathematical/Formal Methods in Information
results based on an instantaneous user-intent model. By Retrieval, 2002.
comparing clickthrough rates, it was shown that real-time
implicit personalization can dramatically increase the rele-
vancy of presented search results.
Users of web search engines learn to think like the search
engines they are using. As an example, searchers tend to
select words with high IDF (inverse document frequency)
when formulating queries – they naturally select the rarest
terms that they can think of that would be in all documents
they desire. Excellent searchers can often formulate suffi-
ciently specific queries after multiple iterations such that
they eventually find what they need. Properly implemented
implicit relevance feedback would reduce the need for query
reformulations, but it should be noted that in the current
study most users had not yet adjusted their browsing habits
to the modified behavior of the search engine. By tracking
the current users in the future we hope to see changes in
user behavior that can further improve the utility of this
technology. As the user-intent model is cumulative, more
interaction will produce better recommendations once the
users learn to trust the system.
7. REFERENCES
[1] J.J. Rocchio. The Smart Retrieval System
Experiments in Automatic Document Processing.
Prentice Hall, 1971.
[2] D. Harman. Relevance feedback revisited. In
Proceedings of the Fifteenth International ACM SIGIR
Conference, pages 1–10, 1992.
[3] D. Kelly and J. Teevan. Implicit feedback for inferring
user preference: A bibliography. In SIGIR Forum
37(2), pages 18–28, 2003.
[4] Thorsten Joachims, Laura Granka, Bing Pan, Helene
Hembrooke, and Geri Gay. Accurately interpreting
clickthrough data as implicit feedback. In SIGIR ’05,
2005.
[5] Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan
Dumais, and Thomas White. Evaluating implicit
measures to improve web search. ACM Transactions
on Information Systems, 23(2):147–168, April 2005.
[6] Xuehua Shen, Bin Tan, and ChengXiang Zhai.
Context-sensitive information retrieval using implicit