Revisiting User Information Needs in Aggregated Search

Abstract

Aggregated search interfaces are a common way to present web search results, mixing di erent types of results into one single result page. Although numerous e orts have been made to infer users' information needs in \standard" search, we know little about users' information needs within the context of aggregated search. This paper presents the outcomes of a survey of 117 respondents, investigating users' preferences for their type of search result (image, news, video) and their type of information need (informational, navigational and transactional). The survey reveals that users' result preferences di er based on their underlying information needs, suggesting that the taxonomy provided by Broder [1] requires updating to re ect user information needs in the context of aggregated search. For instance, respondents indicated a preference for diverse results (news and reviews about a particular software product) for navigational and transactional queries rather than a single result (the web page to download that software product).

1. INTRODUCTION AND BACKGROUND

Aggregated search is the technique of integrating search results from di erent verticals (e.g., web, image, video, news) on a single search result page so that users can access the increasingly diverse content available on the web. Aggregated search systems aim to facilitate users' access to \nonstandard" web results without having to perform separate searches in the respective verticals, which are source speci c sub-collections provided by search engines [ 13 ].

Throughout the evolution of web search, users' interaction with search results has been studied by many to improve the quality of the search results and the search experience. Efforts were (and are still being) made to understand users' information seeking process, based upon which several taxonomies describing users' behaviours have been proposed [ 1, 5, 6, 9, 10, 11, 16 ].

For instance, in 2002, Broder [ 1 ] created a taxonomy of Presented at EuroHCIR2012. Copyright c 2012 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. web search, classifying users' information needs into three categories, namely, informational, navigational and transactional. For navigational search, the immediate intent is to reach a particular site (e.g., BBC Homepage); for informational search, the intent is to acquire some information likely to be contained in one or more web pages (e.g., global warming); and nally, for transactional search, the intent is to perform some web-mediated activity (e.g., download, purchase).

Others such as Lindley et al. [ 16 ] looked at why people search or go online and identi ed ve main web activities: respite, orienting, opportunistic use, purposeful use and lean-back internet. An example of a respite activity is when people use the web to take a break at work, or through a mobile phone to occupy themselves while waiting. Similarly, Chew et al. [ 10 ] explored the contextual and behavioural details of users' interaction with web-based images as they occur in the course of everyday life, showing that users interact with image results as these help creating connections to other people and remote places, or re ecting on the past.

While there is a substantial body of work on understanding users' information needs and browsing activities in \standard" search, far less is known about these within the context of aggregated search. For instance, it is not clear if the existing taxonomies on information needs for \standard" search hold in an aggregated search scenario. In aggregated search, search results may originate from di erent media (e.g., images, maps) or may be of di erent genres (e.g., news, blogs). This may have an e ect on the way users interact with the results, and a ect their preferences for the types of results. A study in [ 15 ] investigated the former, but the latter remains largely unexplored. For instance, it is not known whether for navigational queries, users prefer to view a speci c website, as would be implied by [ 1 ]. A negative answer would mean that a revisit of Broder's three-main-categories of information needs is needed. Also, building an awareness of web activities in aggregated search, which cut across domains, media types and applications, can highlight important details when designing for interactions with the web [ 16 ].

The focus of this short paper is, therefore, two-fold: (1) to investigate the preference of search results sought by the users; and (2) to investigate the existing frameworks of web activities within the context of aggregated search. For this purpose, users' preferences for results of several media types and genres are investigated. Furthermore, since Broder's taxonomy has been heavily used (e.g. [ 3, 7, 9, 15 ]) we focus on the now classic informational, navigational and transactional categories. We nonetheless aim to extend this work with other taxonomies (e.g., ODP1) in future work. This paper makes the following contributions: (1) Investigates users' preference for search results (media and genres) for informational, navigational and transactional search tasks; and (2) Provides empirical evidence to support the need for updating the above three categories within the context of aggregated search.

We present the results of a survey that investigated users' preferences for results of di erent media types and genres, as answers to informational, navigational and transactional queries.

2. STUDY

A survey containing sixteen questions (4 background questions and 12 search task questions) was distributed on various social networks. The survey allowed us to reach a large and diverse enough number of users, and is a common way to elicit user perceptions and preferences [ 4, 8 ]. A total of 117 respondents completed the survey, of which 60 were female and 54 male; the remaining 3 did not disclose their gender. The respondents' age varied between 20-59 years (mean 29). Geographically, respondents were distributed across the US and Canada (3%), Europe (34%), Asia (62%) and Africa (1%). Most respondents were familiar with search engines and used them frequently. 2.1

Task

The aim of the survey was to elicit users' preferences for the types (media, genres) of search results for informational, navigational and transactional search tasks. To this end, we designed four search topics2 for each of these three categories. The list of topics for each category is listed in Table 1. In total, there were twelve questions for each respondent to answer. The orders of the questions were rotated to minimise ordering bias.

We designed topics that could be understood universally (e.g, global warming, checking emails, buying dvd, software download). Furthermore, the topics were devised to t the informational, navigational and transactional categories. Therefore, we did not manipulate topics to suit speci c media or genre. For instance, for the topic global warming, some people may want to read the latest news about global warming, some others may want to view pictures of melting icebergs, while some others may want to watch a documentary on global warming. Therefore this topic does not have an implicit type intent (e.g. image) but requires the gathering of information (informational search task) from many web pages; it is expected that users will look for multiple results to satisfy the corresponding information need. However, it will depend on users which result types (image, news, video, etc) they prefer to view { only news articles, few pictures, or a combination of both. 2.2

Procedure

For each search topic, the respondents were given ve choices, namely, web, news, image, video and other results3. The re

1http://www.dmoz.org/

2A search topic describes a search task scenario. The concept of a search task scenario was inspired from [ 2 ]. 3The de nitions of these categories were not speci ed in the instructions and were left open to respondents' interpretation. spondents were allowed to select as many options as they desired. That is, they were allowed to select just `one' or `all' options, and therefore were not forced to provide a preference for all the choices listed. This allowed a more natural selection of choices, and hence reduced any design bias. In cases when the respondents selected more than one option, they were asked to rank the choices, by providing \1st", \2nd" ......,\5th" preference for each choice. For instance, if image, news and others were selected as choices, these had to be ranked in order of preference (e.g., 1st preference - news, 2nd preference { image, 3rd preference { others).

Figure 1, shows the screenshot of an example question with the preference options. Next, the outcomes of the survey are presented. 3.

OUTCOMES

As the data obtained from the survey was non-parametric, we report medians and the interquartile range for the preference scores. The results are reported in Table 2, which shows the median rank of each vertical by information need. Friedman tests were performed to estimate the signi cance of preference for the results types, among and across the three categories (navigational, informational and transactional). Finally, multiple Wilcoxon-tests were run in the post-hoc analyses while adjusting the p-values using the Bonferroni method. The outcomes from the post-hoc pair wise comparisons for navigational, informational and transactional categories are shown in Tables 3, 4 and 5 respectively. Each row in these tables indicates whether a particular result type was preferred over each of the other result types.

As can be seen in Table 2, most respondents indicated the `web page' as the most preferred type of results, when

Navigational Topics

1. When you wish to book tickets with British Airways, which results would be useful for you? 2. When you wish to nd an address from yellow pages, which results would be useful for you? 3. When you wish to check courses of a University, which results would be useful for you? 4. When you wish to check your email (e.g, gmail, hotmail, msn, etc), which results would be useful for you?

Informational Topics

5. When you wish to learn about salsa dance, which results would be useful for you? 6. When you wish to gather information about global warming, which results would be useful for you? 7. When you wish to learn on how to make a pancake, which results would be useful for you? 8. When you wish to know about 2011 budget, and how it e ected farmers, which results would be useful for you? Transactional Topics 9. When you wish to download a free software, which results would be useful for you? 10. When you wish to download a song for your iTunes library, which results would be useful for you? 11. When you wish to le a property complaint, which results would be useful for you? 12. When you wish to buy a DVD online, which results would be useful for you? compared to the other four types (image, video, news and others). The di erence was found to be signi cant for navigational, informational, and transactional cases (rows 1-4 in Tables 3, 4 and 5 ); thus suggesting that \standard" web results are the prime source of information sought by most users. After web results, news was the second most preferred type of results when compared to image, video and others (6th row in Table 2). For the navigational category, news results were signi cantly preferred over image, video and others results (rows 6, 8 and 9 in Table 3). However, video was equally preferred to news for informational and transactional categories (row 8 in Tables 4 and 5).

Finally, there is a trend for image and video results to come third in preference from respondents for most categories (4th and 5th rows in Table 2). However, post-hoc analyses suggest a signi cant di erence of preference for video and image over `other results' for all three categories (rows 7 and 10 in Tables 3, 4 and 5). In addition, video results were signi cantly preferred to image results for informational and transactional cases (row 5 in Tables 4 and 5), while no signi cant di erence was observed for the navigational case (row 5 in Table 3 ). Therefore, it is possible that users may prefer image results instead of video results in some cases, and video results in other cases. In addition, image and video being the third preference indicates that providing image and video results for all queries may not be appreciated by users.

In Tables 3 to 5, in only two occasions were the ranking of result types not signi cantly di erent: image-video for navigational, and news-video for informational information needs. This indicates that for navigational needs, neither image or video results are judged as important to users, backing up the results in Table 2, where both are ranked bottom. For informational information needs, both news and video were judged equally important to the search tasks, second only to web (Table 2).

DISCUSSION

The aim of our study was to investigate, via a survey, users' results preference for navigational, informational, and transactional search topics.

Overall, three key observations can be made from this survey. First, for all query categories, web results continue to be the prime source of information sought by users { 90% for navigational, 54% for informational and 85% for transactional { suggesting that for an aggregated search result page, web results should always be provided. This echoes the ndings of [ 14 ] where the importance of web results for aggregated result pages was demonstrated through the mining of query logs.

Second, there appears to be a di erence between the result preferences for navigational and transactional queries. From Broder [ 1 ], the corresponding information needs for these categories were identi ed to be focused (i.e., speci c website, download, etc). In contrast, our study suggests that users also prefer to view other results, and not just one (\to the point") result, or one type of result. More precisely, for the navigational search topics, in addition to web results, respondents also indicated a preference for news and video results. This may be due to the fact that, since an aggregated result page is often provided for most queries by modern search engines4, users are exposed to diverse results and as a consequence, results other than web have now gained prominence. However, whether providing diverse results for informational and transactional information needs facilitates task completion, and/or increases user satisfaction, requires further investigation.

Third, users' preferences for the `type' of results vary with the query category. For instance, for navigational and transactional search topics, web and news results seem to be preferred. The preference is more mixed for informational search topics, with image results least preferred. In itself, it is not surprising that users' preferences vary with query categories. However, concrete knowledge regarding which `types' of sought results are preferred would allow for more appropriate aggregation of the di erent verticals under consideration. Similar investigations were carried out in [ 12 ] by Sushmita et al. where, associations between query classi cations (e.g., arts, health, etc) and result types were indeed identi ed. Such knowledge may then be used by search systems, to present particular types of result for di erent queries, for example, a system may not present (or demote in importance) image results in response to an informational query.

4http://www.slideshare.net/rankabove/com-score

rankabove- nal

We presented the analysis of a survey of 117 respondents' preferences regarding the di erent types of results for navigational, informational, and transactional information needs. Although small in terms of the number of users and acknowledging the limitation of an online survey, interesting insights emerged from our investigation. The outcomes of the survey support the aggregated search paradigm, showing that users' preferences are for a diverse range of result types. The analysis also indicates a need to revisit the de nition of the three categories of information needs [ 1 ], within the context of aggregated search. This work initiates two future research questions: (1) What information needs exist within the context of aggregated search? and (2) How to identify suitable results satisfying those information needs? 6.

References

[1] A. Broder . A taxonomy of web search . Journal of SIGIR Forum , 2002 .

[2] P. Borlund . The IIR evaluation model: a framework for evaluation of interactive information retrieval systems . JASIST , 2003 .

[3] L.A. Granka , T. Joachims & G. Gay, Eye-tracking analysis of user behavior in WWW search , SIGIR, 2004 .

[4] S.A. Grandhi , Q. Jones & S. Karam, Sharing the big apple: a survey study of people, place and locatability , SIGCHI , 2005 .

[5] M. Kellar , C. Watters & M. Shepherd , A Goal-based Classi cation of Web Information Tasks , ASIST 2006 .

[6] H. Dai , L. Zhao , Z. Nie , J.-R. Wen , L. Wang & Y. Li , Detecting online commercial intention , WWW , 2006 .

[7] B.J. Jansen , D.L. Booth & A. Spink, Determining the user intent of web search engine queries , WWW , 2007 .

[8] M.R. Morris , A survey of collaborative web search practices , SIGCHI , 2008 .

[9] B.J. Jansen , D.L. Booth & A. Spink, Determining the informational, navigational, and transactional intent of Web queries , IP&M, 2008 .

[10] B. Chew , J.A. Rode and A. Sellen , Understanding the Everyday Use of Images on the Web , NordiCHI, 2008

[11] S. Stamou & L. Kozanidis Impact of search results on user queries , WSDM , 2009 .

[12] S. Sushmita , H. Joho , M. Lalmas & J.M. Jose, Understanding domain \relevance" in web search . WSSP at WWW , 2009 .

[13] J. Arguello , F. Diaz , J. Callan & J.-F. Crespo , Sources of evidence for vertical selection . SIGIR , 2009 .

[14] S. Sushmita , B. Piwowarski & M. Lalmas , Dynamics of Domains and Genre Intent, AIRS , 2010 .

[15] S. Sushmita , H. Joho , M. Lalmas & R. Villa , Factors a ecting click through behavior in aggregated interface , CIKM , 2010 .

[16] S.E. Lindley , S. Meek , A. Sellen & R. Harper , \ It's simply integral to what I do" enquiries into how the web is weaved into everyday life , WWW , 2012 .