First Workshop on Supporting Complex Search Tasks

Abstract

There is broad consensus in the eld of IR that search is complex in many use cases and applications, both on the Web and in domain speci c collections, and both professionally and in our daily life. Yet our understanding of complex search tasks, in comparison to simple look up tasks, is fragmented at best. The workshop addressed the many open research questions: What are the obvious use cases and applications of complex search? What are essential features of work tasks and search tasks to take into account? And how do these evolve over time? With a multitude of information, varying from introductory to specialized, and from authoritative to speculative or opinionated, when to show what sources of information? How does the information seeking process evolve and what are relevant differences between di erent stages? With complex task and search process management, blending searching, browsing, and recommendations, and supporting exploratory search to sensemaking and analytics, UI and UX design pose an overconstrained challenge. How do we know that our approach is any good? Supporting complex search task requires new collaborations across the whole eld of IR, and the proposed workshop will bring together a diverse group of researchers to work together on one of the greatest challenges of our eld.

1. INTRODUCTION

One of the current challenges in information access is supporting complex search tasks. A user's understanding of the information need and the overall task develop as they interact with the system. Supporting the various stages of the task involves many aspects of the system, e.g. interface features, presentation of information, retrieving and ranking. Many search systems treat the search process as a series of identical steps of submitting a query and consulting docCopyright c 2015 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.

ECIR Supporting Complex Search Task Workshop ’15 Vienna, Austria Published on CEUR-WS: http://ceur-ws.org/Vol-1338/. uments. Yet information seeking research has shown that users go through di erent phases in their search sessions, from exploring and identifying vague information needs, to focusing and re ning their needs and search strategies, to nalizing their search. To be able to support exploring and discovering strategies we need to understand the characteristics of di erent tasks including open-ended, leisure-focused sessions. This is a highly complex problem that touches upon and bridges areas of information seeking, interactive information retrieval, system-centered (ranking, evaluation), user interface design.

The background for this workshop is derived from the CLEF/INEX Interactive Social Book Search Track (2014{ ) [ 8 ], which investigates scenarios with complex book search tasks and develops systems and interfaces that support the user through the di erent stages of their search process. Book search provides an excellent scenario to investigate these issues. Information needs in book search are highly complex, combining aspects of topical relevance (genre, subject), user relevance (background knowledge, reading level, preferences and interests) and social relevance (recommendations and opinions of friends and other trusted sources). Moreover, book search needs develop from vague notions of interest (books similar to X) to more speci c criteria (likable characters, academic treatment of topic, etc.) This change in the users needs, and the development of the tasks associated with those needs, demonstrates that current search systems provide little active support for such scenarios. Examples from the ISBS collection, ndings based on the user studies, and prototypes of information seeking stage sensitive search systems are available, and will be used to focus the discussion in the breakout groups. 2.

GOALS AND OBJECTIVES

The overall goal of the workshop is to create and foster an interdisciplinary forum where researchers can exchange and contribute to the development of alternative experiments and prototypes.

The main aim is to better understand how to support complex search tasks, addressing many open research questions to be explored, including: Context What are the obvious use cases and applications of complex search? In what sense are these \complex"? What generic characteristic do they share? How can search become an integral part of its context, and the context integral part of search? Tasks What are essential features of work tasks and search tasks to take into account? And how do these evolve over time? How do can complex tasks be decomposed into manageable sub-tasks, and partial results composed into comprehensive answers? How can we monitor and support task progress? Heterogeneous sources With a multitude of information, varying from introductory to specialized, and from authoritative to speculative or opinionated, when to show what sources of information? When to show more or other types of information than directly requested by the searcher? Do we know when the user has gotten enough? Search process How does the information seeking process evolve and what are relevant di erences between di erent stages? What search tactics and search strategies are e ective? How can we promote the use of e ective search strategies? How does the information need evolve and what are relevant success criteria for the end result and intermediate steps? How can we cast these as e ective complex queries, and how to (interactively) construct such queries? UI and UX Does the need of complex task and search process management, blending searching, browsing, and recommendations, and supporting exploratory search to sense-making and analytics, make UI and UX design an overconstrained challenge? What a ordances are required and in what stage of the search process? How can we make the search process transparent to the user? How and when does the initiative shift between system and user? Evaluation How do we know that our approach is any good? Can we carve out one or a range of generic aspects testable on a suitable benchmarks? Is there enough empirical evidence to ground simulated interactive search? What kind of novel retrieval models are needed to combine topical, contextual and preferential aspects?

FORMAT

SCST 2015 was a half day workshop on supporting complex search tasks|a work shop proper where discussion was central, and all attendees were active participants.

The workshop started with a keynote by Diane Kelly (University of North Carolina, Chapel Hill) to set the stage and ensure all attendees were on the same page. A small number of the short/position papers were selected for short oral presentation (10-15 minutes), all other papers had a 2 minute boaster, and all papers were presented as posters in an interactive poster session. The second half of the workshop consisted of 3-4 breakout groups, seeded from the open research questions (see x2) and the contributed papers, each group thoroughly prepared by a chair who guided the discussion, with examples from relevant IR evaluation campaigns such as the TREC Session and Tasks Tracks and the SBS Interactive and Suggestion Tracks. Finally, the breakout groups reported to the audience and a panel of experts, with continued discussion on what we learned, concrete plans for the next year, and a road-map for the longer term.

The workshop brought together a varied group of researchers with experience covering both user and system centered approaches, to work together on the problem and potential solutions, and identify the barriers to success and work on ways of addressing them. 4.

ACCEPTED PAPERS

We invited submissions of 3 page papers to be presented as posters. All submissions were peer reviewed by at least three reviewers. In this section we brie y describe each of the seven accepted papers.

Dori-Hacohen et al. [ 6 ] present searching on controversial topics as a complex search task. Challenges include how one searches for such views on such topics and in what way a system should engage users, by e.g. showing all the di erents views or censoring certain views.

Gunadi et al. [ 7 ] highlight the problems of distributed IR, such as dealing with heterogeneous document collections and information systems, which presents a challenge for systems to support users requiring information from multiple resources.

Balog [ 2 ] introduces the notion of task completion engines and argues for re-thinking the search experience. The author presents use cases such as travel planning and shopping and sketches an interface that integrates functionality to support users in the various steps of their complex tasks.

Walsh and Hall [ 14 ] discuss the scenario of users with no particular goal or information need, who engage the system with the aim to explore collections in the domain of cultural heritage, where the system helps them in the initial phase to discover areas or topics of interest.

Kumpulainen and Huurdeman [ 11 ] describe the relation between searcher's con dence in their search skill and understanding of the search topic and/or the system, focusing on those users with limited knowledge but high con dence. For such users, the system could shake up the search experience to help them formulate their search goals better and change their search behaviour.

Toms [ 13 ] argues that search tasks in and of themselves are often not complex, but that the complexity arises from the related work task for which the searches are conducted. The focus of the system should be on supporting search within the larger work task. The author also describes di erent kinds of support that a system could or should give.

Dean-Hall et al. [ 5 ] present an evaluation framework for Point-Of-Interest recommender systems and scenarios where users want personalised suggestions that takes many aspects of their context into account, such as geographical location, time of day, season as well as their interests and previous experiences. 5.

DISCUSSION AND CONCLUSIONS

This workshop is closely related to the INEX Interactive Social Book Search Track (ISBS) at CLEF 2014 [ 8 ] and CLEF 2015. The ISBS track is focused on the domain of book search, whereas the proposed workshop addressed issues around the search process and system interaction from a broader perspective. The ISBS track of CLEF'15 ran in a number of cycles, with the last and main cycle starting just after the workshop at ECIR'15.

Some of the organizers were involved in the SIGIR 2011 Workshop on "entertain me" Supporting Complex Search Tasks [ 3 ]; in related discussion within the SWIRL'12: Strategic Workshop on Information Retrieval in Lorne [ 1 ]; and the NSF Task-Based Information Search Systems Workshop [ 9 ]. There is a broad research agenda emerging that attracts interest from research in all area's of information retrieval.

The workshop built on the results of the earlier discussion, and through the CLEF/INEX SBS track [ 10 ] has already been pushing this line of research with a range of user studies, novel user interfaces, and analysis of large scale social data. The workshop was held to have a more focused discussion based on the results so far, and in time to inform new experiments running within the CLEF'15 Social Book Search Track [ 12 ].

The workshop provided a comprehensive overview of current work on supporting complex tasks in a variety of settings, and fostered new collaboration within our eld on one of the most important topics in the coming years.

ACKNOWLEDGMENTS

We thank the ECIR workshop chairs Guido Zuccon and Andras Benczur and the local organizers for their support.

References

[1] J. Allan , B. Croft , A . Mo at, and M. Sanderson . Frontiers, challenges, and opportunities for information retrieval: Report from SWIRL 2012 the second strategic workshop on information retrieval in Lorne . SIGIR Forum , 46 ( 1 ):2{ 32 , May 2012 . ISSN 0163- 5840 . doi: 10 .1145/2215676.2215678. URL http://doi.acm. org/ 10 .1145/2215676.2215678.

[2] K. Balog . Task-completion Engines: A Vision with a Plan . In Proceedings of the rst international workshop on Supporting Complex Search Tasks , volume 1338 of CEUR Workshop Proceedings. CEUR-WS.org , 2015 .

[3] N. J. Belkin , C. L. Clarke , N. Gao , J. Kamps , and J. Karlgren. Report on the sigir workshop on "entertain me": Supporting complex search tasks . SIGIR Forum , 45 ( 2 ): 51 { 59 , Jan . 2012 . ISSN 0163- 5840 . doi: 10 .1145/2093346.2093354. URL http://doi.acm. org/ 10 .1145/2093346.2093354.

[4] L. Cappellato , N. Ferro , M. Halvey , and W. Kraaij, editors. Working Notes for CLEF 2014 Conference, She eld, UK, September 15-18 , 2014 , volume 1180 of CEUR Workshop Proceedings , 2014 . CEUR-WS.org . URL http://ceur-ws. org/ Vol- 1180 .

[5] A. Dean-Hall , C. Clarke , J. Kamps , and J. Kiseleva . Online evaluation of point-of-interest recommendation systems . In Proceedings of the rst international workshop on Supporting Complex Search Tasks , volume 1338 of CEUR Workshop Proceedings. CEUR-WS.org , 2015 .

[6] S. Dori-Hacohen , E. Yom-Tov , and J. Allan . Navigating Controversy as a Complex Search Task . In Proceedings of the rst international workshop on Supporting Complex Search Tasks , volume 1338 of CEUR Workshop Proceedings. CEUR-WS.org , 2015 .

[7] E. Gunadi , T. Plumbaum , and S. Albayrak . Applied Distributed Information Retrieval in Enterprise Search . In Proceedings of the rst international workshop on Supporting Complex Search Tasks , volume 1338 of CEUR Workshop Proceedings. CEUR-WS.org , 2015 .

[8] M. M. Hall , H. C. Huurdeman , M. Koolen , M. Skov , and D. Walsh . Overview of the INEX 2014 interactive social book search track . In Cappellato et al. [4] , pages 480 { 493 . URL http://ceur-ws. org/ Vol- 1180 / CLEF2014wn -Inex-HallEt2014.pdf.

[9] D. Kelly , J. Arguello , and R. Capra . Nsf workshop on task-based information search systems. SIGIR Forum , 47 ( 2 ): 116 { 127 , Jan . 2013 . ISSN 0163- 5840 . doi: 10 . 1145/2568388.2568407. URL http://doi.acm. org/10 . 1145/2568388.2568407.

[10] M. Koolen , T. Bogers , J. Kamps , G. Kazai, and M. Preminger . Overview of the INEX 2014 social book search track . In Cappellato et al. [4] , pages 462 { 479 . URL http://ceur-ws. org/ Vol- 1180 / CLEF2014wn -Inex-KoolenEt2014.pdf.

[11] S. Kumpulainen and H. Huurdeman . Shaken, not steered { the value of shaking up the search process . In Proceedings of the rst international workshop on Supporting Complex Search Tasks , volume 1338 of CEUR Workshop Proceedings. CEUR-WS.org , 2015 .

[12] SBS. CLEF'15 Social Book Search track , 2015 . http: //social-book-search.humanities.uva.nl/.

[13] E. Toms . Complex Tools for Complex Tasks . In Proceedings of the rst international workshop on Supporting Complex Search Tasks , volume 1338 of CEUR Workshop Proceedings. CEUR-WS.org , 2015 .

[14] D. Walsh and M. M. Hall . Just looking around: Supporting casual users initial encounters with Digital Cultural Heritage . In Proceedings of the rst international workshop on Supporting Complex Search Tasks , volume 1338 of CEUR Workshop Proceedings. CEUR-WS.org , 2015 .