Current Research in Supporting Complex Search Tasks

Abstract

Œere is broad consensus in the €eld of IR that search is complex in many use cases and applications, both on the Web and in domain speci€c collections, and both professionally and in our daily life. Yet our understanding of complex search tasks, in comparison to simple look up tasks, is fragmented at best. Œe workshop addresses many open research questions: What are the obvious use cases and applications of complex search? What are essential features of work tasks and search tasks to take into account? And how do these evolve over time? With a multitude of information, varying from introductory to specialized, and from authoritative to speculative or opinionated, when to show what sources of information? How does the information seeking process evolve and what are relevant di‚erences between di‚erent stages? With complex task and search process management, blending searching, browsing, and recommendations, and supporting exploratory search to sensemaking and analytics, UI and UX design pose an overconstrained challenge. How do we evaluate and compare approaches? Which measures should be taken into account? Supporting complex search tasks requires new collaborations across the €elds of CHI and IR, and the proposed workshop will bring together a diverse group of researchers to work together on one of the greatest challenges of our €eld.

INTRODUCTION

One of the current challenges in information access is supporting complex search tasks. A user’s understanding of the information need and the overall task develop as they interact with the system. Supporting the various stages of the task involves many aspects of the system, e.g. interface features, presentation of information, retrieving and ranking. Many search systems treat the search process as a series of identical steps of submiŠing a query and consulting documents. Yet information seeking research has shown that users go through di‚erent phases in their search sessions, from exploring CHIIR 2017 Workshop on Supporting Complex Search Tasks, Oslo, Norway. Copyright for the individual papers remains with the authors. Copying permiŠed for private and academic purposes. Œis volume is published and copyrighted by its editors. Published on CEUR-WS, Volume 1798, hŠp://ceur-ws.org/Vol-1798/. and identifying vague information needs, to focusing and re€ning their needs and search strategies, to €nalizing their search. To be able to support exploring and discovering strategies we need to understand the characteristics of di‚erent tasks including open-ended, leisure-focused sessions. Œis is a highly complex problem that touches upon and bridges areas of information seeking, interactive information retrieval, system-centered (ranking, evaluation) and user interface design.

Œe background for this workshop is derived from the Interactive Track (2014–2016) of the Social Book Search Lab at CLEF [ 11 ], which investigates scenarios with complex book search tasks and develops systems and interfaces that support the user through the di‚erent stages of their search process. But the aims of the workshop are broader and addresses all aspects of supporting complex search tasks. 2

GOALS AND OBJECTIVES Œe overall goal of the workshop is to create and foster an interdisciplinary forum where researchers can exchange and contribute to the development of alternative experiments and prototypes. Œe main aim is to beŠer understand how to support complex search tasks, addressing many open research questions to be explored, including:

Context What are the obvious use cases and applications of complex search? In what sense are these “complex”? What generic characteristic do they share? How can search become an integral part of its context, and the context integral part of search? Tasks What are essential features of work tasks and search tasks to take into account? And how do these evolve over time? How can complex tasks be decomposed into manageable sub-tasks, and partial results composed into comprehensive answers? How can we monitor and support task progress? Heterogeneous sources With a multitude of information, varying from introductory to specialized, and from authoritative to speculative or opinionated, when to show what sources of information? When to show more or other types of information than directly requested by the searcher? Do we know when the user has goŠen enough? Search process How does the information seeking process evolve and what are relevant di‚erences between di‚erent stages? What search tactics and search strategies are effective? How can we promote the use of e‚ective search strategies? How does the information need evolve and what are relevant success criteria for the end result and intermediate steps? How can we cast these as e‚ective complex queries, and how to (interactively) construct such queries? UI and UX Does the need of complex task and search process management, blending searching, browsing, and recommendations, and supporting exploratory search to sensemaking and analytics, make UI and UX design an overconstrained challenge? What a‚ordances are required and in what stage of the search process? How can we make the search process transparent to the user? How and when does the initiative shi‰ between system and user? How can interfaces support various user groups and their special needs? Evaluation How do we evaluate and compare approaches? How can we design realistic evaluation campaigns for interactive search? Can we carve out one or a range of generic aspects testable on a suitable benchmarks? Is there enough empirical evidence to ground simulated interactive search? What kind of novel retrieval models are needed to combine topical, contextual and preferential aspects? 3

WORKSHOP FORMAT

SCST 2017 was a half day workshop on supporting complex search tasks—a workshop proper where discussion is central, and all aŠendees are active participants.

Œe workshop started with a full round of introductions of all participants, making everyone feel welcome and part of the workshop. Œen, the workshop continued with two short keynotes to set the stage and ensure all aŠendees are on the same page. 3.1

Keynote Speakers

Mark Hall (Edge Hill University, UK) gave a keynote on “Where does it end? Complex Search Tasks and Evaluation” [ 12 ]. He takes an academic perspective and explores the blurring boundary between complex search tasks and the larger work tasks that motivate the search. Œis has important implications for what aspects of the process we should evaluate and how we do the evaluation in a meaningful and measurable way.

Jussi Karlgren (Gavagai, Sweden) gave a keynote on ”Complex Aspects of Seemingly Simple Information Needs” [ 15 ]. He presented the industrial views and discussed how the typical information needs of corporate customers are o‰en posed in short and basic questions but are surprisingly hard to formulate in meaningful queries, and answering them requires complex processes of curating and aggregating diverse and disparate data. 3.2

Paper Contributions

Œe workshop invited short paper contributions that are presented as posters. We received 11 submissions and accepted 9 (for an acceptance rate of 81%). Each paper was reviewed by at least three reviewers. Paper contributions are presented as a 1-minute boaster talk and as a poster during the interactive poster session.

RuŠer et al. [ 20 ] discuss a case study of a type of complex task that at face value is simple and straightforward, but turns out to be complex to resolve: how do you make a phone safe for a child. Œere is a lot of opinion online, many possibilities for actions, many variations in hardware and so‰ware, but ultimately no one clear and correct answer for everyday phone users.

Bogers et al. [ 4 ] report on the experiences and challenges in organizing the CHIC and SBS Interactive Tracks from 2013 to 2016 in the form of a list of important properties. Œese properties inform the design of new IIR evaluation campaigns and related researcher communities in ways that expand our understanding of information (seeking) behavior.

Koesten and Singh [ 17 ] focus on how a large governmental data portal in the UK supports users in conducting complex search tasks involving data, identify problems with the used interface, and discuss potential research directions to improve interfaces for complex data related search tasks.

Hoeber et al. [ 13 ] examine the use of exploratory search strategies for purposive sampling from large text collections. Œe use of exploratory search strategies that leverage visual analytics enables them to consider the relevance of the data in addition to more traditional sampling methods.

Egusa et al. [ 7 ] investigate the use of concept maps—graphical representations that allow people to represent their knowledge explicitly—to evaluate the e‚ects of interactive complex search. Œeir study showed a signi€cant change in the concepts maps produced before and a‰er executing a complex search.

Huurdeman [ 14 ] proposes a framework for the design of search user interfaces for complex search tasks. His framework covers three di‚erent types of features—personalizable features, informational features, and input & control features—and discusses the di‚erent stages of complex information seeking where these features are relevant.

Ventocilla et al. [ 21 ] suggest a boŠom-up approach to displaying and exploring relations and correlations in data sets. Using billiards as a metaphor, a graph-representation of (cor)relations in a data set are unfolded in directions based on the user’s choices. Œis provides an intuitive exploratory faceted search interface with quantitative analyses calculated at run-time.

Novin [ 19 ] argues that studies on complex search tasks should make their designs more context-based, which will make them more applicable to real-world scenarios, as well as more reproducible and falsi€able. Œe paper reviews literature on cognitive experiments that stress the importance if situation on actions and proposes a outside-in approach where the context is de€ned €rst, then the work task, a‰er which di‚erent experimental variables can be considered.

Arora and Jones [ 2 ] conducted a user study to investigate how users perceive relevance and importance of highlighted document fragments related to speci€c search topics, to beŠer understand how to generate e‚ective summaries of documents. Œe results provide insights on what types of information are e‚ective for satisfying information needs and why users €nd some parts more relevant than others. 3.3

Breakout Discussions

Œe second half of the workshop consists of 3-4 breakout groups, seeded from the open research questions (see §2) and the contributed papers, each group thoroughly prepared by a chair who guides the discussion, with examples from relevant IR evaluation campaigns such as the TREC Session and Tasks Tracks and the SBS Interactive and Suggestion Tracks, and from concrete examples of complex support systems with their UX and UI challenges. One of the topics will be an Interactive IR task that is planned for the TREC Tasks Track.1 Finally, the breakout groups report to the audience and a panel of experts, with continued discussion on what we have learned. A report of the workshop including a summary of the break-out discussions will be published as a SIGIR Forum report.

Œe discussion will continue during a social event in a more informal way over food and drinks, deep into the Oslo night. Œe organizers have gained a proud reputation for their open and inclusive workshops, leading to new research collaborations, other workshops, and new evaluation tracks.

Œe workshop will bring together a varied group of researchers —bridging CHI and IR in a natural way—with experience covering both user and system centered approaches, to work together on the problem and potential solutions, and identify the barriers to success and work on ways of addressing them. Œe format allows us deal with a relatively large number of participants while still preserving the interactive workshop character – the last edition at ECIR’15 had 41 paid registrations (and the head count was even higher on the day of the workshop). 4

RELATED WORKSHOPS

Œis workshop is a follow-up to the €rst SCST workshop at ECIR 2015 [ 8, 9 ] and is closely related to the Interactive Track of the CLEF Social Book Search Lab of 2015 and 2016 [ 10, 11 ]. Œe Interactive track is focused on the domain of book search, whereas the proposed workshop addresses issues around the search process and system interaction from a broader perspective.

Some of the organizers were involved in the SIGIR 2011 Workshop on “Entertain me” Supporting Complex Search Tasks [ 3 ] and in the spin-o‚ TREC Contextual Suggestion Track [ 5, 6 ]; in related discussion within the SWIRL ’12: Strategic Workshop on Information Retrieval in Lorne [ 1 ]; and the NSF Task-Based Information Search Systems Workshop [ 16 ]. Œere is a broad research agenda emerging that aŠracts interest from research in all areas of HCI and IR.

Œe workshop builds on the results of the earlier discussion, and through the CLEF Social Book Search Lab [ 18 ] has already been pushing this line of research with a range of user studies, novel user interfaces, and analysis of large scale social data. Œe workshop will be held to have a more focused discussion based on the results so far.

Œe workshop provides a comprehensive overview of current work on supporting complex tasks in a variety of seŠings, and fosters new collaboration within across the €elds of CHI and IR, on one of the most important topics in the coming years. 5

ACKNOWLEDGMENTS

We thank the CHIIR workshop chair Preben Hansen and the local organizers for their support. We also thank the programme commiŠee members for their timely and constructive reviews. We gratefully acknowledge EasyChair2 and CEUR-WS3 for providing platforms for submiŠing, reviewing and publishing the contributions of this workshop. 1Information about the 2017 Tasks Track is available at hŠp://www.cs.ucl.ac.uk/ tasks-track-2017/ 2See hŠp://easychair.org/ 3See hŠp://ceur-ws.org/

References

[1] J. Allan , B. Cro‰ , A. Mo‚at, and M. Sanderson . Frontiers, Challenges, and Opportunities for Information Retrieval: Report from SWIRL 2012 the Second Strategic Workshop on Information Retrieval in Lorne. SIGIR Forum , 46 ( 1 ): 2 - 32 , May 2012 .

[2] P. Arora and G. Jones . How do Users Perceive Information: Analyzing user feedback while annotating textual units within Web Documents . In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017 ), pages 7 - 10 , 2017 . URL hŠp://ceur-ws. org/ Vol- 1798 /.

[3] N. J. Belkin , C. L. Clarke , N. Gao , J. Kamps , and J. Karlgren . Report on the SIGIR Workshop on ”Entertain Me”: Supporting Complex Search Tasks. SIGIR Forum , 45 ( 2 ): 51 - 59 , Jan. 2012 .

[4] T. Bogers , M. Ga¨de, M. Hall, V. Petras , and M. Skov . Lessons Learned from the CHiC and SBS Interactive Tracks: A Wishlist for Interactive IR Evaluation . In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017 ), pages 11 - 14 , 2017 . URL hŠp://ceur-ws. org/ Vol- 1798 /.

[5] A. Dean-Hall , C. L. A. Clarke , J. Kamps , P. Œomas, and E. M. Voorhees . Overview of the TREC 2012 Contextual Suggestion Track . In E. M. Voorhees and L. P. Buckland, editors, Še Twenty-First Text REtrieval Conference Proceedings (TREC 2012 ). National Institute for Standards and Technology: NIST Special Publication 500-298 , 2013 .

[6] A. Dean-Hall , C. L. A. Clarke , J. Kamps , P. Œomas, N. Simon, and E. M. Voorhees . Overview of the TREC 2013 Contextual Suggestion Track . In E. M. Voorhees, editor, Še Twenty-Second Text REtrieval Conference (TREC 2013 ) Notebook . National Institute for Standards and Technology . NIST Special Publication 500-302 , 2014 .

[7] Y. Egusa , M. Takaku , and H. Saito . Evaluating Complex Interactive Searches Using Concept Maps . In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017 ), pages 15 - 17 , 2017 . URL hŠp://ceur-ws. org/ Vol- 1798 /.

[8] M. Ga ¨de, M. Hall, H. Huurdeman , J. Kamps , M. Koolen , M. Skov , E. Toms, and D. Walsh, editors. Proceedings of the €rst international workshop on Supporting Complex Search Tasks , volume 1338 of CEUR Workshop Proceedings , 2015 . CEURWS.org. URL hŠp://ceur-ws. org/ Vol- 1338 .

[9] M. Ga¨de , M. M. Hall , H. C. Huurdeman , J. Kamps , M. Koolen , M. Skov , E. Toms, and D. Walsh . Report on the First Workshop on Supporting Complex Search Tasks. SIGIR Forum , 49 ( 1 ): 50 - 56 , 2015 .

[10] M. Ga¨de , M. M. Hall , H. C. Huurdeman , J. Kamps , M. Koolen , M. Skov , E. Toms, and D. Walsh . Overview of the SBS 2015 Interactive Track . In L. Cappellato, N. Ferro , G. J. F. Jones , and E. SanJuan, editors, Working Notes of CLEF 2015 - Conference and Labs of the Evaluation forum , Toulouse, France, September 8- 11 , 2015 ., volume 1391 of CEUR Workshop Proceedings. CEURWS.org , 2015 . URL hŠp://ceur-ws. org/ Vol- 1391 / 78 -CR.pdf .

[11] M. Ga¨de , M. M. Hall , H. C. Huurdeman , J. Kamps , M. Koolen , M. Skov , T. Bogers , and D. Walsh . Overview of the SBS 2016 interactive track . In Working Notes of CLEF 2016 - Conference and Labs of the Evaluation forum , E´vora, Portugal, 5 - 8 September, 2016 ., volume 1609 of CEUR Workshop Proceedings , pages 1024 - 1038 . CEUR-WS.org, 2016 . URL hŠp://ceur-ws. org/ Vol- 1609 /16091024.pdf .

[12] M. M. Hall . Where does it end? Complex Search Tasks and Evaluation . In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017 ), page 5, 2017 . URL hŠp: //ceur-ws. org/ Vol- 1798 /.

[13] O. Hoeber , L. Hoeber , R. Snelgrove , and L. Wood . Interactively Producing Purposive Samples for ‹alitative Research using Exploratory Search . In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017 ), pages 18 - 20 , 2017 . URL hŠp://ceur-ws. org/ Vol- 1798 /.

[14] H. C. Huurdeman . Dynamic Compositions: Recombining Search User Interface Features for Supporting Complex Work Tasks . In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017 ), pages 21 - 24 , 2017 . URL hŠp:// ceur-ws. org/ Vol- 1798 /.

[15] J. Karlgren . Complex Aspects of Seemingly Simple Information Needs . In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017 ), page 6, 2017 . URL hŠp://ceur-ws. org/ Vol- 1798 /.

[16] D. Kelly , J. Arguello , and R. Capra . NSF Workshop on Taskbased Information Search Systems. SIGIR Forum , 47 ( 2 ): 116 - 127 , Jan. 2013 .

[17] L. Koesten and J. Singh . Searching Data Portals - More Complex Œan We Œought? In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017 ), pages 25 - 28 , 2017 . URL hŠp://ceur-ws. org/ Vol- 1798 /.

[18] M. Koolen , T. Bogers , M. Ga¨de, M. M. Hall , I. Hendrickx, H. C. Huurdeman , J. Kamps , M. Skov , S. Verberne , and D. Walsh . Overview of the CLEF 2016 Social Book Search Lab . In Experimental IR Meets Multilinguality, Multimodality, and Interaction - 7th International Conference of the CLEF Association, CLEF 2016 , E´vora, Portugal, September 5-8 , 2016 , Proceedings, volume 9822 of Lecture Notes in Computer Science, pages 351 - 370 . Springer, 2016 .

[19] A. Novin . Contextualizing the Cognition Crisis . In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017 ), pages 29 - 32 , 2017 . URL hŠp://ceur-ws. org/ Vol- 1798 /.

[20] S. Ru Šer, V. Blinzler , C. Ye , M. B. Twidale , and M. L. Wilson . Complex Search Task: How to Make a Phone Safe for a Child . In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017 ), pages 33 - 35 , 2017 . URL hŠp://ceur-ws. org/ Vol- 1798 /.

[21] E. Ventocilla , J. Bae , M. Riveiro , and A. Said . A Billiard Metaphor for Exploring Complex Graphs . In CHIIR 2017 Second Workshop on Supporting Complex Search Tasks (SCST 2017 ), pages 36 - 40 , 2017 . URL hŠp://ceur-ws. org/ Vol- 1798 /.