Shaken, Not Steered: The Value of Shaking Up the Search Process Sanna Kumpulainen Hugo Huurdeman University of Amsterdam University of Amsterdam Amsterdam, the Netherlands Amsterdam, the Netherlands s.w.kumpulainen@uva.nl h.c.huurdeman@uva.nl ABSTRACT However, the formulation of this focus does not always take The search engines of our times have brought ubiquitous access place. to information into the reach of nearly everyone. A wealth of In a large-scale study in a library context, Kuhlthau [17, p.68] information is just one click away, and streamlined search found that half of the observed students “did not show evidence engines have become increasingly efficient at looking up of reaching a focused perspective on their topic at any time information. However, the fact that it is simple to look up during their search process.” The formulation of a focus is information does not necessarily mean that it is easy to find the dependent on several factors. Focus formulation can only take material one really needs, especially in the context of complex place if people have built up enough background knowledge on tasks. At the early stages of the search process a novice searcher a topic [24]. The exploratory pre-focus and focus formulation has to create a focus in order to succeed in the task at hand, stages of search are therefore essential for developing rich requiring deep exploration and an understanding of the involved knowledge. topic. However, these novices may perform only superficial Employing interventions in the information seeking process is searches; worse still, they do not even realize it. Therefore, we one way to stimulate the critical exploration and use of argue that shallow information seeking needs a “shake-up”: information in educational settings. Different types of shallow information seeking ventures should be deepened by interventions are possible, ranging from group-based ‘disrupting’ the search process, especially of inexperienced users (information literacy) instructional sessions to counseling at the This paper discusses the potential value of shaking up the search process level of an individual. In this paper, we focus on process, and aims to stimulate discussion about the level of interventions in the information seeking process of individuals. support a searcher needs. These interventions should only be done when necessary, and at Categories and Subject Descriptors the “right” moment called a “Zone of Intervention”, which is a moment of increased uncertainty [17]. We distinguish between H.3.3 [Information Search and Retrieval]: Search process; H.5.2 two different aims of interventions. One aim may be to reduce [User Interfaces]: User-centered design; H.1.2 [User/Machine the uncertainty of a searcher. As Kuhlthau’s model indicates, in Systems]: Human information processing moments of increased uncertainty and doubt, careful intervention can improve a searcher’s performance, by providing General Terms thoughtful feedback and guidance in the process. In some cases, Design, Human Factors however, a searcher may be too certain. In those situations, an intervention could be aimed at increasing uncertainty. This type Keywords of intervention is inspired by the finding that many novice Information seeking, search process, stages, search systems searchers regard themselves as highly competent searchers, while their search strategies are poor [11]. This is only 1. INTRODUCTION exacerbated by the fact that current search systems generally In our daily lives, studies and work, information acquired from offer focused results without requiring much effort from a the Internet plays a pivotal role. We carry out tasks, ranging searcher. Hence, searchers might experience an information from simple lookup tasks to complex and advanced tasks, by seeking process that is too straightforward, needing some using omnipresent search engines. The more complex tasks are “shaking up”. performed in stages in which thoughts, feelings and actions evolve. For instance, university students working on a research This position paper takes the idea of the potentially shallow assignment might perform unfocused searches at the beginning search process of particularly novice searchers as its basis, and of their search process, while at some point they may formulate looks at ways to “shake-up” their search process. In the a focus, and perform more targeted searches [17, 24]. The following section, we discuss information seeking in the context students’ understanding of a topic evolves from vague to clear of complex tasks. Next, we define the concept of “a searcher in during the task process [24]. need”, before elaborating on tentative ways to disrupt searchers experiencing a too limited search process, in order to improve their task outcomes. ECIR Supporting Complex Search Task Workshop'15, Vienna, Austria. Copyright 2015 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. Published on CEUR-WS: http://ceur-ws.org/Vol-1338/ 2010 ACM 1-58113-000-0/00/0010 …$15.00. 2. THE EVOLVING INFORMATION They might be overconfident and resistant to take any help. Focus formulation enables assessing relevance, pertinence and SEEKING PROCESS IN COMPLEX what is enough to accomplish the task [11, 14]. Therefore, the TASKS searchers are neither able to assess relevance or usefulness of the Complex tasks are tasks in which the actor lacks understanding retrieved items nor when to stop the searching. They might be on the information needed, the task process and of the goals of happy just with few information items at hand that are the tasks [7]. In complex task accomplishment “understanding, “somewhat relevant” to their topic and terminate searching [12]. sense-making and problem formulation” are essential. Complex We claim that these searchers without any feelings of tasks require more complex types of information, involve uncertainty and with overconfidence are the searchers in need. learning and their goals and processes are vague [18]. Novices Overconfidence leads to straightforward workflows, following a are in this paper such actors that have not conducted the same predefined pattern and little exploration. The searcher is not task previously and have low domain knowledge level. actually looking for evidence to back up her ideas, but for a – Information seeking is not a one-shot querying and retrieval of possibly non-existing − correct answer using search engines. satisfying results but rather an evolving process including This kind of behavior leads to shallow use of information learning. According to Toms [23] the process delineates the instead of deep learning. This might be further strengthened by success of the outcome, and therefore research should focus on cognitive biases, namely the use of heuristics and anchoring the process and the sets of human actions. Kuhlthau’s [17] [16]. In anchoring the searcher sticks to the first idea found and Information Search Process Model (ISP) examined the process does not accept any opposite views. People also tend to avoid and is based on a series of field studies. The ISP model consists information that is against their attitudes and beliefs [9]. of six task stages. Each of these stages produces varying Interacting with information might be unpleasant and too hard information needs and searching. At the start of ISP the actors’ work and therefore a searcher might terminate the information understanding of the task is vague, their information needs are seeking too early. Thus, they might arrive to conclusions based unclear, and information searching is exploratory. After finding on limited information, and feel satisfied with that. Most current a focus, their notion of the task becomes clearer, information search engines may even increase cognitive biases by the needs more articulated and information searching more directed. employed personalization, leading to “filter bubbles”. Moreover, Vakkari [24] adapted this model to study students during an search systems may encourage overconfidence [16]. essay-writing task. His model consists of pre-focus, formulation There might be an inverse relation between confidence and the and post-focus phases. He found, that at the end phase almost all correctness of solutions: Davidson, Deuser, Sternberg [8] the students in his study were able to construct a focus. The showed that persons with high feelings of confidence tended to successful formulation of a focus also facilitates a transition arrive at incorrect solutions whereas individuals who felt they from the mere ability to identify facts to the ability to perform were far from solving the problems tended to give correct more abstract and deep analyses [21]. solutions. This supports our central thesis that if we “shake” the In the pre-focus phase, where exploration takes place, berry- searchers out of their confidence and comfort zone, they may picking information behavior occurs, which is an iterative and achieve better results. In the next section we explore whether adaptive search process that also incorporates the idea of there are ways to do this shake-up in actual search systems. collecting information objects as the search progresses over time. This explorative stage is needed in order to build an 4. SHAKING UP “A SEARCHER IN understanding of the task, task domain and the task goals. NEED” Contrary to the common point of view that information reduces Current search engines support searchers quite effectively in uncertainty, Kuhlthau indicates that information may actually locating sources. However, the “searcher in need” is not able to increase uncertainty. Uncertainty is a necessary critical element assess if the located source is a correct one. Therefore, search in any process of knowledge construction [17]. The engines should also take “process problems” [17] into account, comprehension of the problem at hand is supposed to be i.e. the stage of a user in the information seeking process. These changed and evolving during the process and learning from the process problems are currently not identified nor ameliorated by retrieved documents is to occur. In case the information seeking search systems. One way to do this is to support the search process is too shallow, intervention might be necessary in order process of a user and possibly intervene. to make people realize that they need to gain greater First of all, a search system should roughly be aware of the understanding. [17, p. 114]. Kuhlthau defines a “Zone of current search stage of a user. To do so, it could actively monitor Intervention” as “that area in which a user can do with guidance a user’s progress in complex information seeking tasks [4]. For and assistance what he or she cannot do alone or can do only linear tasks with a closed set of answers, a system could know with difficulty” [17, p.129]. However, interventions are not related concepts, associated sources and the “right” answers, and always necessary: if an individual is “self-sufficient”, actively monitor if a user touches upon them in their information interventions may even have adverse effects, as they can be searches. For complex tasks involving knowledge construction, intrusive and annoying [17]. intervening can be harder, however. Vakkari [24] provides some indications of searchers’ behavior, stating that pre-focus 3. DEFINING “A SEARCHER IN NEED” searches are more precision-oriented, while searches in post- A searcher on the right path needs to be separated from “a focus stages are more recall-oriented. Conceptual constructs searcher in need” to avoid unnecessary interventions. Here, we evolve from vague to precise. At the pre-focus stage a user is define a searcher in need as a novice searcher whose focus is not satisfied with any piece of information that is even remotely formulated, and does not realize that is critical. A novice relevant to the topic, since they are trying to gather searcher does not have a sufficient level of subject knowledge or understanding on it. This kind of “shooting blind” behavior search experience [13]. According to Holliday & Li [12] and might be detected by monitoring search sessions. Vakkari [24], many students skip focus formulation entirely. On way to find the users in need is that the system could build Thirdly, the provision of additional search support features at its identification of ‘user stage’ on search characteristics moments when a searcher is “in need” may disrupt the shallow reflecting different types of search tactics [3, 19], information information seeking process. Various categories of search needs [5] or different types of search goals [23] and prompt its features could support search in different stages [15]. For support when needed. However, there is need for more research example, a searcher in need could be offered explicit query in which stages of ISP these occur. suggestions promoting diversity (“did you try …”) at strategic Systems would support searchers in need in various ways. Here, moments, or word clouds showing explored and unexplored concepts to promote exploratory browsing of material. a paradox emerges: excellent search tools providing a user with perfectly tailored results might not necessarily be desirable. However, the searcher may be unwilling to accept the “shaking” With this kind of tool, a searcher does not need to put in the due to the cognitive biases. Some solutions to solve this grand effort necessary for focus formulation. However, increasing challenge may include persuasive techniques and gamification. search effort has been shown to decrease perceived precision, Deep learning might be helped with the emergence of a prepared but to lead to better outcomes [24]. Hence, we need more mind, motivation and curiosity. blurred set of search results for better task results. 5. LIMITATIONS AND FUTURE WORK One way to support searchers in need would be to provide high- This position paper presented an idealistic approach to user quality search tools with the ability to introduce interventions, or stimulation in order to improve task outcome. At this point of “disruptions” in the search process. We want to stimulate the time shallow information processing is acceptable, both in user to reflect and possibly change her search tactics. Motivation practice and in research. Maybe the biggest challenge to for changing tactics may be triggered by providing interesting applying more deep information processing is the persuasion of new angles towards topics or problems, showing violations of people to change their behavior and habits. However, regardless previous expectations requiring further investigation, and the state of the art, this “shaking approach” might be one way to creating sudden awareness of inconsistencies and gaps in support the complex task processes. understanding [20]. If information environments are normally orderly, it is the disorder that catches our attention, and this The posited “shaking approach” needs to be studied in order to causes cognitive reactions [6]. In effect, interventions in the prove its usefulness. The improvement that instructions and search process should provide new insights to searchers in need. other interventions can provide to actual learning outcomes might be further studied and tested with students, both in the At system level, users should be stimulated to investigate type of instruction or tool support and in the timing of the matters more deeply. This may be done by providing surprising intervention. One challenge is to find the possibly overconfident search results, by asking the searcher to reflect on the search searchers, and to observe the kinds of behaviors described here results and with specific support tools at suitable points of the during their information search processes. search process. Firstly, a search system could change the ranking algorithms and suddenly provide different results to 6. CONCLUSION queries. For instance, a system could promote diversity ranking This paper aims at stimulating discussion about the level of and reduce redundancy to invite searchers to explore more support a searcher needs during a search process. At the early diverse information sources. Further, systems could even stages of the search process, a novice searcher needs to create a stimulate serendipity, thereby promoting chance encounters [22] focus in order to succeed in the task at hand. Sometimes this and “intellectual leaps of understanding” [1] or even eureka does not happen, and we suggest that by shaking the user a bit, moments [26]. However, any piece of information must be they might reach the focus formulation phase and ultimately a related to the actor’s prior experiences and to be exposed to the better task outcome. This shaking may be done by providing actor in order to be found serendipitously [22]. surprising but at the same time coherent search results, Hence, a search system would move away from considering the instructional ways and by reflecting on the search process and algorithmic relevance of retrieved results to a certain query, but tactics, and by stage-specific support tools. However, the provide results relevant at a certain point of the search process searcher needs to be motivated, and therefore some persuasion is of a user. Moreover, a highly coherent information source may needed. It is commonly agreed, that without any cognitive effort increase learning for searchers with low knowledge, but not in task performance, the level of outcome may remain low. In offer enough diversity and challenge for a high knowledge order to trigger critical thinking and knowledge creation, searcher [9]. Consequently different kinds of sources may be searchers need to be shaken to realize that there might be other useful in different situations, and the provided information ways to solve their problem, if the search route they follow should match with searchers’ cognitive and affective states [17]. seems too obvious to them. Secondly, shake-ups in the search process could also be 7. ACKNOWLEDGMENTS provided using applied instructional interventions, which are This research was supported by the Netherlands Organization stimulating reflection. For example, at an appropriate point, a for Scientific Research (NWO project # 640.005.001 - system could provide instructional guidance by providing WebART). The authors wish to thank Jaap Kamps for fruitful guiding instructions based on established information literacy discussions on the topic. models. Also, a system could interactively ask questions to make the searcher reflect on the found information sources (for 8. REFERENCES instance about content and authority of sources1), or even [1] André, P., Schraefel, M. C., Teevan, J., and Dumais, S. T. provide search dashboards providing feedback on personal 2009. Discovery is never by chance: Designing for search behavior [2]. (un)serendipity. In Proc. Conference on Creativity and Cognition. New York, NY: ACM Press, 2009, 305-314. 1 http://library.sasaustin.org/webEvaluationTraining.php [2] Bateman, S. ,Teevan, J. and White, R. W. 2012. The search [14] Huang, X., and Soergel, D. (2004). Relevance judges' dashboard: how reflection and comparison impact search understanding of topical relevance types: An explication of behavior. In Proc. CHI '12. ACM, New York, NY, USA, an enriched concept of topical relevance. Proceedings of 1785-1794. the American Society for Information Science and [3] Bates, M. J. 1979. Information search tactics. Journal of the Technology, 41(1), 156-167. American Society for information Science, 30(4), 205-214. [15] Huurdeman H.C., and Kamps, J. 2014. From multistage [4] Belkin, N. J. , Clarke, Charles L.A. , Gao, N., Kamps, J. information-seeking models to multistage search systems. and Karlgren, J. 2012. Report on the SIGIR workshop on In Proc. IIiX '14, 145-154. "entertain me": supporting complex search tasks. SIGIR [16] Kelly. D. 2012. Cognitive consequences of search. In Proc. Forum 45, 2 (January 2012), 51-59. IIIX '12, 2. [5] Borlund, P., and Dreier, S. (2014). An investigation of the [17] Kuhlthau, C.C. 2004. Seeking meaning: a process approach search behaviour associated with Ingwersen’s three types to library and information services. 2nd. ed. Westport, CT: of information needs. Information Processing & Libraries Unlimited. Management, 50(4), 493-507. [18] Kumpulainen, S. 2013. Task-based information access in [6] Bowler, L. and Mattern, E. 2012. Visual Metaphors for molecular medicine: task performance, barriers, and Modeling Metacognitive Strategies that Support Memory searching within a heterogeneous information environment. During the Information Search Process. In Proc. IIiX ‘12. Tampere: Tampere University Press. August 21-24, 2012. Nijmegen, The Netherlands, 250-253. [19] Kumpulainen, S. 2014. Trails across the heterogeneous [7] Byström, K. and Järvelin, K. 1995. Task complexity affects information environment: manual integration patterns of information seeking and use. Information Processing & search systems in molecular medicine. Journal of Management, 31(2), 191 - 213. Documentation, 70(5), 856-877. [8] Davidson. E., Deuser, and R.J. Sternberg. 1994. The role of [20] O'Day, V. L. and R. J. 1993. Orienteering in an information metacognition in problem solving. In Metcalfe & landscape: how information seekers get from here to there. Shimamura (eds.) Metacognition. Cambridge: MIT press. Proc. INTERACT'93 and CHI'93. ACM, 438-445. 207-226. [21] Todd, R. 2006. From information to knowledge: Charting [9] Festinger, L. 1957. A Theory of Cognitive Dissonance. and Measuring Changes in Students' Knowledge of a Stanford, CA: Stanford University Press. Curriculum Topic. Information Research, 11 (4). [10] Freund, L., O’Brien, H., and Kopak, R. (2014). Getting the [22] Toms, E. G. 2000. Serendipitous Information Retrieval. In big picture: supporting comprehension and learning in DELOS Workshop: Information Seeking, Searching and search. Querying in Digital Libraries. [11] Gross, M. 2005. The Impact of Low-Level Skills on [23] Toms, E. G. 2011. Task-based information searching and Information-Seeking Behavior: Implications of retrieval. Interactive Information Seeking, Behaviour and Competency Theory for Research and Practice. Reference Retrieval. Facet Publishing, 43-59. & User Services Quarterly. Vol. 45, No. 2, pp.155-162. [24] Vakkari, P. 2001. A theory of the task-based information [12] Holliday, W., and Li, Q. 2004. Understanding the retrieval process: a summary and generalisation of a millennials: updating our knowledge about students. longitudinal study. Journal of documentation, 57(1), 44-60. Reference services review, 32(4), 356-366. [25] Vakkari, P., & Huuskonen, S. 2012. Search effort degrades [13] Hsieh­‐Yee, I. 1993. Effects of search experience and search output but improves task outcome. JASIST, 63(4), subject knowledge on the search tactics of novice and 657-670. experienced searchers. Journal of the American Society for [26] White, R. and Roth, R. 2008. Exploratory Search. Morgan Information Science, 44(3), 161-174. & Claypool Publishers