Conversational Interfaces for Search As Learning Sihang Qiu, Alessandro Bozzon and Ujwal Gadiraju Web Information Systems, Delft University of Technology Abstract Searching the web to learn new things or gain knowledge has become a common activity. Recent advances in conversa- tional user interfaces have led to a new research opportunity – that of analyzing the potential of conversational interfaces in improving the effectiveness of search as learning (SAL). Addressing this knowledge gap, in this position paper we present conversational interfaces to support search as learning and novel methods to measure user performance and learning. Our experimental results reveal that conversational interfaces can improve user engagement, augment user long-term memora- bility, and alleviate user cognitive load. These findings have important implications on designing effective SAL systems. Keywords Conversational interface, search, learning, chatbot 1. Introduction ically seek to explore whether CUIs can improve user learning, user experience in terms of user engagement, Over 4 billion people around the globe actively use the cognitive load, and long-term memorability of the in- Internet today; that is over half of the world popula- formation consumed. To this end, we make the follow- tion. Web search is one of the most common activities ing contributions. on the Internet, particularly for the purpose of gain- i) We designed a conversational interface supported by ing new knowledge [1, 2]. Therefore, learning has in- a rule-based conversational agent to assist workers in evitably become an important part of web search, ei- web-based information retrieval (web search based on ther actively or passively. Meanwhile, there has been a desktop browsers). Through experiments in a typical rise in the use of conversational user interfaces (CUIs) microtask crowdsourcing setup with search tasks, we – applications aiming to provide users with seamless investigated whether a dialogue-based system can be means of interaction via virtual assistants, chatbots, or an alternative to the conventional web search inter- messaging services. This paper lies at the confluence face. We found that the task execution supported by of SAL and CUIs, and explores how learning through conversational agents can produce high user satisfac- web search sessions can be improved by leveraging tion, while resulting in similar outcomes compared to conversational interfaces. conventional means [8]. Prior studies in online learning have revealed that conversational systems can improve learning outcomes i) We conducted experiments to assess whether a con- in some specific scenarios [3, 4, 5]. However, to what versational interface can better engage users. We found extent CUIs can improve learning environments to bet- that users using CUIs exhibit a higher retention rate, ter engage learners and alleviate their cognitive load suggesting that conversational interfaces can signifi- remains unexplored. Furthermore, as the goal of learn- cantly improve user engagement [9]. ing is to develop a deep understanding of some in- ii) To predict user performance and understand how formation, memorization is an important element [6, conversational interfaces can alleviate cognitive load, 7]. Although conversation can produce unique con- we proposed a coding scheme to estimate users’ con- text linked with information, the effect of conversa- versational styles. We found that users’ conversational tional systems on human memorability needs further styles are highly correlated to their performances, and exploration. CUIs have a strong potential to reduce the cognitive In this position paper, we aim to fill this knowledge load of users [10]. gap by designing conversational interfaces to improve iii) To study the impact of CUIs on human memora- learning effects during web search sessions. We specif- bility, an important by product of learning, we con- Proceedings of the CIKM 2020 Workshops, October 19–20, Galway, ducted an online user study in a classical information Ireland retrieval setup. Our results suggest conversational in- email: s.qiu-1@tudelft.nl (S. Qiu); a.bozzon@tudelft.nl (A. terfaces can serve as a useful means for augmenting Bozzon); u.k.gadiraju@tudelft.nl (U. Gadiraju) long-term human memorability and improving long- orcid: © 2020 Copyright for this paper by its authors. Use permitted under Creative term knowledge gain in search as learning [11]. Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) General Effects of Conversational Interfaces Higher User Retention Conversational Interfaces in Search as Learning Higher User Satisfaction 1. Better engage learners. 2. Give learners higher satisfaction. Less Cognitive Load 3. Let learners perceive less cognitive load. 4. Improve long-term knowledge gain. Stronger Long-term Memory Figure 1: Conversational interfaces in search as learning. 2. Conversational Interfaces for web interfaces. We found that a suitable conversa- tional style has the potential to engage workers further SAL (in specific task types). This work reveals the general As illustrated in Figure 1, we carried out user studies understanding of conversational interfaces for infor- to explore the potential benefits of using CUIs. mation searching tasks. The details of the experimen- tal settings and result analysis can be found in [9]. 2.1. Improving User Satisfaction, Engagement 2.2. Alleviating Cognitive Load To study how conversational interfaces could alleviate We investigated the effects of CUIs with regard to user the cognitive load of users, we classified users into two satisfaction and engagement in typical microtask crowd- categories according to their conversational styles and sourcing setups, where users were asked to complete measured their perceived cognitive loads. information retrieval related tasks, along with other We first conducted research to understand user con- common types of crowdsourcing microtasks. Further- versational styles. Our previous work about user en- more, previous works have shown that monotonous gagement investigated whether different conversational batches of microtasks pose challenges with regards to styles of an agent can increase user engagement. Fur- engaging users, potentially leading to sloppy work due thermore, previous works in the field of psychology to boredom and fatigue. Therefore, whether conversa- have shown the important role that conversational styles tional interfaces could improve user engagement re- have on inter-human communication [13, 14, 15]. Hav- mains unexplored. We conducted a study involving ing been developed in the context of human conversa- 800 unique workers and five task types (Information tions, the insights and conclusions of these works are finding, Sentiment analysis, Human OCR, Audio tran- not directly applicable to conversational microtasking, scription, and Image annotation) across different ex- since the contrasting goal of workers is to optimally al- perimental conditions to address to what extent con- locate their effort rather than being immersed in con- versational interfaces can improve the user engage- versations. To the best of our knowledge, current con- ment while completing information searching tasks in versational agents (particularly for crowdsourcing) have typical crowdsourcing setups, and how conversational only studied the effects of the conversational style of agents with different conversational styles affect the agents, rather than the conversational style of online user engagement while completing tasks. users (i.e., workers in the context of microtask crowd- We used worker retention (the number of answered sourcing). Therefore, we designed a coding scheme optional microtasks) in the batches of tasks and self- inspired by previous work [15] and corresponding to reported scores on the short-form user engagement conversational styles based on the five dimensions of scale [12] to measure user engagement. Our results linguistic devices that have been examined. We also show that conversational interfaces have positive ef- designed and implemented a conversational interface fects on user engagement in comparison to traditional that supports our experiments by extracting linguistic features from the text-based conversation between the conversational interfaces are promising tools for aug- user and the agent. menting human memorability in information retrieval. Understanding the role of workers’ conversational Furthermore, we also delve into the research ques- styles in crowdsourcing can help us better predict user tion: how the use of text-based conversational inter- performance, and better assist and guide workers in faces affects the search behavior of users. Through our the training process. To this end, we also delved into experiments, we found that users leveraging conversa- the research question: to what extent the conversa- tional interfaces input more queries but opened links tional style of crowd workers relates to their work out- less frequently compared to users leveraging the tra- comes and cognitive task load in information retriev- ditional Web interfaces. In addition, the users of con- ing tasks. versational interfaces tend to type notes themselves, We designed information retrieving tasks with three while the Web users input significantly longer notes difficulty levels, where users are asked to find the mid- by copying content directly from the search engine re- dle name of famous people. We recruited 180 unique sult pages. Our findings have important implications online crowd workers from AMT and conducted ex- for building information retrieval systems that cater to periments to investigate the feasibility of conversational optimizing the memorability of information consumed style estimation. We also analyzed the impact of con- and improving long-term learning effects. The details versational style on output quality and perceived task of the experimental settings and result analysis can be load (using the NASA-TLX instrument). Our experi- found in [11]. mental findings revealed that workers with an Involve- ment conversational style have significantly higher out- put quality, higher user engagement, and less cogni- 3. Challenges and Opportunities tive load while they are completing a high-difficulty We conducted rigorous experiments to understand the task, and have less task execution time in general. The role of conversational interfaces in general informa- findings have important implications on user perfor- tion retrieval crowdsourcing tasks, which has impor- mance prediction and cognitive load evaluation in web tant implications for the realm of search as learning. search session. The details of the experimental set- We argue that the use of conversational interfaces can tings and result analysis can be found in [10]. provide a number of potential benefits, such as im- proving user engagement, reducing cognitive load, and 2.3. Augmenting Long-term augmenting long-term memorability. Our research pro- Memorability vides plenty of inspirations for future research direc- tions. Naturally, more research is needed to better un- Since memorization is an essential element of the learn- derstand whether a conversational agent could aid search ing process [6, 7], we aim to fill this knowledge gap by as learning in general. proposing novel approaches to improve human mem- Specifically, in terms of the conversational user in- orability during information retrieval. We specifically terface, we only focus on the text-based conversation focus on web search activities carried out through the across all these studies. In general, there are various desktop browsers. Through rigorous experiments, we means to interact with conversational agents (e.g., voice- seek to address the following research question: how based agent, video-based agent). The effects of voice- human memorability of information consumed in in- or video-based conversational agents on worker per- formational web search sessions can be improved. formance and mental conditions still remain unexplored. Inspired by prior work in psychology and human Furthermore, text-based conversation ignores several computer interaction, we propose novel search inter- paralinguistic features (pitch, voice) and nonlinguistic faces that provide a conversational interface. We pro- features (smile, laughter, gestures), which could play pose methods to quantify knowledge gain and long- important roles in human-computer interaction. Con- term memorability of information consumed, and in- versational agents and corresponding style estimation vestigate the impact of the proposed search interfaces methods based on voice or video could be an interest- on the memorability of information consumed. We ing direction to explore. conducted an online user study, with 140 online work- Our findings also reveal that users employing con- ers, in a classical information retrieval setup. Results versational interfaces in informational search sessions reveal that conversational interfaces have the poten- exhibit a different search behavior compared to tradi- tial to augment long-term memorability (7.5% lower tional web search: they rely primarily on text-based long-term information loss). Our findings suggest that conversation, resulting in a significantly higher fre- quency of issuing queries but a significantly lower fre- novate Learning, Association for the Advance- quency of opening SERP (search engine results page) ment of Computing in Education (AACE), 2005, links. These users appear to consume information by pp. 3913–3918. means of viewing titles and snippets rather than open- [4] A. Latham, K. Crockett, D. McLean, B. Edmonds, ing links and exploring SERPs in detail. We found that A conversational intelligent tutoring system to users employing conversational interfaces have the po- automatically predict learning styles, Comput- tential to better retain information consumed. This is ers & Education 59 (2012) 95–109. possibly due to the fact that conversational interfaces [5] D. Song, E. Y. Oh, M. Rice, Interacting with a can generate unique context connected to the infor- conversational agent system for educational pur- mation during the search session. Our inspection of poses in online courses, in: 2017 10th interna- users’ notes also corroborates that users using con- tional conference on human system interactions versational interfaces tend to generate information by (HSI), IEEE, 2017, pp. 78–82. themselves rather than copying content from sources [6] D. Kember, The intention to both memorise (Web users’ preference). These findings suggest that and understand: Another approach to learning?, both note-taking and conversational interfaces can be Higher Education 31 (1996) 341–354. promising tools towards achieving memorable search [7] J. B. Biggs, Student Approaches to Learning and as learning in the future. Studying. Research Monograph., ERIC, 1987. In the experiments about memorability in web search, [8] P. Mavridis, O. Huang, S. Qiu, U. Gadiraju, we found that only around half of the users returned A. Bozzon, Chatterbox: Conversational inter- for our long-term memory test, which is typical of such faces for microtask crowdsourcing, in: Proceed- experiments. Our results show that the users with a ings of the 27th ACM Conference on User Model- relatively higher knowledge gain were more willing ing, Adaptation and Personalization, ACM, 2019, to return and participate in our memory test. It should pp. 243–251. be noted that this participation bias presents a threat [9] S. Qiu, U. Gadiraju, A. Bozzon, Improving worker to the representativeness of our findings. In our im- engagement through conversational microtask minent future research on search as learning, we will crowdsourcing, in: Proceedings of the 2020 explore whether a higher user engagement relates to a CHI Conference on Human Factors in Comput- better user memorability or a better long-term learn- ing Systems, 2020, pp. 1–12. ing effect. [10] S. Qiu, U. Gadiraju, A. Bozzon, Estimating con- versational styles in conversational microtask crowdsourcing, Proceedings of the ACM on Acknowledgements Human-Computer Interaction 4 (2020) 1–23. [11] S. Qiu, U. Gadiraju, A. Bozzon, Towards mem- This work was carried out on the Dutch national e- orable information retrieval, in: Proceedings of infrastructure with the support of SURF Cooperative. the 2020 ACM SIGIR International Conference on the Theory of Information Retrieval, ACM, 2020, References pp. 69–76. [12] H. L. O’Brien, P. Cairns, M. Hall, A practical ap- [1] U. Gadiraju, R. Yu, S. Dietze, P. Holtz, Analyzing proach to measuring user engagement with the knowledge gain of users in informational search refined user engagement scale (ues) and new ues sessions on the web, in: Proceedings of the 2018 short form, International Journal of Human- Conference on Human Information Interaction & Computer Studies 112 (2018) 28–39. Retrieval, 2018, pp. 2–11. [13] R. T. Lakoff, Stylistic strategies within a grammar [2] R. Yu, U. Gadiraju, P. Holtz, M. Rokicki, of style, Annals of the New York Academy of P. Kemkes, S. Dietze, Predicting user knowledge Sciences 327 (1979) 53–78. gain in informational search sessions, in: The [14] D. Tannen, Conversational style, Psycholinguis- 41st International ACM SIGIR Conference on Re- tic models of production (1987) 251–267. search & Development in Information Retrieval, [15] D. Tannen, Conversational style: Analyzing talk 2018, pp. 75–84. among friends, Oxford University Press, 2005. [3] B. Heller, M. Proctor, D. Mah, L. Jewell, B. Che- ung, Freudbot: An investigation of chatbot tech- nology in distance education, in: EdMedia+ In-