Searching the web to learn new things or gain knowledge has become a common activity. Recent advances in conversational user interfaces have led to a new research opportunity - that of analyzing the potential of conversational interfaces in improving the efectiveness of search as learning (SAL). Addressing this knowledge gap, in this position paper we present conversational interfaces to support search as learning and novel methods to measure user performance and learning. Our experimental results reveal that conversational interfaces can improve user engagement, augment user long-term memorability, and alleviate user cognitive load. These findings have important implications on designing efective SAL systems.
ically seek to explore whether CUIs can improve user learning, user experience in terms of user engagement, Over 4 billion people around the globe actively use the cognitive load, and long-term memorability of the inInternet today; that is over half of the world popula- formation consumed. To this end, we make the followtion. Web search is one of the most common activities ing contributions. on the Internet, particularly for the purpose of gain- i) We designed a conversational interface supported by ing new knowledge [1, 2]. Therefore, learning has in- a rule-based conversational agent to assist workers in evitably become an important part of web search, ei- web-based information retrieval (web search based on ther actively or passively. Meanwhile, there has been a desktop browsers). Through experiments in a typical rise in the use of conversational user interfaces (CUIs) microtask crowdsourcing setup with search tasks, we – applications aiming to provide users with seamless investigated whether a dialogue-based system can be means of interaction via virtual assistants, chatbots, or an alternative to the conventional web search intermessaging services. This paper lies at the confluence face. We found that the task execution supported by of SAL and CUIs, and explores how learning through conversational agents can produce high user satisfacweb search sessions can be improved by leveraging tion, while resulting in similar outcomes compared to conversational interfaces. conventional means [8].
Prior studies in online learning have revealed that conversational systems can improve learning outcomes i) We conducted experiments to assess whether a conin some specific scenarios [3, 4, 5]. However, to what versational interface can better engage users. We found extent CUIs can improve learning environments to bet- that users using CUIs exhibit a higher retention rate, ter engage learners and alleviate their cognitive load suggesting that conversational interfaces can signifiremains unexplored. Furthermore, as the goal of learn- cantly improve user engagement [9]. ing is to develop a deep understanding of some in- ii) To predict user performance and understand how formation, memorization is an important element [6, conversational interfaces can alleviate cognitive load, 7]. Although conversation can produce unique con- we proposed a coding scheme to estimate users’ context linked with information, the efect of conversa- versational styles. We found that users’ conversational tional systems on human memorability needs further styles are highly correlated to their performances, and exploration. CUIs have a strong potential to reduce the cognitive
In this position paper, we aim to fill this knowledge load of users [10]. gap by designing conversational interfaces to improve iii) To study the impact of CUIs on human memoralearning efects during web search sessions. We specif- bility, an important by product of learning, we conProceedings of the CIKM 2020 Workshops, October 19–20, Galway, ducted an online user study in a classical information Ireland retrieval setup. Our results suggest conversational inemail: s.qiu-1@tudelft.nl (S. Qiu); a.bozzon@tudelft.nl (A. terfaces can serve as a useful means for augmenting Bozzon); u.k.gadiraju@tudelft.nl (U. Gadiraju) long-term human memorability and improving longorcid: term knowledge gain in search as learning [11].
© 2020 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmUmoRns WLiceonrsekAsthtriobuptioPnr4o.0cIneteerdnaitniognasl ((CCC EBYU4R.0)-.WS.org)
General Effects of Conversational Interfaces
Higher User Satisfaction
Less Cognitive Load Stronger Long-term Memory
Higher User Retention Conversational Interfaces in Search as Learning 1. Better engage learners. 2. Give learners higher satisfaction. 3. Let learners perceive less cognitive load. 4. Improve long-term knowledge gain.
web interfaces. We found that a suitable conversational style has the potential to engage workers further (in specific task types). This work reveals the general understanding of conversational interfaces for information searching tasks. The details of the experimental settings and result analysis can be found in [9].
We investigated the efects of CUIs with regard to user To study how conversational interfaces could alleviate satisfaction and engagement in typical microtask crowd- the cognitive load of users, we classified users into two sourcing setups, where users were asked to complete categories according to their conversational styles and information retrieval related tasks, along with other measured their perceived cognitive loads. common types of crowdsourcing microtasks. Further- We first conducted research to understand user conmore, previous works have shown that monotonous versational styles. Our previous work about user enbatches of microtasks pose challenges with regards to gagement investigated whether diferent conversational engaging users, potentially leading to sloppy work due styles of an agent can increase user engagement. Furto boredom and fatigue. Therefore, whether conversa- thermore, previous works in the field of psychology tional interfaces could improve user engagement re- have shown the important role that conversational styles mains unexplored. We conducted a study involving have on inter-human communication [13, 14, 15]. Hav800 unique workers and five task types (Information ing been developed in the context of human conversaifnding, Sentiment analysis, Human OCR, Audio tran- tions, the insights and conclusions of these works are scription, and Image annotation) across diferent ex- not directly applicable to conversational microtasking, perimental conditions to address to what extent con- since the contrasting goal of workers is to optimally alversational interfaces can improve the user engage- locate their efort rather than being immersed in conment while completing information searching tasks in versations. To the best of our knowledge, current contypical crowdsourcing setups, and how conversational versational agents (particularly for crowdsourcing) have agents with diferent conversational styles afect the only studied the efects of the conversational style of user engagement while completing tasks. agents, rather than the conversational style of online
We used worker retention (the number of answered users (i.e., workers in the context of microtask crowdoptional microtasks) in the batches of tasks and self- sourcing). Therefore, we designed a coding scheme reported scores on the short-form user engagement inspired by previous work [15] and corresponding to scale [12] to measure user engagement. Our results conversational styles based on the five dimensions of show that conversational interfaces have positive ef- linguistic devices that have been examined. We also fects on user engagement in comparison to traditional designed and implemented a conversational interface that supports our experiments by extracting linguistic features from the text-based conversation between the conversational interfaces are promising tools for auguser and the agent. menting human memorability in information retrieval.
Understanding the role of workers’ conversational Furthermore, we also delve into the research quesstyles in crowdsourcing can help us better predict user tion: how the use of text-based conversational interperformance, and better assist and guide workers in faces afects the search behavior of users. Through our the training process. To this end, we also delved into experiments, we found that users leveraging conversathe research question: to what extent the conversa- tional interfaces input more queries but opened links tional style of crowd workers relates to their work out- less frequently compared to users leveraging the tracomes and cognitive task load in information retriev- ditional Web interfaces. In addition, the users of coning tasks. versational interfaces tend to type notes themselves,
We designed information retrieving tasks with three while the Web users input significantly longer notes dificulty levels, where users are asked to find the mid- by copying content directly from the search engine redle name of famous people. We recruited 180 unique sult pages. Our findings have important implications online crowd workers from AMT and conducted ex- for building information retrieval systems that cater to periments to investigate the feasibility of conversational optimizing the memorability of information consumed style estimation. We also analyzed the impact of con- and improving long-term learning efects. The details versational style on output quality and perceived task of the experimental settings and result analysis can be load (using the NASA-TLX instrument). Our experi- found in [11]. mental findings revealed that workers with an Involvement conversational style have significantly higher output quality, higher user engagement, and less cogni- 3. Challenges and Opportunities tive load while they are completing a high-dificulty task, and have less task execution time in general. The ifndings have important implications on user performance prediction and cognitive load evaluation in web search session. The details of the experimental settings and result analysis can be found in [10].
role of conversational interfaces in general information retrieval crowdsourcing tasks, which has important implications for the realm of search as learning. We argue that the use of conversational interfaces can provide a number of potential benefits, such as improving user engagement, reducing cognitive load, and augmenting long-term memorability. Our research provides plenty of inspirations for future research direc
Since memorization is an essential element of the learn- tions. Naturally, more research is needed to better uning process [6, 7], we aim to fill this knowledge gap by derstand whether a conversational agent could aid search proposing novel approaches to improve human mem- as learning in general. orability during information retrieval. We specifically Specifically, in terms of the conversational user infocus on web search activities carried out through the terface, we only focus on the text-based conversation desktop browsers. Through rigorous experiments, we across all these studies. In general, there are various seek to address the following research question: how means to interact with conversational agents (e.g., voicehuman memorability of information consumed in in- based agent, video-based agent). The efects of voiceformational web search sessions can be improved. or video-based conversational agents on worker per
Inspired by prior work in psychology and human formance and mental conditions still remain unexplored. computer interaction, we propose novel search inter- Furthermore, text-based conversation ignores several faces that provide a conversational interface. We pro- paralinguistic features (pitch, voice) and nonlinguistic pose methods to quantify knowledge gain and long- features (smile, laughter, gestures), which could play term memorability of information consumed, and in- important roles in human-computer interaction. Convestigate the impact of the proposed search interfaces versational agents and corresponding style estimation on the memorability of information consumed. We methods based on voice or video could be an interestconducted an online user study, with 140 online work- ing direction to explore. ers, in a classical information retrieval setup. Results Our findings also reveal that users employing conreveal that conversational interfaces have the poten- versational interfaces in informational search sessions tial to augment long-term memorability (7.5% lower exhibit a diferent search behavior compared to tradilong-term information loss). Our findings suggest that tional web search: they rely primarily on text-based conversation, resulting in a significantly higher frequency of issuing queries but a significantly lower fre- novate Learning, Association for the Advancequency of opening SERP (search engine results page) ment of Computing in Education (AACE), 2005, links. These users appear to consume information by pp. 3913–3918. means of viewing titles and snippets rather than open- [4] A. Latham, K. Crockett, D. McLean, B. Edmonds, ing links and exploring SERPs in detail. We found that A conversational intelligent tutoring system to users employing conversational interfaces have the po- automatically predict learning styles, Computtential to better retain information consumed. This is ers & Education 59 (2012) 95–109. possibly due to the fact that conversational interfaces [5] D. Song, E. Y. Oh, M. Rice, Interacting with a can generate unique context connected to the infor- conversational agent system for educational purmation during the search session. Our inspection of poses in online courses, in: 2017 10th internausers’ notes also corroborates that users using con- tional conference on human system interactions versational interfaces tend to generate information by (HSI), IEEE, 2017, pp. 78–82. themselves rather than copying content from sources [6] D. Kember, The intention to both memorise (Web users’ preference). These findings suggest that and understand: Another approach to learning?, both note-taking and conversational interfaces can be Higher Education 31 (1996) 341–354. promising tools towards achieving memorable search [7] J. B. Biggs, Student Approaches to Learning and as learning in the future. Studying. Research Monograph., ERIC, 1987.
In the experiments about memorability in web search, [8] P. Mavridis, O. Huang, S. Qiu, U. Gadiraju, we found that only around half of the users returned A. Bozzon, Chatterbox: Conversational interfor our long-term memory test, which is typical of such faces for microtask crowdsourcing, in: Proceedexperiments. Our results show that the users with a ings of the 27th ACM Conference on User Modelrelatively higher knowledge gain were more willing ing, Adaptation and Personalization, ACM, 2019, to return and participate in our memory test. It should pp. 243–251. be noted that this participation bias presents a threat [9] S. Qiu, U. Gadiraju, A. Bozzon, Improving worker to the representativeness of our findings. In our im- engagement through conversational microtask minent future research on search as learning, we will crowdsourcing, in: Proceedings of the 2020 explore whether a higher user engagement relates to a CHI Conference on Human Factors in Computbetter user memorability or a better long-term learn- ing Systems, 2020, pp. 1–12. ing efect. [10] S. Qiu, U. Gadiraju, A. Bozzon, Estimating conversational styles in conversational microtask crowdsourcing, Proceedings of the ACM on Acknowledgements Human-Computer Interaction 4 (2020) 1–23. [11] S. Qiu, U. Gadiraju, A. Bozzon, Towards memThis work was carried out on the Dutch national e- orable information retrieval, in: Proceedings of infrastructure with the support of SURF Cooperative. the 2020 ACM SIGIR International Conference on the Theory of Information Retrieval, ACM, 2020,
[12] H. L. O’Brien, P. Cairns, M. Hall, A practical ap[1] U. Gadiraju, R. Yu, S. Dietze, P. Holtz, Analyzing proach to measuring user engagement with the knowledge gain of users in informational search refined user engagement scale (ues) and new ues sessions on the web, in: Proceedings of the 2018 short form, International Journal of HumanConference on Human Information Interaction & Computer Studies 112 (2018) 28–39.
Retrieval, 2018, pp. 2–11. [13] R. T. Lakof, Stylistic strategies within a grammar [2] R. Yu, U. Gadiraju, P. Holtz, M. Rokicki, of style, Annals of the New York Academy of P. Kemkes, S. Dietze, Predicting user knowledge Sciences 327 (1979) 53–78. gain in informational search sessions, in: The [14] D. Tannen, Conversational style, Psycholinguis41st International ACM SIGIR Conference on Re- tic models of production (1987) 251–267. search & Development in Information Retrieval, [15] D. Tannen, Conversational style: Analyzing talk 2018, pp. 75–84. among friends, Oxford University Press, 2005. [3] B. Heller, M. Proctor, D. Mah, L. Jewell, B. Cheung, Freudbot: An investigation of chatbot technology in distance education, in: EdMedia+ In