=Paper=
{{Paper
|id=Vol-2699/paper23
|storemode=property
|title=Conversational Interfaces for Search As Learning
|pdfUrl=https://ceur-ws.org/Vol-2699/paper23.pdf
|volume=Vol-2699
|authors=Sihang Qiu,Alessandro Bozzon,Ujwal Gadiraju
|dblpUrl=https://dblp.org/rec/conf/cikm/QiuBG20
}}
==Conversational Interfaces for Search As Learning==
https://ceur-ws.org/Vol-2699/paper23.pdf
Conversational Interfaces for Search As Learning
Sihang Qiu, Alessandro Bozzon and Ujwal Gadiraju
Web Information Systems, Delft University of Technology
Abstract
Searching the web to learn new things or gain knowledge has become a common activity. Recent advances in conversa-
tional user interfaces have led to a new research opportunity – that of analyzing the potential of conversational interfaces
in improving the effectiveness of search as learning (SAL). Addressing this knowledge gap, in this position paper we present
conversational interfaces to support search as learning and novel methods to measure user performance and learning. Our
experimental results reveal that conversational interfaces can improve user engagement, augment user long-term memora-
bility, and alleviate user cognitive load. These findings have important implications on designing effective SAL systems.
Keywords
Conversational interface, search, learning, chatbot
1. Introduction ically seek to explore whether CUIs can improve user
learning, user experience in terms of user engagement,
Over 4 billion people around the globe actively use the cognitive load, and long-term memorability of the in-
Internet today; that is over half of the world popula- formation consumed. To this end, we make the follow-
tion. Web search is one of the most common activities ing contributions.
on the Internet, particularly for the purpose of gain-
i) We designed a conversational interface supported by
ing new knowledge [1, 2]. Therefore, learning has in-
a rule-based conversational agent to assist workers in
evitably become an important part of web search, ei-
web-based information retrieval (web search based on
ther actively or passively. Meanwhile, there has been a
desktop browsers). Through experiments in a typical
rise in the use of conversational user interfaces (CUIs)
microtask crowdsourcing setup with search tasks, we
– applications aiming to provide users with seamless
investigated whether a dialogue-based system can be
means of interaction via virtual assistants, chatbots, or
an alternative to the conventional web search inter-
messaging services. This paper lies at the confluence
face. We found that the task execution supported by
of SAL and CUIs, and explores how learning through
conversational agents can produce high user satisfac-
web search sessions can be improved by leveraging
tion, while resulting in similar outcomes compared to
conversational interfaces.
conventional means [8].
Prior studies in online learning have revealed that
conversational systems can improve learning outcomes i) We conducted experiments to assess whether a con-
in some specific scenarios [3, 4, 5]. However, to what versational interface can better engage users. We found
extent CUIs can improve learning environments to bet- that users using CUIs exhibit a higher retention rate,
ter engage learners and alleviate their cognitive load suggesting that conversational interfaces can signifi-
remains unexplored. Furthermore, as the goal of learn- cantly improve user engagement [9].
ing is to develop a deep understanding of some in- ii) To predict user performance and understand how
formation, memorization is an important element [6, conversational interfaces can alleviate cognitive load,
7]. Although conversation can produce unique con- we proposed a coding scheme to estimate users’ con-
text linked with information, the effect of conversa- versational styles. We found that users’ conversational
tional systems on human memorability needs further styles are highly correlated to their performances, and
exploration. CUIs have a strong potential to reduce the cognitive
In this position paper, we aim to fill this knowledge load of users [10].
gap by designing conversational interfaces to improve iii) To study the impact of CUIs on human memora-
learning effects during web search sessions. We specif- bility, an important by product of learning, we con-
Proceedings of the CIKM 2020 Workshops, October 19–20, Galway,
ducted an online user study in a classical information
Ireland retrieval setup. Our results suggest conversational in-
email: s.qiu-1@tudelft.nl (S. Qiu); a.bozzon@tudelft.nl (A. terfaces can serve as a useful means for augmenting
Bozzon); u.k.gadiraju@tudelft.nl (U. Gadiraju) long-term human memorability and improving long-
orcid:
© 2020 Copyright for this paper by its authors. Use permitted under Creative term knowledge gain in search as learning [11].
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org)
General Effects of Conversational Interfaces
Higher User Retention Conversational Interfaces in Search as Learning
Higher User Satisfaction 1. Better engage learners.
2. Give learners higher satisfaction.
Less Cognitive Load 3. Let learners perceive less cognitive load.
4. Improve long-term knowledge gain.
Stronger Long-term Memory
Figure 1: Conversational interfaces in search as learning.
2. Conversational Interfaces for web interfaces. We found that a suitable conversa-
tional style has the potential to engage workers further
SAL (in specific task types). This work reveals the general
As illustrated in Figure 1, we carried out user studies understanding of conversational interfaces for infor-
to explore the potential benefits of using CUIs. mation searching tasks. The details of the experimen-
tal settings and result analysis can be found in [9].
2.1. Improving User Satisfaction,
Engagement 2.2. Alleviating Cognitive Load
To study how conversational interfaces could alleviate
We investigated the effects of CUIs with regard to user
the cognitive load of users, we classified users into two
satisfaction and engagement in typical microtask crowd-
categories according to their conversational styles and
sourcing setups, where users were asked to complete
measured their perceived cognitive loads.
information retrieval related tasks, along with other
We first conducted research to understand user con-
common types of crowdsourcing microtasks. Further-
versational styles. Our previous work about user en-
more, previous works have shown that monotonous
gagement investigated whether different conversational
batches of microtasks pose challenges with regards to
styles of an agent can increase user engagement. Fur-
engaging users, potentially leading to sloppy work due
thermore, previous works in the field of psychology
to boredom and fatigue. Therefore, whether conversa-
have shown the important role that conversational styles
tional interfaces could improve user engagement re-
have on inter-human communication [13, 14, 15]. Hav-
mains unexplored. We conducted a study involving
ing been developed in the context of human conversa-
800 unique workers and five task types (Information
tions, the insights and conclusions of these works are
finding, Sentiment analysis, Human OCR, Audio tran-
not directly applicable to conversational microtasking,
scription, and Image annotation) across different ex-
since the contrasting goal of workers is to optimally al-
perimental conditions to address to what extent con-
locate their effort rather than being immersed in con-
versational interfaces can improve the user engage-
versations. To the best of our knowledge, current con-
ment while completing information searching tasks in
versational agents (particularly for crowdsourcing) have
typical crowdsourcing setups, and how conversational
only studied the effects of the conversational style of
agents with different conversational styles affect the
agents, rather than the conversational style of online
user engagement while completing tasks.
users (i.e., workers in the context of microtask crowd-
We used worker retention (the number of answered
sourcing). Therefore, we designed a coding scheme
optional microtasks) in the batches of tasks and self-
inspired by previous work [15] and corresponding to
reported scores on the short-form user engagement
conversational styles based on the five dimensions of
scale [12] to measure user engagement. Our results
linguistic devices that have been examined. We also
show that conversational interfaces have positive ef-
designed and implemented a conversational interface
fects on user engagement in comparison to traditional
that supports our experiments by extracting linguistic
features from the text-based conversation between the conversational interfaces are promising tools for aug-
user and the agent. menting human memorability in information retrieval.
Understanding the role of workers’ conversational Furthermore, we also delve into the research ques-
styles in crowdsourcing can help us better predict user tion: how the use of text-based conversational inter-
performance, and better assist and guide workers in faces affects the search behavior of users. Through our
the training process. To this end, we also delved into experiments, we found that users leveraging conversa-
the research question: to what extent the conversa- tional interfaces input more queries but opened links
tional style of crowd workers relates to their work out- less frequently compared to users leveraging the tra-
comes and cognitive task load in information retriev- ditional Web interfaces. In addition, the users of con-
ing tasks. versational interfaces tend to type notes themselves,
We designed information retrieving tasks with three while the Web users input significantly longer notes
difficulty levels, where users are asked to find the mid- by copying content directly from the search engine re-
dle name of famous people. We recruited 180 unique sult pages. Our findings have important implications
online crowd workers from AMT and conducted ex- for building information retrieval systems that cater to
periments to investigate the feasibility of conversational optimizing the memorability of information consumed
style estimation. We also analyzed the impact of con- and improving long-term learning effects. The details
versational style on output quality and perceived task of the experimental settings and result analysis can be
load (using the NASA-TLX instrument). Our experi- found in [11].
mental findings revealed that workers with an Involve-
ment conversational style have significantly higher out-
put quality, higher user engagement, and less cogni- 3. Challenges and Opportunities
tive load while they are completing a high-difficulty
We conducted rigorous experiments to understand the
task, and have less task execution time in general. The
role of conversational interfaces in general informa-
findings have important implications on user perfor-
tion retrieval crowdsourcing tasks, which has impor-
mance prediction and cognitive load evaluation in web
tant implications for the realm of search as learning.
search session. The details of the experimental set-
We argue that the use of conversational interfaces can
tings and result analysis can be found in [10].
provide a number of potential benefits, such as im-
proving user engagement, reducing cognitive load, and
2.3. Augmenting Long-term augmenting long-term memorability. Our research pro-
Memorability vides plenty of inspirations for future research direc-
tions. Naturally, more research is needed to better un-
Since memorization is an essential element of the learn-
derstand whether a conversational agent could aid search
ing process [6, 7], we aim to fill this knowledge gap by
as learning in general.
proposing novel approaches to improve human mem-
Specifically, in terms of the conversational user in-
orability during information retrieval. We specifically
terface, we only focus on the text-based conversation
focus on web search activities carried out through the
across all these studies. In general, there are various
desktop browsers. Through rigorous experiments, we
means to interact with conversational agents (e.g., voice-
seek to address the following research question: how
based agent, video-based agent). The effects of voice-
human memorability of information consumed in in-
or video-based conversational agents on worker per-
formational web search sessions can be improved.
formance and mental conditions still remain unexplored.
Inspired by prior work in psychology and human
Furthermore, text-based conversation ignores several
computer interaction, we propose novel search inter-
paralinguistic features (pitch, voice) and nonlinguistic
faces that provide a conversational interface. We pro-
features (smile, laughter, gestures), which could play
pose methods to quantify knowledge gain and long-
important roles in human-computer interaction. Con-
term memorability of information consumed, and in-
versational agents and corresponding style estimation
vestigate the impact of the proposed search interfaces
methods based on voice or video could be an interest-
on the memorability of information consumed. We
ing direction to explore.
conducted an online user study, with 140 online work-
Our findings also reveal that users employing con-
ers, in a classical information retrieval setup. Results
versational interfaces in informational search sessions
reveal that conversational interfaces have the poten-
exhibit a different search behavior compared to tradi-
tial to augment long-term memorability (7.5% lower
tional web search: they rely primarily on text-based
long-term information loss). Our findings suggest that
conversation, resulting in a significantly higher fre-
quency of issuing queries but a significantly lower fre- novate Learning, Association for the Advance-
quency of opening SERP (search engine results page) ment of Computing in Education (AACE), 2005,
links. These users appear to consume information by pp. 3913–3918.
means of viewing titles and snippets rather than open- [4] A. Latham, K. Crockett, D. McLean, B. Edmonds,
ing links and exploring SERPs in detail. We found that A conversational intelligent tutoring system to
users employing conversational interfaces have the po- automatically predict learning styles, Comput-
tential to better retain information consumed. This is ers & Education 59 (2012) 95–109.
possibly due to the fact that conversational interfaces [5] D. Song, E. Y. Oh, M. Rice, Interacting with a
can generate unique context connected to the infor- conversational agent system for educational pur-
mation during the search session. Our inspection of poses in online courses, in: 2017 10th interna-
users’ notes also corroborates that users using con- tional conference on human system interactions
versational interfaces tend to generate information by (HSI), IEEE, 2017, pp. 78–82.
themselves rather than copying content from sources [6] D. Kember, The intention to both memorise
(Web users’ preference). These findings suggest that and understand: Another approach to learning?,
both note-taking and conversational interfaces can be Higher Education 31 (1996) 341–354.
promising tools towards achieving memorable search [7] J. B. Biggs, Student Approaches to Learning and
as learning in the future. Studying. Research Monograph., ERIC, 1987.
In the experiments about memorability in web search, [8] P. Mavridis, O. Huang, S. Qiu, U. Gadiraju,
we found that only around half of the users returned A. Bozzon, Chatterbox: Conversational inter-
for our long-term memory test, which is typical of such faces for microtask crowdsourcing, in: Proceed-
experiments. Our results show that the users with a ings of the 27th ACM Conference on User Model-
relatively higher knowledge gain were more willing ing, Adaptation and Personalization, ACM, 2019,
to return and participate in our memory test. It should pp. 243–251.
be noted that this participation bias presents a threat [9] S. Qiu, U. Gadiraju, A. Bozzon, Improving worker
to the representativeness of our findings. In our im- engagement through conversational microtask
minent future research on search as learning, we will crowdsourcing, in: Proceedings of the 2020
explore whether a higher user engagement relates to a CHI Conference on Human Factors in Comput-
better user memorability or a better long-term learn- ing Systems, 2020, pp. 1–12.
ing effect. [10] S. Qiu, U. Gadiraju, A. Bozzon, Estimating con-
versational styles in conversational microtask
crowdsourcing, Proceedings of the ACM on
Acknowledgements Human-Computer Interaction 4 (2020) 1–23.
[11] S. Qiu, U. Gadiraju, A. Bozzon, Towards mem-
This work was carried out on the Dutch national e-
orable information retrieval, in: Proceedings of
infrastructure with the support of SURF Cooperative.
the 2020 ACM SIGIR International Conference on
the Theory of Information Retrieval, ACM, 2020,
References pp. 69–76.
[12] H. L. O’Brien, P. Cairns, M. Hall, A practical ap-
[1] U. Gadiraju, R. Yu, S. Dietze, P. Holtz, Analyzing proach to measuring user engagement with the
knowledge gain of users in informational search refined user engagement scale (ues) and new ues
sessions on the web, in: Proceedings of the 2018 short form, International Journal of Human-
Conference on Human Information Interaction & Computer Studies 112 (2018) 28–39.
Retrieval, 2018, pp. 2–11. [13] R. T. Lakoff, Stylistic strategies within a grammar
[2] R. Yu, U. Gadiraju, P. Holtz, M. Rokicki, of style, Annals of the New York Academy of
P. Kemkes, S. Dietze, Predicting user knowledge Sciences 327 (1979) 53–78.
gain in informational search sessions, in: The [14] D. Tannen, Conversational style, Psycholinguis-
41st International ACM SIGIR Conference on Re- tic models of production (1987) 251–267.
search & Development in Information Retrieval, [15] D. Tannen, Conversational style: Analyzing talk
2018, pp. 75–84. among friends, Oxford University Press, 2005.
[3] B. Heller, M. Proctor, D. Mah, L. Jewell, B. Che-
ung, Freudbot: An investigation of chatbot tech-
nology in distance education, in: EdMedia+ In-