Presently, most conversational information seeking systems function in a passive manner, i.e., user-initiative engagement. Through this work, we aim to discuss the importance of developing conversational information seeking systems capable of system-initiative interactions. We further discuss various aspects of such interactions in CIS systems and introduce a taxonomy of system-initiative interactions based on three orthogonal dimensions: initiation moment (when to initiative a conversation), initiation purpose (why to initiate a conversation), and initiation means (how to initiate a conversation). This taxonomy enables us to propose a generic pipeline for system-initiative conversations, consisting of three major steps associated with the three dimensions highlighted in the taxonomy. We further delineate the technical and evaluation challenges that the design and implementation of each component may encounter, and provide possible solutions. We finally point out potential broader impacts of system-initiative interactions in CIS systems.
ignored in most recent work in the area of conversational information seeking. This is while mixed-initiative intelThe rapid growth in speech and small screen interfaces ligent systems are believed to ultimately revolutionize has significantly influenced the way users interact with the world of computing [7], and CIS systems provide intelligent systems to satisfy their information needs. an appropriate platform for supporting mixed-initiative The growing interest in personal digital assistants demon- interactions. strates the willingness of users to employ conversational Recently, some form of such interactions have been interactions. This has motivated the information retrieval studied in the context of asking for clarification [ 8, 9, 10] community, both academic researchers and industry prac- or preference elicitation [11, 12]. Developing fully mixedtitioners, to focus on conversational information seeking initiative conversational systems requires support for (CIS) as a major emerging research area.1 It has been also system-initiative (or agent-initiative) interactions, where recognized as one of the strategic directions of the com- the CIS system initiates a conversation with the user(s). munity in the Third Strategic Workshop on Information However, system-initiative interactions have been overRetrieval in Lorne (SWIRL 2018) [1].2 However, current looked in the CIS literature. In this paper, we focus on this models and technology provide limited support to conver- topic and discuss its importance for IR research and insational understanding and various types of interactions. dustry. We believe that real-life intelligent assistants can Recent research has made substantial progress in a num- substantially benefit from supporting system-initiative ber of tasks associated with conversational information interactions and this direction involves a large number of seeking [2, 3, 4, 5], however, each with various simplify- unsolved and non-trivial open questions that are worthy ing assumptions on system abilities and user behavior of research. To better demonstrate diferent aspects of that may not hold in a real-world CIS system [6, 7]. For the problem, we compile a taxonomy of system-initiative instance, mixed-initiative interactions have been largely interactions, based on three dimensions: (1) initiation moment: when to initiate a conversation, (2) initiation purpose: why to initiate a conversation, and (3) initiation means: how to initiative a conversation. We believe that system-initiative interactions can be categorized as either instant initiation or opportune moment initiation interactions. We provide example scenarios for each of these categories in Section 2.
The introduced taxonomy enables us to propose a generic pipeline for system-initiative interactions in CIS systems. The pipeline introduced in Section 3 consists DESIRES 2021 – 2nd International Conference on Design of Experimental Search & Information REtrieval Systems, September 15–18, 2021, Padua, Italy " sominwadhwa@cs.umass.edu (S. Wadhwa); zamani@cs.umass.edu (H. Zamani)
© 2021 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org)
1In this paper, we use CIS to refer to all conversational information seeking and access systems, including conversational search, recommendation, and question answering.
2https://sites.google.com/view/swirl3/ of three major steps, that are aligned with the three di- the other hand, contains the interactions that can be inimensions in our taxonomy. We further review technical tiated at a later time decided by the system.3 Therefore, challenges in both modeling and evaluating each of these the interaction time in instant initiation is derived by steps in addition to discussing potential approaches for the user’s situational context, e.g., user’s location, time, end-to-end evaluation of system-initiative CIS systems. mood, and activity, or the urgency of the interactions We also highlight the dangers of using system-initiative (e.g., health and safety related interactions), while in OMI, interactions in CIS systems if not designed carefully. We this is the CIS system that decides the interaction time. ifnally briefly introduce the broader impact of this research direction. We believe this paper, despite being 2.2. Dimension II: Initiation Purpose sometimes abstract or hypothetical, sheds light on some aspect of developing and evaluating system-initiative conversational information seeking systems.
ity of a new data that may be of interest to user, by the current situation of user such as time and location, or by modifications to the CIS system . The latter may happen 2. A Taxonomy of for example if a new deployment of the CIS models leads System-Initiative CIS to an understanding that the system provided false information to a sensitive topic in the past interactions and
Interactions now wants to initiate a conversation to correct its past In this section, we review diferent interactions that may mistake. Given these three triggering reasons, we identify be taken by a CIS system to initiate a conversation. We ifve main purposes for initiating a conversation in a CIS study these interactions with respect to the following system. They include information filtering, recommenthree orthogonal dimensions: dation, following up a past conversation, contributing to multi-party conversation, and feedback request. Note • initiation moment: when to initiate a conversation? that this paper only focuses on information seeking conversations, therefore there exist some non information • initiation purpose: why to initiate a conversation? seeking initiation purposes that are not covered in this • initiation means: how to initiate the conversation? section.
In the following, we describe each of the identified ini
We believe that any CIS system should be able to an- tiation purposes. For each initiation purpose presented swer all the above questions in order to make system- below, we provide instant initiation and opportune moinitiative interactions. In the rest of this section, we ment initiation example use-cases in Table 1. explain these dimensions. This paper also proposes a pipeline for system-initiative interactions in CIS systems, Filtering streaming information Information filterwhich is inspired by these three dimensions introduced ing systems aim for delivering information to the user in the taxonomy. from a stream of information contents based on the user’s preferences. Belkin and Croft [13] identified information 2.1. Dimension I: Initiation Moment retrieval and information filtering as two sides of the same coin, because of their fundamental similarities in Given the first dimension, i.e., when to initiate a con- representing unstructured or semi-structured documents versation, we partition system-initiated conversational and computing their relevance to the user’s (short- or interactions into two categories: long-term) information needs. A few years later, Robertson and Hull [14] organized the TREC Filtering Tracks to • Instant initiation: defined as instant initiation of a promote the field and provide resources for fostering reconversation is by a conversational information seek- search in the filtering tasks. Conversational information ing system mostly based on the user’s current situation. seeking systems may initiate a conversation with the goal • Opportune moment initiation (OMI): defined as of information filtering. For instance, introducing the initiation of a conversation that can be postponed to breaking news headlines based on the user’s preferences an opportune moment that is decided by the conversa- is considered as an information filtering task that may tional information seeking system. have applications in system-initiative CIS systems.
considered as a subcategory of information filtering sys
nient time.
Health and safety related information is often time-sensitive. For instance, attacks or events that may lead to a safety risk or hazard for the user should be instantly mentioned by a CIS system that is watching these streaming information sources. Many users create and maintain to-do lists for their daily activities. A few recent recommender systems have been developed to re-rank and recommend the next to-do item. Some of the items in a to-do list can be timesensitive and a CIS system can instantly initiate a conversation to notify the user that the deadline for doing one of the yet-to-be-done tasks in the to-do list is approaching, otherwise the user will not be able to complete the task.
Any modification to the system’s response for a health or safety related question of the user which was asked in the past may need a prompt conversation initiation. For instance, if the user asks about the number of daily COVID-19 cases in an institute, and the system responds with zero, it may need to instantly initiate a conversation upon discovering a new case in the day. (note that many examples in this category also involve filtering of streaming information, however such filtering should happen with respect to the past user-system interactions, which is different from the first row in this table.) While it is largely unexplored in the literature, one possible use-case of a systeminitiative CIS engagement in a humanhuman interaction could be that of monitoring the factual accuracy of the underlying content exchanged in human conversations (if and where necessary). The CIS system may engage in retrieval-based fact-checking and initiate a conversation to contribute to the ongoing human conversation by providing the fact-checking results and details. Asking for a location- and time-specific feedback may need to happen promptly. For example, while a user is driving and passing by a specific location, a CIS system may initiate a conversation for feedback request by asking about a car accident in that location.
News agencies are constantly publishing new content on their website. Users, on the other hand, have diferent preferences and tastes in the news topics and sources. A system-initiative CIS system may initiate a conversation, based on the opportune moment initiation scheme, to inform the user based on their preferences.
Active engagement through CIS can also occur in broad opportune moments like the pre-holidays. People are often known to exchange gifts during some special occasions and holidays and a CIS could play an active role in ofering gifting recommendations to the user. Such an active engagement would be time-sensitive, and in addition to user-preferences for gift recommendations, a window-of-initiation would be equally as relevant.
CIS systems are not by any means perfect and they make mistakes in responding to user’s requests. Based on new information or new models deployed in the system, a CIS system may initiate a conversation at an opportune moment to accept and correct its mistakes that was made in the past.
Similar to the previous case with a focus on the monitored past human conversations (i.e., following up a past human conversation).
An example of an opportune moment feedback request is that of e-commerce shopping. Under the current popular systems, users often indiscriminately required to provide reviews of products right after they purchase them or after a pre-defined period of time. Factoring-in the category of products along with user meta-data could enhance a CIS’s ability to gauge what moments would be most opportune in terms of engaging an active conversation about seeking product feedback. tems, however, we intentionally separate these two in this paper to highlight their diferences and important applications in system-initiative CIS systems. Unlike information filtering tasks that deal with a stream of data, in this paper, recommendation tasks refer to recommending entities or information from an existing data source. For instance, recommending a restaurant based on the user’s location and preferences can be considered as a recommendation task but does not fit well within the definition of information filtering tasks provided above. CIS systems may initiate a conversation to make a recommendation to the user.
Following up a past conversation A CIS system may follow up a past conversation for many diferent reasons, such as providing new information that was not available at the time of past conversation, correcting a mistake that was made by the system in a past conversation, and continuing a conversation that was interrupted and left incomplete. System-initiation enables CIS systems to follow up past conversations to better serve their ultimate information seeking and access purpose.
Existing conversational information seeking systems are mainly designed for user-system interactions. However, CIS systems can contribute to multi-party human conversations, such as collaborative conversations. For instance, based on a conversation between two people, a CIS system that is permitted to monitor the conversation may contribute to the topic of the discussion, e.g., by factchecking the claims made in the conversation and taking an initiative if a false claim is made by one party.
related to information seeking, however, user’s feedback, such as product reviews, plays a key role in development Step I: Producing system initiation instances In of several information seeking systems. On the other the first step, system initiation instances are produced hand, users often forget or refuse to provide a feedback. by the processes described in Section 2.2, such as recomIn some cases, a CIS system may initiate a conversation mendation and contributing to a multi-party conversawith the goal of collecting feedback about the user’s ex- tion. They are shown as initiation purposes in Figure 1. periences. Such conversation may convince the users to These processes monitor the environment and produce provide feedback in cases where they normally do not. instances that can lead to system initiation by observing new filtered information or recommendation based on 2.3. Dimension III: Initiation Means the user’s context (see Section 2 for more detail about these processes). The produced conversation initiation How to initiate a conversation shapes the third dimen- instances will be added to the instances collection (or sion in our conversation initiation taxonomy. In case database). Note that a system initiation instance is a data of multi-device setting, the system should decide which object that contain all the information required for inidevice should be used to initiate a conversation. Or in tiating a conversation, including the initiation purpose, case of multi-modal setting, the system should decide the data, context, or reason that led to the production which interaction channel (e.g., visual through a screen of the instance, the initiation features and content, etc. or aural through the speaker) or processing modality This step can be formalized as a function of , , and (e.g., verbal through text or non-verbal through an im- that produces one or more system initiation instances, age) should be used for initiating a conversation. One can imagine a system that can ask for a permission to initiate a conversation, for example via a light vibration.
conversations can be initiated by new information, by the situational user context, and by new model deployment. In this section, we present a general high-level pipeline for initiating a conversation in CIS systems. Due to the complexity of developing and evaluating the pipeline for system-initiative interactions, we additionally provide a formal definition of each step. This formalization enables us to easily discuss evaluation methodologies for each component in Section 6. It also helps future work to see these steps in isolation. The pipeline is depicted in Figure 1. It consists of the following steps that use the notation introduced in Table 2. past user-system interactions lection for initiating a conversation. Although some conversations need to be initiated promptly (i.e., instant initiation), in this pipeline all instances are inserted into and this is the job of the initiator to promptly identify instant initiation requests. In more details, the initiator component is constantly monitoring all entries in and based on the user’s situational context decides what instance should be selected at each timestamp for conversation initiation. This step can be formalized as (, ) = Pr(initiation = 1|, ) where is a system initiation instance in and “initiation” is a binary hidden variable representing the event of initiating the conversation. Note that although everything mentioned in this paper is about system-initiative conversations, note that the initiator can be also triggered by the user (for instance, the user may say “I’m board, tell me something”).
Once the initiator component selects one of the instances from , a natural language utterance will be generated by the conversation generation component and it will be presented to the user based on an appropriate device and interaction modality (in case of multi-device or multi-modal settings). Therefore, this step can be formally defined as a function that generates a conversation based on a given instance and presented to the user based on the user profile and situational context, i.e., (, ).
While users are free to respond in any form they may see fit, for a substantive functioning of the system we propose a certain categorization of responses based on how they are processed by the system: • Null action: User provides no response to the initiated conversation by the CIS system. Note that null action should not necessarily be interpreted as a negative feedback, since the user may find some initiation useful, while they are not interested in further engagement. • Interruption or negation: User provides a response consistent with the interpretation of shutting down any further engagement by the CIS system. Such response can be safely assumed as a negative feedback. • Relevant response: User responds to the initiated conversation by a relevant answer. This is often expected to happen when the initiated conversation involves a question or asks for feedback. • Postpone: User responds to the initiated conversation
and asks the system to remind them at a later time. • Critique or clarification-seeking response: The kind of responses here would include users further engaging in a back-and-forth conversation with agent about ei- goal is either filtering of streaming information, recomther seeking further information or critiquing existing mendation, conversation follow-up, contributing to a engagement. One key technical challenge here that we multi-party conversation, or feedback request. talk about in the next section would be the processing of the user response in order to inculcate it to make 5.2. Developing an Initiator Model the system better. • Follow up: User responds with a follow up response to get further information or perform actions related to the initiated conversation. • Topic drift: User responds but changes the topic of the initiated conversation.
Given the current status of text classification models and the complexity of the task, it is possible to achieve an acceptable classification accuracy in classifying user responses to the above categories. They can be further used for training or evaluating the conversation initiation process. For instance, interruption or negation may be considered as negative feedback. Such feedback can be used to modify the models deployed for each of the three steps (, , and ) in the pipeline (see Section 3). On the other hand, receiving a relevant response may be considered as a positive feedback for the system.
5.1. Producing System-Initiative
Instances The second step in the provided pipeline is selecting a system-initiative instance from the instance collection by an initiator component (see Figure 1). This is indeed equivalent with implementing the function (, ). We believe that the most challenging part of implementing such component is our lack of knowledge on what is generally the right moment for initiating a conversation. Therefore, we believe that future research should focus on conducting user studies in the wild to explore what are the right time to initiate a conversation. Some weak supervision signals can be mined from user interactions with the current conversational systems, even if they do not support system-initiative interactions. For example, the times when the user initiate an unimportant conversation (due to being board for example) can provide a weak (noisy) signal as a potentially good time to initiate a conversation and thus machine learning based models can be trained based on the situational context and the user profile to predict such moments. Of course, a nice property of interactive systems that log the user interactions is to iteratively improve the system ability to accurately predict such moments based on the feedback received from the user (see Section 4 for various types or user responses to system-initiative interactions). 5.3. Generating System-Initiative
Utterances
The first step in the system-initiation pipeline is to iden- to generate a conversation based on a system-initiative tify reasons for initiating a conversation and generate instance and present it to the user, equivalent to implea system-initiative instance. As described in the last menting the function (, ). We believe that many section, system-initiative instances are data objects that techniques developed in the dialogue systems and text contain all the information about a system-initiative con- generation research can be used for implementing this versation, such as the purpose, content, and context. component. Each instance is a structured data object, This step can be cast to implementing each of the five therefore, neural models for unstructured text generainitiation purpose components discussed in Section 2.2. tion from structured data, e.g., tables, can be potentially In other words, one needs to implement the function adopted. Since the users mostly do not expect system(, , ) with a focus on each initiation purpose. initiative utterances, an interesting technical challenge This has roots in various IR tasks, such as filtering and here would be providing some context in the generated recommendation. However, some of the initiation pur- utterance to make sure that the user understands why poses are relatively unstudied in the literature, such as such conversation being initiated. This context may refer following up a past conversation or contributing to a to a previous interaction of the user with the system, a multi-party conversation. Even feedback request in the past experience of the user, or an explanation on the reaform of active conversation is underexplored. There- son that led to the generation of such system-initiative fore, one of the major technical challenges in producing conversation. system-initiative instances is to develop models that can identify the reasons for conversation initiation when the
system-initiative CIS systems. IR research has a long history of collection creation for various information seeking tasks, however, they are mostly created based on a set of pre-defined information needs (e.g., most TREC tracks) or a set of observations (e.g., clickthrough data). Such evaluation methodologies do not easily extend to an active interaction scenarios, such as system-initiation in conversation.
Although evaluating system-initiative CIS systems is yet to be explored in the literature, in this section, we detail our perspective on potential evaluation methodologies that can be pursued. information related to the user’s profile and context (e.g., time and location), formalized as (, ) in Section 3. This approach assumes that the importance of initiation moment is binary (good or bad). However, this is not the case. Two situations may be bad for initiating a conversation, but one may be the worst. Therefore, we believe that this task should be evaluated as a ranking task: rerank the list of situational context information associated with a user given for an initiation command. This setting allows us to have multi-level (or graded) labels for each situational context and use relevant metrics (e.g., similar to NDCG [16]) to evaluate the quality of the system in identifying the right situation (moment) to initiate the conversation. 6.3. Evaluating the Content of Initiated Conversations ( ) 6.1. Evaluating system initiation
instances () stanced formalized using a function (, , ). The initiation command should include all information about the nature of conversation initiation. To evaluate this component, we should provide all the required information at the timestamp to the system as input and evaluate the produced initiation command. The required information (as depicted in Figure 1) includes past user-system interactions, user profile, user situational context, and a stream of new information content. The model should produce an initiation command or NUll, meaning that no initiation is needed. Both precision and recall-oriented metrics should be used to evaluate the model’s performance. In fact, the produced initiation instances should be of high quality (all initiations should be relevant to the user) with high coverage (all required initiations should be produced by the model).
Based on this evaluation methodology, a reusable collection can be created. The data collection can be either sampled from a real user’s interaction history (a realistic setting, but requires access to real user-system conversational interactions), or constructed based on information seeking interactions between two or more people. The latter can be done in a lab study using a wizard of oz setting, similar to [15].
system needs to produce a natural language sentence or As pointed out in Section 3, the first step towards initi- utterance (in most cases) and select the initiation means ating a conversation is to produce system initiation in- if needed for initiating the conversation. In Section 3, we formalize this as (, ). To evaluate this ability of the system, we can assume that the initiation commands are accurately produced and are complete (i.e., a hypothetically ideal system with perfect precision and recall).
Based on this assumption, the focus of this evaluation step would be to generate a conversation utterance based on a given initiation instance . The generated utterance should contain all required information in addition to being precise and fluent. A number of ground truth reference utterances can be generated through manual annotation and popular text generation metrics such as BLEU [17], ROUGE [18], and BERTScore [19] may be used to evaluate the model. As discussed in [20], despite the popularity of these metrics, they do not necessarily relfect the quality of the produced dialogue, and ultimately human annotation of the model’s outputs is desired. 6.4. End-to-End Evaluation of
System-Initiated Conversations 6.2. Evaluating Initiation Moments ( )
pipeline (see Figure 1), one can cast the problem to a binary classification task. In more detail, we can formalize the task as predicting whether to initiate the conversation or not given an initiation instance and a set of
uation of system-initiative CIS systems. As mentioned above, each is based on some simplifying assumptions of other components of the system, which is unrealistic.
Therefore, an end-to-end evaluation of system-initiated conversations should be explored. To do so, both ofline and online evaluation strategies can be adopted. For ofline evaluation, each instance would include all the required information for the system at a timestamp as input, including past user-system interactions, user profile, situational context, and a stream of new information. The model will be evaluated based on the produced system-initiated conversations (if needed). Having a sin- own. For example, one lingering question could be, how gle evaluation metric that can reflect all aspect of con- should we derive user consent of all the humans involved versation initiation evaluation would be challenging and whose data the system processes? Fully studying privacy require further investigation. Approaches like economic implications of such a system would require extensive models of interactive information retrieval that model the user studies, often on a task-by-task basis, and assessing system by assigning cost and benefit to each interaction perception of the system behavior itself on the end-users. may be relevant. In case of online evaluation, the typical A/B tests can be used to evaluate the system, and the system can be evaluated by interpreting the positive and negative feedback received from the user. Such feedback can be obtained by identifying the user response type (see Section 4).
most important components of an active engagement CIS would be its initiator decision making system that decides when to initiate a conversation and perhaps more importantly, when not to initiate one. Engagements made by the system at a bad time can be counter-productive or 6.5. End-to-End Evaluation of even downright dangerous. For example, while initiating Mixed-Initiative Conversations a non time-sensitive conversation, the agent must not disturb or distract the user in any way. An unexpected System-initiated conversations are just one type of inter- system engagement when the user is engaged in a critactions that a mixed-initiative CIS system may support. ical activity, e.g. driving, can be extremely dangerous There exist several other interactions, such as the typical and distracting. Therefore, while developing an initiator user-initiated information seeking conversations and ask- module we must also account for actively penalizing the ing clarifying questions for intent disambiguation [21]. module if it engages at a particularly bad time. The ultimate evaluation methodology should assess the quality of the system in all of these diferent settings.
Such complex end-to-end evaluation can be again done 8. Broader Impact using both online and ofline evaluation using a data that contains all diferent sorts of interactions. Similar approaches as the one mentioned in the last subsection can be adopted, however, designing an evaluation metric for this purpose would be even more challenging.
Privacy Concerns Even with existing conversational information systems, users often have privacy concerns about how their information is processed, to whom it is disclosed and what is the associated risk [22]. We envision that those concerns will only be exacerbated, if left unaddressed, with a system capable of processing far more sensitive user information and engaging in an active form of conversation. Hence we believe that certain privacy concerns must be addressed while designing and implementing active CIS systems. Secure information retrieval and data sharing protocols would be needed to safeguard and ensure users that their identifiable information remains secure. Ensuring and safeguarding user information may or may not instill a sense of security among the end users if the activity format of the underlying system comes of as too intrusive. For instance, one of the use cases for an active engagement CIS is that of contributing to a multi-party human conversation (Table 1). Active system engagement in a multi-party setting has been a largely unexplored area in the IR literature and raises new and unique privacy concerns of its
system-initiative interactions in conversational information seeking systems. The authors believe that research progress in modeling and evaluating system-initiative CIS can potentially lead to a broader impact. Several health and safety related conversations can be initiated by CIS systems to warn users of potential harms and hazards. Such system-initiative interactions can be triggered based on the user’s situational context, such as location or health-related signals captured by various sensors embedded into smartphones and wearable devices. Furthermore, these systems can potentially inform the victims of misinformation or abusive content which targets the users through human conversations, written documents, or ads. Diferent types of entertainment can also be an application of system-initiative interactions, which can be or not be related to information seeking.
Moreover, with the progress of virtual and augmented reality devices, system-initiative interactions (especially those with the information seeking nature) would be of great importance, since the user can experience a virtual environment and a system-initiative CIS can guide the users as they are exploring the virtual environment.
The study of interaction has a long history in information retrieval research, starting in the 1960s [23]. Much of the earlier research studied how users interacted with intermediaries during information seeking dialogues but this rapidly shifted to studying how users interacted with op- and push notifications in desktop and mobile apps (Secerational retrieval systems, including proposals for how tion 9.6). In the following, we present an overview of to improve the interaction. Information retrieval systems these related domains to position our work in context. based on this research were also implemented. Oddy [24] developed an interactive information retrieval sys- 9.1. Mixed-Initiative Interactions tem with rule-based dialogue interactions in 1977. Croft and Thompson [25] later proposed the first interactive Most approaches to human-computer interactions with information retrieval system that models user, I3R, using intelligent systems are either controlled by human or a mixture of expert architecture. A few years later [26] system. However, developing intelligent systems that characterized information seeking strategies for interac- support mixed-initiative interactions has always been detive IR, ofering users choices in a search session based sired. Allen et al. [7] believed that development of mixedon case-based reasoning. initiative intelligent systems will ultimately revolution
Since the development of web search engines, research ize the world of computing. Mixed-initiative interachas mostly focused heavily on understanding user in- tions in dialogue systems have been explored since the teraction with search engines based on an analysis of 1980s [34, 35, 36]. Early attempts to build systems that the search logs available to commercial search engine support mixed-initiative interactions include the Lookproviders. Since then, explicit modeling of information Out system [37] for scheduling and meeting manageseeking dialogues or conversations with the aim of im- ment in Microsoft Outlook, Clippit4 for assisting users proving the efectiveness of retrieval has not been a focus in Microsoft Ofice, and TRIPS [ 38] for assisting users in of research until recently. One exception is the TREC problem solving and planning.
Session Track [27] that focused on the development of Horvitz [37] identified 12 principles that systems with query formulation during a search session and improv- mixed-initiative user interfaces must follow. In summary, ing retrieval performance by incorporating knowledge of mixed-initiative interactions should be taken at the right the session context. On the other hand, commercial per- time in the light of cost, benefit, and uncertainties. Many sonal assistants such as Apple Siri and Google Assistant factors can impact cost and benefit of interactions that are have become commonplace and there is a clear incen- covered in multiple principles. In addition, systems with tive to develop better conversational models for search. mixed-initiative interactions should put user at the cenA promising development has been the efectiveness of ter and allow eficient invocation and termination. They neural models for generating conversational responses are expected to memorize past interactions and continwhen trained on large amounts data (e.g., [28]). uously learn by observation. Based on these principles,
In recent years, conversational information seeking conversational systems by nature raise the opportunity systems have attracted attention in both academia and of mixed-initiative interactions. the industry [1, 29]. They include conversational search, Allen et al. [7] defined four levels of mixed-initiative recommendation, and question answering systems. CIS interactions in the context of dialogue systems, as folsystems are suficiently broad to cover a wide range lows: of tasks. The research community has so far studied 1. Unsolicited reporting: An agent notifies others of a number of them, including conversational answer re- critical information as it arises. For example, an agent trieval [2], conversational answer extraction (often re- may constantly monitor the progress for the plan unferred to as conversational question answering) [3], con- der development. In this case, the agent can notify versational query re-writing [30], next question pre- the other agents (e.g., user) if the plan changes. diction [31], speech-only interfaces for conversational search [32], and question-based recommendation (often 2. Subdialogue initiation: An agent initiates subdiareferred to as conversational recommendation) [33]. In logues to clarify, correct, and so on. For example, all of these tasks, the user initiates the conversation with in a dialogue between a user and a system, the systhe CIS system and the system responds. Even in case tem may ask a question to clarify the user’s intent. of existing conversational recommender systems, the Since the system asks the question and the user should conversations are initiated by users [33]. In this work, answers the question, and clarification may take mulhowever, we discuss challenges and possible solutions for tiple interactions, the system has temporarily taken extending existing models to support system-initiative the initiative until the issue is resolved. This is why it conversations. is called subdialogue initiation.
System-initiative conversations are indeed related to mixed-initiative interactions (Section 9.1). There are 3. Fixed subtask initiation: An agent takes initiative some other related research directions that may be out- to solve predefined subtasks. For example, if an agent side of the IR community, including dialogue acts (Sec- is supposed to complete a task that involves multiple tion 9.2), system-initiative dialogue systems (Section 9.3) 4https://en.wikipedia.org/wiki/Ofice_Assistant subtasks. In this case, the agent can takes initiative to ask questions and complete the subtask. Once the subtask is completed, initiative reverts to the user. 4. Negotiated mixed-initiative: Agents coordinate and negotiate with other agents to determine initiative. This is mainly defined for multi-agent systems in which agents decide whether they are qualified to complete a task or it should be left for other agents.
Some of the personalization methods leverage long-term user behavioral histories [45], while others analyze shortterm implicit feedback [46]. A key challenge that we see with personalization, especially when applied to activeengagement conversational systems, is that of collecting user profiles with suficiently rich features, while balancing privacy concerns. We leave that as an essential component of our future work.
When it comes to open-domain conversational infor- 9.3. Initiative Control in Dialogue mation seeking, some of these mixed-initiative levels Systems remain valid. Mixed-initiative conversational information seeking has relatively less explored, nevertheless Discourse segmentation through transfer of control in identified as critical components of a conversational sys- dialogue systems was first studied decades ago by Walker tem [6, 39]. Perhaps clarification [ 8, 40, 10] and pref- and Whittaker [47] to enable flexible human-computer erence elicitation [11, 12] are the two areas related to conversations to take place that allow for corrections mixed-initiative interactions that have attracted much and clarifications. Since then a number of studies have attention in recent years. However, they are mostly been done to determine the ideal behavior of a virtual unrelated to system-initiative interactions, which are assistant [48, 49]. One of the key aspects of such an ideal relatively unexplored. Nevertheless, the unsolicited re- behavior is initiative. In prior work, a number of authors porting level of mixed-initiative interactions mentioned have considered what constitutes initiative [50, 51, 52]. above include several interesting example use-cases for Instead of a human, at certain relevant points in time, system-initiative CIS systems. the system may take the initiative to engage in a conversation. Among all such systems, there is some form 9.2. Dialogue Acts in Conversational of dialogue management component which determines Systems what to prompt for and/or what to accept next based on the conversation history and its context [53]. Such a Spoken dialogue systems (SDS) have allowed for interac- management component plays a central role in the tradition with computer-based applications (e.g., smart speak- tional architecture of a dialogue system and is primarily ers) through spoken natural language. Certain SDS mech- concerned with the flow of the dialogue (information anisms are specifically designed to carry out well-defined providing, feedback request, etc) while simultaneously tasks, e.g., scheduling [41], and most of them are based maintaining a discourse history. For instance, Vakulenko on a finite state-based dialogue control. Although the et al. [54] have shown how an agent might efectively focus of CIS research is mostly on open-domain informa- take initiative to elicit or clarify information when aption seeking tasks, such dialogue acts can be potentially propriate. used to support a diverse set of modes and scenarios in In addition to standalone engagement by a conversystem-initiative CIS systems. A range of prior studies in sational agent, studies have also shown that sources of dialogue acts also ofer insights into designing models for information that led to that engagement are equally imconveying information through conversations e.g. prior portant – e.g., diferent sources have varying influence work by Bunt et al. [42] ofers promising features de- on purchase decisions, implying that the efectiveness rived from broad dialogues to better model information of a conversational information system depends on the needs, however in our work we assume that an alternate system saying why it made a specific decision or recommethod might be required to extract similar information- mendation [28, 55]. needs from user meta-data. Dialogue acts can potentially serve as communicative functions of dialogue segments, 9.4. Contextual Suggestions such as request, inform, question, suggest and ofer . The general taxonomy of dialogue acts is complex with dif- Contextual suggestion track within TREC [56] is aimed ferent markup schemes. One segment of particular in- at providing personalized point-of-interest recommenterest to us, and often not examined in the IR literature, dations to users in a ranked manner. The task assumes is that of turn taking [43]. For instance, our conversa- a certain setting – a user in a specific place (geographic tional agent will have control over the dialogue and the location) with a trip type. Given the same user’s personal segments might be produced through an analysis of user profile (interests, endorsements etc), the system makes data. Such an analysis over user meta-data (e.g. Location) recommendations for attractions. The track consisted of to personalize an IR task isn’t new, and is most commonly two phases, Phase 1: participants could select any venue applied in the context of personalized web-searches [44]. from the reference collection. Phase 2: participants had to rank a given list of venues for each user and thus allowing 9.7. Information Need in Collaborative for ground truth data against which the system could be Conversations evaluated. Early works on this task involved rule-based approaches by mapping user-profiles to specific venues. Over time, a number of definitions for information need Recently, people have experimented with standard ma- have been conceptualized [70, 71]. For our work, we chine learning [57] and neural methods [58, 59] for best consider the one by Case [72] i.e. information need is mapping user profiles to relevance-rated documents. For a recognition that the user’s knowledge is inadequate to example, Seyler et al. [58] create graph embeddings from satisfy their own goals, as it implies that the information a heterogeneous information network (HIN) using the need must emerge from the user’s end. Collaborative conTREC Contextual Suggestion dataset achieving state-of- versations ofer one such instance, as articulated by Shiga the-art performance. et al. [73], that information needs in such conversations are naturally verbalized and therefore can be captured by end-user devices. Furthermore, we can utilize the tax9.5. Incident Streams onomy of information needs defined by Taylor [74] to TREC-IS track in 2018 [60] focused on curating feeds of diferentiate between perceived needs and actual queries social media posts and classify them based on actionable since Taylor’s model consists of visceral, conscious, forinformation for enhanced situational awareness (such as malized and compromised needs. For the purposes of a emergencies). Incident streams are relevant in context conversational information system to actively engage in of system-initiative conversations due to the underlying a collaborative discussion, we primarily focus on the connature of the task – analyze large sets of textual infor- scious needs, which are defined as “ambiguous and rammation related to user profiles and act in a time-sensitive bling statement” but ultimately evolve into formalized manner. In addition to the type of information, TREC-IS needs (qualified and rational). Prior work by Jansen et al. evaluation tasks also include a criticality-score indicat- [75] on analyzing conversation query logs has shown that ing how important it is for a specific content to be acted users often frame a short and under-specified query to upon. information seeking systems, however community-based modern QA models are often capable to formalizing such 9.6. Push Notifications information needs on QA sites or speech-oriented search systems. The degree of interest in collaborative converPush notifications have been mostly studied in the con- sational information search has increased since then and text of mobile applications, largely with e-commerce has led to quantitative analysis of conversations during goals [61, 62, 63]. Much of the prior research on push no- search [76, 77, 78]. For example, Foster [79] performed tifications has focused on their disruptive nature [ 64, 65]. a full discourse analysis on group conversations to deFor example, Mehrotra et al. [66] provided an in-depth ifne the relationship between functions of verbal context study evaluating how the user-response time of a non and information seeking activity. While these studies time sensitive notification is influenced by the notifi- remain either conceptual or use small amounts of textual cation’s presentation, modality as well as the sender- chat data, they nevertheless suggest that collaborative recipient relationship. Mehrotra et al. [67] further de- conversations can be a useful source for conversational tailed, through extensive user studies, how push noti- information seeking. ifcations with diferent context and timings can cause In this research, we also highlight some applications disruptions. This is an especially important component of system-initiative conversational interactions in the since one of the main goals of an active engagement context of multi-party and collaborative conversations. system is to minimize disruptions caused to the enduser. Other works in the area have explored the use of push notifications for meta-learning [ 68] and self-logging 10. Summary [69] to better adapt the underlying framework for adjusting user preferences. Push notifications are basically system-initiative interactions, however they mostly do not concern with information seeking tasks and are fundamentally diferent from system-initiative interactions in conversational systems.
In this work, we explored applications and the ways to model an active engagement conversational information seeking system. We defined a taxonomy upon which a framework for an active engagement system could be built. Our taxonomy defines three broad dimensions of an active engagement framework – initiation moment (when to initiate a conversation), initiation purpose (why to initiate a conversation) and interaction means (how to initiate a conversation). Subsequently we show, through the explained examples and a pipeline, how the described
This work was supported in part by the Center for Intelligent Information Retrieval. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors. characteristics are both necessary and suficient to allow for the functioning of an active engagement information seeking system, for a number of initiation purposes.
In doing so, we also generalized several components of the pipeline that have been implemented before with proven efectiveness. We view the contribution of our work as this taxonomy and the proposed generic pipeline, which can be employed towards building true activeengagement information seeking systems. Implementing and evaluating the proposed framework in a user-centric way remains the most important future directions suggested by this work. It is further worth considering the the numerous technical and evaluation challenges that come with the proposed approach. Finally, we believe that our identified use cases of active engagement CIS systems would only serve as founding basis for other, broader case-specific applications such has aiding people with disabilities, as highlighted in section 8 and we hope this work spurs additional work in this largelyunexplored area of information seeking research. [6] F. Radlinski, N. Craswell, A theoretical framework for conversational search, in: Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, CHIIR ’17, ACM,
New York, NY, USA, 2017, p. 117–126. [7] J. E. Allen, C. I. Guinn, E. Horvitz, Mixed-initiative interaction, IEEE Intelligent Systems and their Applications 14 (1999) 14–23. [8] M. Aliannejadi, H. Zamani, F. Crestani, W. B.
Croft, Asking clarifying questions in open-domain information-seeking conversations, in: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’19, ACM, New York, NY, USA, 2019, p. 475–484. [9] H. Zamani, M. Bendersky, X. Wang, M. Zhang, Situational context for ranking in personal search, in: Proceedings of the 26th International Conference on World Wide Web, WWW ’17, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 2017, p.
1531–1540. [10] H. Zamani, B. Mitra, E. Chen, G. Lueck, F. Diaz, P. N.
Bennett, N. Craswell, S. T. Dumais, Analyzing and learning from user interactions for search clarification, in: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, ACM, New York,
NY,
Coached conversational preference elicitation: A
case study in understanding movie preferences, in:
[1] J. S. Culpepper, F. Diaz, M. D. Smucker, Research Proceedings of the 20th Annual SIGdial Meeting on
frontiers in information retrieval: Report from the Discourse and Dialogue, ACL, Stockholm, Sweden,
third strategic workshop on informa
in lorne (swirl 2018), SIGIR Forum 52 (2018) 34–90. [12] A. Sepliarskaia, J. Kiseleva, F. Radlinski, M. de
Ri[2] C. Qu, L. Yang, C. Chen, M. Qiu, W. B. Croft, jke, Preference elicitation as an optimization
probM. Iyyer, Open-Retrieval Conversational Question lem, in: Proceedings of the 12th ACM Conference
Answering, SIGIR ’20, ACM, New York, NY, USA, on Recommender Systems, RecSys ’18,
tics 7 (2019) 249–266. [14] S. Robertson, D. A. Hull, The TREC-9 Filtering [4] Y. Sun, Y. Zhang, Conversational recommender Track Final Report, in: Proceedings of Ninth Text system, in: The 41st International ACM SIGIR Con- REtrieval Conference, TREC-9, National Institute ference on Research & Development in Information of Standards and Technology, Special Publication, Retrieval, SIGIR ’18, ACM, New York, NY, USA, 2000, pp. 25–40.
2018, p. 235–244. [15] A. Weiss, R. Bernhaupt, D. Dürnberger, M.
Alt[5] P. Thomas, D. McDuf, M. Czerwinski, N. Craswell, maninger, R. Buchner, M. Tscheligi, User experience
Misc: A data set of information-seeking conver- evaluation with a wizard of oz approach: Technical
sations, in: Proceedings of the 1st International and methodological considerations, 2010, pp. 303 –
Workshop on Conversational Approaches to Infor- 308.
mation Retrieval, CAIR ’17, 2017. [16] K. Järvelin, J. Kekäläinen, Cumulated gain-based
evaluation of ir techniques, volume 20, ACM, New
ration of evaluation metrics for mobile push notifi- [74] R. S. Taylor, The process of asking questions,
Amercations, in: Proceedings of the 39th International ican Documentation 13 (1962) 391–396.
ACM SIGIR Conference on Research and Devel- [75] B. J. Jansen, A. Spink, T. Saracevic, Real life, real
opment in Information Retrieval, SIGIR ’16, ACM, users, and real needs: a study and analysis of user
New York, NY,
D. Weber, A. Schmidt, Large-scale assessment of [76] R. Kelly, S. J. Payne, Collaborative web search in mobile notifications, in: Proceedings of the SIGCHI context: A study of tool use in everyday tasks, in: Conference on Human Factors in Computing Sys- Proceedings of the 17th ACM Conference on Comtems, CHI ’14, ACM, New York, NY, USA, 2014, p. puter Supported Cooperative Work & Social Com3055–3064. puting, CSCW ’14, ACM, New York, NY, USA, 2014, [65] M. Pielot, K. Church, R. de Oliveira, An in-situ study p. 807–819.
of mobile phone notifications, in: Proceedings [77] C. I. Seeking, Collaborative information seeking of the 16th International Conference on Human- best practices, new domains and new thoughts | Computer Interaction with Mobile Devices & Ser- preben hansen | springer, 2015. vices, MobileHCI ’14, ACM, New York, NY, USA, [78] C. Shah, R. González-Ibáñez, Exploring information 2014, p. 233–242. seeking processes in collaborative search tasks, in: [66] A. Mehrotra, V. Pejovic, J. Vermeulen, R. Hendley, Proceedings of the 73rd ASIS&T Annual Meeting M. Musolesi, My phone and me: Understanding on Navigating Streams in an Information Ecosyspeople’s receptivity to mobile notifications, in: Pro- tem - Volume 47, ASIS&T ’10, American Society for ceedings of the 2016 SIGCHI Conference on Human Information Science, USA, 2010.
Factors in Computing Systems, CHI ’16, ACM, New [79] J. Foster, Understanding interaction in information
York, NY,
Mining user’s preferences for intelligent mobile
notification management, in:
New York, NY,
D. Jurafsky, P. Taylor, R. Martin, C. Van EssDykema, M. Meteer, Dialogue act modeling for automatic tagging and recognition of conversational speech, Computational Linguistics 26 (2000) 339– 374. [72] D. Case, Looking for information—a survey of research on information seeking, needs, and behavior,
Inf. Res. 8 (2003).
[73] S. Shiga, H. Joho, R. Blanco, J. R. Trippas, M.
Sanderson, Modelling information needs in collaborative
search conversations, in: Proceedings of the 40th
International ACM SIGIR Conference on Research
and Development in Information Retrieval, SIGIR
’17,