In traditional web search interfaces, information seekers reformulate their queries by editing the terms in the search box in order to guide the retrieval process. Such kind of editing is at odds with the natural language interaction paradigm in conversational interfaces, and for purely voice-based interfaces it is impossible. Conversational search studies reveal that participants instead describe their changes to a query; however, the principles of such “editing conversations” have not been analyzed in depth. The paper in hand formalizes the problem of conversational query reformulation. We cast reformulations as meta-queries that imply operations on the original query and categorize the operations following the standard CRUD terminology (create, read, update, delete). Based on this formalization we crowdsource a dataset with 2694 human reformulations across four search domains. Our analysis of the meta-queries reveals a large variety in word usage and indicates ambiguous reformulations as an important research topic of its own.
During web search, information seekers frequently find a search engine’s results either too specific, too generic, or containing results relevant only to an unintended interpretation of their query. In such cases seekers may want to reformulate their queries [1, 2]. In a traditional search interface, the seeker would directly edit the previous query in the search field, creating, updating, or deleting terms. Such reformulations account for about half of all queries [3, Sec. 6.3]. However, conversational search interfaces—be they chat-like or voice-based—usually do not allow modifying the previous query. Though some chat interfaces not used for search allow to edit previous messages, such a functionality breaks temporal continuity, making the interaction significantly less conversational. Still, reformulations are also frequent in conversational search lab studies [4] and can be seen as one user-facing service of the search interface’s conversational layer, as illustrated in Figure 1 (a).
As the example in Figure 1 (b) illustrates, reformulations allow to specify information in small steps. As the main advantage of incremental formulation, seekers do “Show me news about COVID-19.” COVID-19
Search engine Conversation
layer Laconic layer
TCP/IP layer “OK, here is the latest news on COVID-19.” + <serp>
{”hits”: {”total”: 142, ... }, ...} “Ah. Please no articles about vaccination this time.” COVID-19 ∧ (¬ vaccination) “Sure, I remove articles on vaccination.” + <serp>
{”hits”: {”total”: 89, ... }, ...} time ...
... not have to formulate the complete query in advance, ask for information connected to information just resubstantially reducing the required mental efort. 1 More- trieved. However, whereas query reformulations change over, incremental formulation might simplify and thus the criteria that identify relevant information, follow-up increase the use of search operators for seekers, like the questions request a completely new answer. This difexclusion in Figure 1 (b), allowing seekers to formulate ference in intent causes linguistic diferences between even complex needs more intuitively.2 query reformulations and follow-up questions that war
Though the seeker may not consciously ask to create, rant separate investigations. Moreover, conversational update, or delete query terms, conversational reformu- query reformulation relates to much of the available relations are essentially such meta-queries that request search on queries. changes to the previous query. While these operations are not implemented in standard retrieval engines, one 2.1. Queries in Conversational Search can imagine a “conversation” layer on top of such engines (Figure 1 (a)) that—among others—resolves reformula- A central promise of the conversational search paradigm tions similar to co-reference resolution in conversational is to bring search closer to real-world assistance of a question answering systems (e.g., [6]). reference librarian [8]. In this regard, conversational
However, critical issues for conversational systems reformulations are one piece for allowing the seeker to
concerning reformulations have barely been analyzed specify their need on a more natural level [9]. Still, unlike
in the literature, especially the reformulations’ inherent “query by babbling” [
To foster research on conversational query reformula- In their history of IR research, Sanderson and Croft tions, we contribute the following: (1) a conceptualization [14] divided interactions against some first text-based that casts conversational reformulations as meta-queries conversational search systems into natural language or following CRUD terminology (cf. Section 3); (2) the first keyword-based and into non-querying and querying. dataset on conversational query reformulations,3 contain- In a later fine-grained study of conversational seeker ing 2694 messages and associated meta-queries, crowd- messages for passage retrieval, Lin et al. [15] categorize sourced from 284 study participants from 5 countries query ambiguity, though they focus on ambiguity regardin 4 diferent search domains (cf. Section 4); and (3) an ing (not) referenced entities, whereas the paper at hand in-depth analysis of the reformulations’ word patterns, focuses on ambiguity regarding the desired operation. emphasizing ambiguous word patterns as important re- Trippas et al. [4] present a model for spoken conversasearch direction and suggesting the general feasibility of tional search that also covers information requests bea domain-independent rewriting system (cf. Section 5). yond queries, for example, within a result document. In their lab study, they observe that both seekers conversationally reformulate queries (“query embellishments”) 2. Related Work and that participants in the system’s role conversationally ofer reformulations based on what they see.
Though conversational search is an active research area, conversational query reformulation has attracted little attention so far. In contrast, several recent publications target co-reference resolution for follow-up questions in conversational question answering [7]. Reformulations and follow-up questions are similar in that both
1This efect likely also holds for more instruction-like (but still conversational) interactions, like in Adobe’s “phonic filters” image search demo, https://blog.adobe.com/en/2019/05/29/preview-technology-givesyour-voice-the-power-of-a-creative-director.html.
2Some years ago, only about 1% of web search queries contained operators [5].
3Publicly available at https://doi.org/10.5281/zenodo.5031960
Query reformulations are queries based on the previous one with a similar information need [1]. Boldi et al. [1] classified query reformulations on two axes from generalization to specification and from being the same query to starting a new search mission. Sanderson and Croft [14] proposed a categorization based on the latter axis. Jiang et al. [16] analyzed voice query repeats after voice input errors. They found that seekers tended to stress words that the system misunderstood—a way of conversational reformulation that is specific to voice queries but outside the scope of the paper at hand.
Though studies took note of conversational query reformulations (e.g., [17]), they have rarely been analyzed. As an exception, Sa and Yuan [18] asked 32 participants in a Wizard of Oz study to perform one generalization and one specialization of a displayed query, speaking to the system like to a human. The participants preferred conversational query reformulations (which Sa an Yuan call “partial query modification”) over repeating the query with changes (“complete query modification”), even though several participants reported employing the latter for being used to it. We build upon this work, asking for more complex reformulations in longer query sessions and focusing on analyzing the language employed and ambiguities therein.
to processing the query before the retrieval. This task currently attracts much attention in conversational question answering, mostly concerning co-reference resolution. Available datasets for this task include CSQA [19], CoQA [20], QReCC [6], QuAC [21], and TREC CAsT [22]. The availability of datasets has already led to several approaches, often employing sequence-to-sequence learning [23, 24, 20] and previous interactions [25, 26, 27].
using natural language. While query reformulations in web search usually are stand-alone queries that can directly be submitted to retrieve results, formulating queries from conversational reformulations requires an additional step that “adds” the conversational context.
This section discusses the implications on three levels: (1) a model of conversational reformulations as metaqueries; (2) the problem of algorithmically understanding reformulations; and (3) the process of creating the “laconic” queries from the reformulations.
Several studies analyzed natural language queries even before conversational interfaces. Belkin et al. [28] found that, in a text-based search interface, the average query length increased by nearly 50% when the search box label encouraged to write a problem description. Still, the automatic “translation” of long queries to shorter keyword queries later also gained attention with systems reducing natural language queries to some key concepts more compatible with keyword-based interfaces [29, 30]. Moreover, 3.1. Casting Conversational also the translation of natural language to database or knowledge graph queries attracts much attention (e.g., Reformulations as Meta-Queries [31, 32, 33]). For smart assistants, Tanaka et al. [34] used From the information system’s perspective, the informacrowdsourcing like us to gather implicit and highly am- tion seeker uses a meta-query language when expressing biguous seeker requests for specific non-conversational conversational reformulations: they tell the system to tasks, e.g., “I’m thirsty” as a request to search for a nearby perform specific operations on the previous query. On café or “I’m having trouble getting good reception” as a a syntactic level, the basic reformulation operations in “request” to search for a WiFi spot. As example of spoken traditional search interfaces are adding, changing, and reformulations in a diferent setting, researchers have for removing a term. These correspond to the basic operadecades investigated ways to edit text by voice [35, 36], tions create, update, and delete of data systems [38]. The using commands like “Capitalize the first letter in each fourth basic operation of data systems, read, may also be word in each title” [37]. useful if the previous query is not visible, like in some conversational interfaces. For illustration, Table 1 shows conversational examples for each basic operation. One query reformulation can contain several basic operations. 3.2. Algorithmically Understanding with 80 participants to minimize the interface’s influence Natural Language Reformulations on the participant’s choice of words. Based on insights from the pilot studies, we formulated the tasks as bullet In the past years, impressive advancements have been points with a sentence structure clearly diferent from the achieved in natural language understanding. Still, when reformulations we asked for. Moreover, automatic checka message can be interpreted as diferent meta-queries, ing routines help the participants to stick to the task (e.g., the problem is far from solved. For example, what if the alerts for undesired repetitions or missing terms). The inseeker would have asked “OK, how about vaccinations?” terface resembles a WhatsApp chat to prime participants as their second message in Figure 1 (b)? Is the intent to on chat messages [43]. specify the previous query or to start a new one? Hints After an initial “ready”-interaction to illustrate the task on the true intent might be found in previous messages (cf. top of Figure 2), each participant completed twelve (relation between ‘vaccination’ and the previous query) assignments from one domain as a single search session. or previous results (maybe the seeker read something that To analyze reformulation diversity, we changed the task caused the question). Other conversations may suggest domain and topic between participants: either finding quite diferent interpretations. For example, if asked arguments on banning plastic bags, finding books on (Sciafter “Can you show me articles about its treatments?”, Fi) viruses, finding news on COVID-19, or finding trips one could interpret ‘vaccination’ as a replacement for to San Jose (a very widespread city name). However, ‘treatments.’ To resolve such ambiguities, search systems the search tasks for each participant had the same strucmay ask the seeker for clarification [ 39, 40] or they may ture of abstract operations (e.g., create one term) with try heuristic disambiguation. Such heuristics could, for only keywords being replaced between the domains.5 To example, employ word or entity relationships (e.g., using ensure a variety of reformulations, we formulated the WordNet or knowledge graphs) or query performance instructions to cover all four CRUD operations, to vary predictors like term specificity and result coherence [ 41]. the targets from a single literal to the whole query, to cover conjunctions and disjunctions, and to include some 3.3. Formulating Laconic Queries for special cases like a filter attribute, an unspecified literal, Keyword-based Retrieval or a negation. Participants completed a session in about 12 minutes (observed in the pilot studies and the final The example in Figure 1 (b) shows that conversational study) and we adjusted the payment to cover the minreformulations (like the seeker’s second message) have imum wage of the respective country.6 Unfortunately, to be converted to context-independent queries to sub- we had to stop our study in India and Australia after the mit them to standard retrieval systems. This is similar to ifrst domain (news). Only 22% of the Indian participants “query rewriting,” a process that resolves co-references provided reasonable messages for the tasks (61% in other in conversational question answering (e.g., [6]). In fact, countries) while getting answers from 20 Australian parsimilar methods may be efective for conversational re- ticipants alone exhausted our time constraints. In total, formulations. In the protocol stack of Figure 1 (a), con- we accepted the work of 284 participants. versational reformulations can thus be seen as a service To ensure the dataset’s quality and ease processing, of the conversation layer that builds upon the retrieval we manually checked each message. Of the initially service of the laconic layer. 3408 messages, 2917 are grammatically and semantically meaningful in the respective context. Of these, 2694 (79% 4. Crowdsourcing Reformulations of all) can be interpreted as the respective intended metaquery and form the final dataset of 558 messages for the argument domain, 573 for book, 961 for news, and 602 for trip (cf. Table 2 for other key statistics).
To foster research on conversational query reformulations, we publish a respective crowdsourced dataset4 that accounts for diversity in seeker location (five Englishspeaking countries) and search domain (four diferent ones). The dataset’s goal is to allow for analyzing the diversity and ambiguity in the language of conversational reformulations. However, the dataset also allows to bootstrap the natural language understanding component of conversational systems [42].
Figure 2 shows the interface used in Amazon’s Mechanical Turk marketplace to collect “natural” reformulations.
We iteratively refined the interface in eight pilot studies
systems that allow for conversational query reformulations demands investigating the peculiarities of the
The annotation interface for each domains is presented by the respective <domain>-interface.html file.
6https://medium.com/ai2-blog/crowdsourcing-pricing-ethics-and-bestpractices-8487fd5c9872 respective language. To this end, Section 5.1 provides a across both search domains and countries, indicating that general overview of the messages collected in our dataset, a generic reformulation resolution system might be feahighlighting diferences in language use between search sible. In Section 5.2, we report on our detailed analysis domains and countries. Though the exact patterns occur of the ambiguities in the messages, showcasing both the with diferent frequencies, they, in general, are similar ambiguities and possible ways to avoid them.
To investigate on the word patterns of conversational query reformulations and diferences between search domains and countries, our study design employs the same sequence of twelve abstract tasks for each participant, only exchanging a few keywords to specify the diferent search domains. Table 3 shows the formal tasks and provides general characteristics of the collected messages. The ∑︀-column shows the total number of valid messages per task. Though this number is close to the maximum of 284 (the number of participants) for most tasks, it is relatively low for Tasks 4, 9, and 10, indicating a misunderstanding of the participants. We see such misunderstandings as an artifact of our study setup and filter out the afected messages from our below analyses. 7
We have systematically analyzed the 2694 messages of our dataset. Besides the more general analyses of message types, we also focus on word frequencies and patterns. Apart from a small diference in preposition frequencies for trips (more frequent use of “to” when “about” is used in the other domains), argument search has one diference to the other domains in that a few participants formulated their requests not as a query but asking for the system’s opinion (e.g., “What do you think about banning plastic bags?”).
ally annotated each message as being a command, a question, or a statement. The left part of Table 2 shows the message type usage per country. While participants from Australia and India used more commands, participants from Great Britain used more questions, and participants from the United States used more statements on average. Overall, we found this observation to show the main dif
along with the dataset. ference between countries, with no notable diference in word choice in the relatively small amount of data.
Table 3 shows the message type usage per task. The most frequent are commands (e.g., “Please remove arguments about banning plastic bags.” for Task 12) that often make up at least 60% of all messages per task. The one task where the majority of messages are questions is Task 9, which is the only task that involves a read operation (e.g., “Can you please remind me of my previous commands?” and “What did I search for?”). Though statements (e.g., “I would like to see news that are not about COVID-19.” for Task 12) are relatively rare in general, they are the dominant type for the two tasks that deal with correcting a misunderstanding: Task 6 is to tell the system that it misinterpreted an acronym (but leaving the acronym unspecified, e.g., “I did not mean NI as North Ireland.”), whereas Task 7 is to specify the intended meaning (e.g., “I meant National Insurance.”). Participants switching to statement messages might thus be an indicator that they are pointing out problems. References to current list As a potential signal to identify reformulations, participants sometimes, but unfortunately rarely, refer to the items in the (imagined) result list when reformulating the query (e.g., “Which of these include vaccination?”). Specifically, in the rare cases for the respective tasks (all but Tasks 1, 6, and 7), participants use those (3%), ones (3%), these (1%), or them (0.4%). Somewhat frequently, 16% refer to the list and 2% to results (e.g., “only show me results that include vaccination”). As a diference between domains, 1% of messages in the respective tasks of the trip domain use there (e.g., “I want to have a travel to there by ship.”). More common than references to the current list is the use of the domain-specific item ( argument, book, article, trip), with the special case of the phrases ‘pros and cons’ and ‘for and against’ to specify that both sides should be considered in argument search (e.g., “Can you give me arguments for and against banning plastic bags please”). However, these do not indicate reformulations.
nal for reformulations, many participants explicitly expressed whether the result list should grow or shrink.
Verbal expressions for shrinking the current list (Tasks 2, 5, 12) are remove (9% of messages in these tasks), filter (5%, e.g., “Filter list to just about vaccination.”), exclude (4%), narrow down (3%), reduce (1%), limit (0.7%), filter out (0.5%, e.g., “Please filter out all articles that are not about vaccination.”) and shorten (0.5%). Note that some of these verbs indicate which items to remove, whereas others indicate which items to keep. Overall, 26% of the messages in these tasks contain a verb that explicitly requests to shrink the list. Other signals for shrinking are the use of only (18%) and just (6%), though these percentages may Specializing a query or starting a new one When be inflated as the task descriptions also contain these. asked to specialize the query by adding a new term
Frequently used verbal expressions for growing the (Task 2), the majority of our participants (66%, cf. Tacurrent list (Tasks 3, 4) are add (25%), add back (2%, e.g., ble 3) used a message that one could also interpret as “add back the trips by car.”), expand (3%, e.g., “Expand the starting a new query with that one term (e.g., “Just show list to include books about plants too.”). Other signals for me arguments about CO2 emissions”). We observe the growing are the use of also (23%), as well (4%), too (3%, same ambiguity in other specialization tasks (Task 3, 16% e.g., “can you also add those including treatment too?”), of messages, e.g., “Can you show me arguments that are and as well as (1%). Interestingly, participants used the about renewable resources?”; Task 5, 52%, e.g., “May I verb keep both to shrink the list in Task 2 (2% of messages please see the articles that have NCD, NI, or WHO in the for Task 2, e.g., “Keep articles related to vaccination”) and headline?”) and in tasks that ask to start a new query in a lexically indistinguishable way to partially undo such (Task 11, 48%, e.g., “Find a list of books about evolution.”).8 shrinking in Task 3 (1%, e.g., “Please keep the arguments Still, some participants directly used unambiguous mesabout renewable resources”). sages, either by explicitly referencing the current list (e.g., “Great can you refine that to articles with NCD, NI Summary The observed diferences between countries or WHO in the headline?”) or indicating a new list or and domains are relatively small. A change of the seeker search (45% of the messages for Task 11, e.g., “Find a list from asking questions to expressing statements often of books about evolution,” “New search on evolution,” or indicates specific unusual requests, though diferences “Disregard all previous instructions and now only find me between countries need to be considered. Finally, many books about evolution”). Moreover, 8% of the messages participants directly requested the growing and shrinking for Task 11 are unambiguous due to explicitly expressing of the result list in their reformulations. a replacement, for example, “Show me trips to Santiago instead.”
the ambiguity of natural language. For reformulations, this means that the same message can be interpreted as diferent operations. In our study, we found the below three main ambiguities.
Unclear precedence Though there are precise rules for operator precedence in logics, no such rules exist for 8Taken literally, also several messages for Task 1 and 12 would be ambiguous. However, the alternative interpretations make no sense in the respective contexts. We ignore these strictly lexical ambiguities in our considerations. natural language. Indeed, 72% of the messages for Task 3 do not clearly express whether the new term should be an alternative just to the last term (as asked for) or to the entire query (e.g., “Could you please also include arguments about renewable resources?”). About 11% of the participants’ messages are unambiguous by explicitly stating the relation (e.g., “Show me trips by ship or by car,” where ‘ship’ is the previously added query term). Though not asked to do so, a few of these participants also hinted at a reason for asking for the alternative (e.g., “Show me trips by ship, if ship trips are not available then I would like to select trips by car.”). A few participants (1%) made use of an explicitly stated filter term from the previous query, which then allowed them to refer back to it (e.g., “Filter list for infected animals” and then “Add plants to iflter as alternative”).
Ambiguous negation Surprisingly, several messages submitted to tell the system that it misinterpreted an acronym are lexically indistinguishable from filtering by the acronym. While the majority of messages for Task 6 are unambiguous as expected (61%, e.g., “I’m not asking for North Ireland”), 39% of the messages are ambiguous and could easily be misunderstood (e.g., “Do not include articles about North Ireland”). Indeed, only the fact that the user had just added ‘NI’ as an acronym hints at the intended meaning. query. However, seekers may also want to fetch a query they used some time ago, maybe to continue or refresh a previous search.
More search operators. We considered only the standard logical operators (∨, ∧, ¬) and attribute-specific iflters, but most retrieval systems support more. How would seekers highlight phrases (words to be retrieved in that sequence), initiate boosting (a term or attribute being especially important), or fuzzy / strict term matching? As hypothesized in Section 1, step-wise query formulation might increase the use of diverse search operators. Clarifications. We considered only messages from the seeker, but studying possible system reactions is equally essential to account for implicit feedback (repeating what was understood) or asking clarification questions. Both methods are likely helpful to explain and resolve ambiguities, and could at the same time allow the system to showcase unambiguous formulations in an attempt to teach the seeker how to prevent the ambiguities in the future.9
We can only hypothesize how a seeker’s interactions differed if a search system supported conversational query reformulations. As mentioned above and in Section 1, one possibility is that the reduced cognitive efort due to step-wise and natural language query formulation encourages more complex queries that contain more search 6. Conclusion operators. Extending on these considerations, we expect that some seekers will desire to regularly use the same We have formalized the problem of supporting reformu- query like a feed (e.g., for news, but also for professional lations in conversational search systems. By casting re- activities like scholarly search [45]) and may see query formulations as meta-queries that imply standard CRUD formulation as an act of personalization. Therefore, some operations on the “actual” query, we demonstrate that systems may even aim to support reformulations like “A such functionality could be implemented in a conversa- bit more on soccer.” At the same time, we believe that tion layer on top of standard retrieval architectures. An supporting reformulations will be essential for having a analysis of a new dataset of 2694 crowdsourced human conversation between seeker and search system, as rereformulations across four search domains shows that a formulations implement a straightforward way for the generic reformulation component is feasible when consid- seeker to ground the conversation [46], complementing ering the peculiarities of the respective search domains. clarification questions from the system. At such stage, However, we also find that ambiguities in the reformula- the conversations will be more natural. And the users tions will likely be a major challenge for conversational will be “relieved” from the below laconic layer—just like systems. Ambiguities thus merit further investigations. today’s users of the laconic layer do not need to know any details about the underlying TCP/IP layer.
Other languages. We considered only English mes- This work has been partially funded by the Google Digisages so far but expect at least some of the results to be tal News Initiative as part of the “Conversational News” language-dependent. Further analyses will need to be project. conducted for other languages.
Generalized read operation. We considered only the most simple read-operation: reading the current
upon early work on explaining ambiguities in formal query languages [44].
CoRR abs/2001.06910 (2020). URL: https://arxiv.org/ abs/2001.06910. [46] H. H. Clark, S. E. Brennan, Grounding in communication, in: Perspectives on Socially Shared Cognition, APA, 1991, pp. 127–149.