Finding experts, publications, and topics is a daily task not only of every scientist and student but also for journalists and people who search for sources when consuming information. To support this process, we aim to develop a conversational search engine with which it is possible to search for experts interactively and to explore interesting publications and topics where existing tools reach their limits. An important aspect of the search is that the search query is formulated in such a way that it leads to the desired result. However, formulating a query by a user or understanding a query by a system are challenging tasks. For example, when a query is formulated too unspecific, the search results might not entirely cover the information need whereby small further pieces of information can help immensely. Current systems do little to accurately understand the user's search intent and ofer little support during the search process. Thus, we designed an interactive search engine which runs in a chat window, so that the query can be specified over several turns until the desired search results are obtained. The search engine initiates the conversation by asking the user what they want to search for. The user answers in natural language or can choose adequate answers suggested by the system. The conversation continues until the user has fulfilled their search need or wants to start the conversation from the beginning in order to perform a new search.
not well-known in the subject area because the topic of In almost every area of research, it is necessary to find the search query is too broad a topic. As an example, a experts and publications for a topic. Whether to form a user is looking for an expert for Fairness in Information new research group, to invite scientists to events, or to Retrieval, but only enters the query “Information Retrieval” recommend reviewers, the challenge is to find suitable in the search engine. Since experts for Conversational experts. However, it is not only scientists who need to Information Retrieval are also experts for Information ifnd experts, but also people who are no experts such as Retrieval to a certain extent, they are also included in the journalists, e.g., to select suitable interview guests on cur- search results, even though they are irrelevant for the rent news-relevant topics, and laypeople who consume user. information and seek to have sources with experts sup- Popular search engines for scientific papers, such as porting these information. Another important research Google Scholar1 or Semantic Scholar2, search large cattask is to find interesting publications or related work. alogs of publications and also ofer the possibility to For a scientist, it is important to know the current state- browse publications of an author in their profile. Furof-the-art in order to contextualize one’s own work and thermore, statistics such as the number of citations of a to emphasize what is novel and special about one’s own publication or the h-index of an author can also be diswork. played. However, the search options are limited, and the
However, identifying suitable experts or publications user is only assisted to a small extent in fulfilling their are dificult tasks not only for computers but also for search goal, e.g., by displaying related search terms. humans. For example, when a user is looking for experts, To solve the limitations of insuficient attention to the they often enter a topic into the search engine, which user’s search intent and lack of search support, a converthen checks an index to see which people have published sational search engine is indispensable. A conversational on this topic. A problem arises when a user does not search engine assists users in achieving their search inmake their query specific enough which can occur on tent through a dialogue using natural language. Thereby purpose, e.g., when the user makes a navigational search it should be possible with the search not only to find as well as without purpose, e.g., when the user lacks experts, but also to explore interesting publications or knowledge. As a result, the quality of the search results related topics to the search query. The search is to take is not very high and the best results may not be found. place via chat interface and can take several turns.
The conversation is started by the system with an introductory question (such as “Hello, what are you looking 35th GI-Workshop on Foundations of Databases (Grundlagen von Datenbanken), May 22-24, 2024, Herdecke, Germany. $ nillesm@uni-trier.de (M. Nilles) 0000-0002-3449-9319 (M. Nilles) 1https://scholar.google.com © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License 2https://www.semanticscholar.org
Attribution 4.0 International (CC BY 4.0). & Slot Filling component has two tasks. The goal of User Intent Classification is to determine the purpose the user wants to accomplish by using the search engine, e.g., finding experts, publications, or topics. Slot Filling extracts the needs from the user utterance. For a predeifned set of slots, e.g., author name or publication title, it determines the values for (slot, value)-pairs. The Search Module uses the determined information of the User Intent Classification & Slot Filling component to query an index and retrieve the data, e.g., persons or publications.
The Conversational Module generates the natural language answer for the retrieved data, asks clarification questions and suggests follow-up queries. To evaluate the components we also created a dataset consisting of user utterances in the context of bibliographic search.
In the future, we not only want to evaluate the individual components of this conversational information retrieval system for bibliographic data, but also want to work out the advantages and disadvantages of the conversational information retrieval system for bibliographic data, in comparison to already existing systems that do not support the user in their search process via natural language conversations. Which leads us to the research question: How beneficial is a conversational information retrieval system for the search of bibliographic data?
Chat systems such as ChatGPT3 or Microsoft Bing’s new for? You can search for publications and authors. Just enter chat mode4 make conversations between humans and your natural language query.”). After the user entered computers more and more natural, and there are virtutheir natural language query the system infers the user’s ally no limits to what computers and humans can talk search intent and uses textual methods to retrieve search about. With Bing’s new chat mode, Microsoft wants to results from an index. The search results as well as a nat- support the user in web search, so that they can submit ural language answer are displayed to the user. Figure 1 his query in natural language. When systems support shows the GUI with a sample conversation and search a user in searching for information through natural lanresults. guage interaction, they are called conversational informa
The system response consists of multiple parts. It ex- tion retrieval systems or conversational search systems.
plains how the system understands the user query so that McTear [
Besides the presentation of the system architecture Zhang et al. [
4https://www.bing.com/search?q=Bing+AI&showconv=1&FORM= hpcodx of task-oriented dialogue systems in their survey. User Query
There are few systems that represent an entire con- Who published ab2o0u2t3C?onversational IR in
versational information retrieval system. These systems
include the above mentioned ChatGPT and Bing’s chat User Intent C&lassification
mode, as well as XiaoIce (Zhou et al. [
While there exist further works that deal with
individsuyasltceomm, pe.ogn.,eunstesroifnatecnotncvlaesrssaifictaiotinoanl ainnfdorsmloat tfililoinngr(eLtoriue-val NaturalPSLeerasaonrngcsuh:+[aprge1,espu2R,lt.es.]sponse
van and Magnini [
Current search engines for bibliographic metadata, the session as the input, and outputs an user intent label such as dblp5 [10], ResearchGate6, Semantic Scholar for the whole utterance and a slot label for each token or Google Scholar allow only keyword-based searches. of the utterance. For the slot labeling the BIO-tagging Kreutz et al. [11] presented SchenQL, a query language format is used. A token can be classified as the Beginning for bibliographic metadata that allows users to make of a slot value, as Inside of a slot value, or as Outside of their queries more easily and precisely. With the conver- a slot value. sational search system the user should be able to search With the determined user intent and (slot, value)-pairs, for bibliographic metadata with the support of the system the Search Module searches an index which contains the without prior knowledge. bibliographic data. The slots are mapped to diferent
Another important part in the development a conver- fields in which a query processor searches for the corresational information retrieval system is to understand sponding values. the search behaviour of users. Kuhlthau [12] created a The Conversation History Module stores the user seven step model of users information search process. In queries as well as the system responses to update the another study, Kuhlthau [13] observed students in their dialogue state after each turn. The dialogue state is consearch process while they were in high school and again sidered by the User Intent Classification & Slot Filling four years later when they were in college to examine component to predict the user intent and slot values of changes in their search behavior. the next user utterance. The user utterances can also be used to analyze the user’s search tasks and to improve the system.
The preliminary architecture of the conversational search 4. Preliminary Implementation system for bibliographic data is shown in Figure 2. It consists of four main components: User Intent Classification In this section we give an overview of the progress we & Slot Filling, Search Module, Conversational Module, made so far and describe the implemented components of and Conversation History Module. the system architecture (Section 3) in more detail. To eval
The task of User Intent Classification & Slot Filling is uate the components, we built a dataset of user utterances to determine the search intent of the user and to extract (Section 4.1). The dataset is used to train and evaluate the information given by the user to fulfill their intent. A the User Intent Classification & Slot Filling component search intent can be that the user is searching for persons, (Section 4.2). Initially, the conversational information publications or topics. The User Intent Classification & retrieval system supports 29 predefined user intents (17 for searching publications and 12 for searching persons) and 33 predefined slots (16 from publications and 17 from persons).
With the obtained information from the User Intent Classification & Slot Filling component an index containing the bibliographic data (Section 4.3) is queried by the Search Module (Section 4.4). The Conversational Module (Section 4.5) uses the results from the User Intent Classification & Slot Filling component and the results from the Search Module to generate a natural language answer to the user.
BERT model adds a special token at the beginning of the user utterance. The output for the special token is the user intent label of the user utterance. For each token of the user utterance the slot label is returned. A token can either be classified as the Beginning of a slot value, as Inside of a slot value, or as Outside of a slot value. The BERT model achieves an accuracy of 0.955 for predicting the user intent. The f1-score for predicting slot values is 0.968.
main of search for bibliographic data with labelled slots As a database, we use the dblp and extended it with data for the tokens of the user utterances, we created a dataset from standard data sources such as Semantic Scholar. containing 620 user utterances. The dblp is a bibliography containing information of
For the predefined set of user intents and slots, we ran- computer science journals and proceedings. We enriched domly combined a user intent with a slot. We created one the dblp data with information retrieved from the Se(user intent, slot)-pair for each user intent and one (user mantic Scholar API8. The Semantic Scholar API provides intent, slot)-pair for each slot. We formulated for each information about publications and persons which is not pair a user utterance, a user of a conversational informa- available in the dblp, e.g., the abstract, a summary or tion retrieval system might ask a conversational search academic categories of the publication. This information system for bibliographic data. E.g., we formulated for will be displayed to the user. We use the dblp xml file 9 to the pair (publication, publication.topic) the ut- build a Lucene10 index containing the dblp data and the terance "Who has published a paper about Conversational retrieved information from Semantic Scholar. The index Information Retrieval?. is queried by the Search Module.
Then we used ChatGPT to rephrase these utterances and checked the returned reformulations. We made sure 4.4. Search Module that the reformulated utterances still have the same meaning and contain the same slots as the given utterances. The Search Module queries a Lucene index containing We added nine reformulations and the original utterance the bibliographic data. The results of the User Intent to the dataset. In total the dataset contains for each ut- Classification & Slot Filling determine how the query terance ten diferent formulations. will be build. The query consists of multiple subqueries.
The slot values of each formulation are filled with ran- For each slot a subquery will be added to the query if the dom values from the bibliographic database, e.g., the User Intent Classification & Slot Filling module detected publication.topic slot is filled by keywords con- values for this slot. The subquery of each slot will then tained in the database. To allow the system to recognize search in one or multiple fields of the index. The results questions that are not related to any of the predefined will be displayed to the user. intents, we added questions from the Quora-QuestionPairs dataset7. We used the created dataset to train and 4.5. Conversational Module evaluate the User Intent Classification & Slot Filling component.
guage answer of the system. It uses templates to generate the answer. The determined user intent and slots by the 4.2. User Intent Classification & Slot User Intent Classification & Slot Filling module and the
Filling retrieved results by the Search Module are inserted in The User Intent Classification & Slot Filling module has the templates. Currently, we use templates to summarize two tasks. The first task is to determine the user intent the results of the User Intent Classification & Slot Fillfrom the user utterances and the second task is to ex- ing components, and to formulate the natural language tract the slot values from the user utterances. We used a response to the user request. joint intent classification and slot filling model based on BERT [14, 15].
graphic metadata by means of a conversational retrieval system. We presented an architecture of such a system and so far, we implemented three components of the system: User Intent Classification & Slot Filling, Search Module and Conversational Module. Next, we will implement the remaining component, the Conversation History Module. The Conversation History Module tracks the dialogue state of the conversation between the system and the user. It incorporates previous turns of the conversation into the User Intent Classification & Slot Filling component. With the current dialogue state from the conversation history and a new user utterance the system is able to detect changes in the user intent and the (slot, value)-pairs. To train the system for multi-turn conversations, we will create a multi-turn conversation dataset for bibliographic search through studying how a conversation between a user and conversational search system might evolve during a search session. Each turn of the dataset’s conversations will be annotated with the current user intent and (slot, value)-pairs.
Besides the evaluation of the individual components of our proposed conversational information retrieval system Conversational Bibliographic Search, we will evaluate the system as an entirety in user studies. To examine the usefulness of a conversational retrieval system for bibliographic data and to answer the research question, we plan to evaluate our system by comparing it to existing bibliographic search engines in terms of efectiveness, eficiency, and user satisfaction. We will also compare our system with approaches that use Large Language Models (LLMs). Because of the high computation and storage cost we are currently not planning to train a single LLM for bibliographic search. Another disadvantage of using a single LLM instead of our approach could be the problem of LLM hallucination.
Instead, we want to explore the extent to which LLMs can be used in each component. For example, we could use LLMs to generate the natural language response. LLMs could also be trained to ask clarification questions and to suggest follow-up queries to the user.
In user studies, we want to identify how a conversational information retrieval system can help users to fulfill their information need. We also want to gain insights into the information tasks of diferent user groups, e.g., students vs. more advanced researchers. After identifying user tasks, we examine for which task users benefit the most from an information retrieval system supporting conversational search and infer from these results for which user group a conversational information retrieval system would be most useful. 18653/V1/2020.COLING-MAIN.42. [9] W. Lajewska, K. Balog, Towards filling the gap in conversational search: From passage retrieval to conversational response generation, in: I. Frommholz, F. Hopfgartner, M. Lee, M. Oakes, M. Lalmas, M. Zhang, R. L. T. Santos (Eds.), Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023, ACM, 2023, pp. 5326–5330. URL: https: //doi.org/10.1145/3583780.3615132. doi:10.1145/ 3583780.3615132. [10] M. Ley, DBLP - some lessons learned, Proc.
VLDB Endow. 2 (2009) 1493–1500. URL: http://www. vldb.org/pvldb/vol2/vldb09-98.pdf. doi:10.14778/ 1687553.1687577. [11] C. K. Kreutz, M. Wolz, J. Knack, B. Weyers, R. Schenkel, Schenql: in-depth analysis of a query language for bibliographic metadata, Int. J. Digit. Libr. 23 (2022) 113–132. URL: https://doi.org/10.1007/s00799-021-00317-8. doi:10. 1007/s00799-021-00317-8. [12] C. C. Kuhlthau, Inside the search process: Information seeking from the user’s perspective, J. Am.
Soc. Inf. Sci. 42 (1991) 361–371. [13] C. C. Kuhlthau, Perceptions of the information search process in libraries: a study of changes from high school through college, Inf. Process. Manag. 24 (1988) 419–427. URL: https://doi. org/10.1016/0306-4573(88)90045-3. doi:10.1016/ 0306-4573(88)90045-3. [14] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Association for Computational Linguistics, 2019, pp. 4171–4186. URL: https:// doi.org/10.18653/v1/n19-1423. doi:10.18653/V1/ N19-1423. [15] Q. Chen, Z. Zhuo, W. Wang, BERT for joint intent classification and slot filling, CoRR abs/1902.10909 (2019). URL: http://arxiv.org/abs/ 1902.10909. arXiv:1902.10909.