In recent years, conversational systems have attracted considerable attention from both academic researchers and industrial practitioners. In the field of information retrieval (IR), conversational information seeking (CIS) has been identified as one of the most important research directions. Remarkable efforts have been made from different aspects, which include, but not limited to, conversational search conceptualization [1,2,3], conversational query re-writing [4,5,6], generating and selecting clarifying questions [7,8,9,10] and conversational response generation [11,12,13].
Despite the successes achieved by the aforementioned studies, fundamental research questions remain open. For example, providing high-quality user-specific response is still a challenging problem. Take the case by Aliannejadi et al. [14] as an example, for the search about alternatives to cow's milk, two personas can be: (A) Alice is a vegan who is deeply concerned about the environment; and (B) Bob has been recently diagnosed with diabetes, has a nut allergy, and is lactose intolerant. Given Alice and Bob's personas, their corresponding conversations with the system would evolve and develop in very different ways. Put another way, the responses that are helpful to Alice may not be necessarily useful to Bob, and vice versa. In fact, information needs of this kind are prevalent in daily information searches, which include, but not limited to, job finding, healthcare search and online shopping. Given the information needs expressed as a sequence of search queries (or questions) and different personas, it is of great importance that the CIS system can effectively incorporate the personalized context and provide relevant responses to users. Motivated by this observation, we focus on developing a unified CIS system, which enables to incorporate personalized context during the interactive search process. The main contributions of this work are listed as follows:
β’ By following the CIS paradigm presented in the TREC iKAT track, we propose different methods for tackling the core tasks, namely personal textual knowledge base (PTKB) statement ranking, passage ranking, and response generation. For PTKB statement ranking, we explore how to fuse multiple large language models (LLMs). The experimental results show that our method achieves the best performance in terms of MRR on the set of iKAT organizers' assessments which relies on a larger assessment pool. Moreover, our method also shows superior performance over the GPT-4-based baseline. This highlights that it is not straightforward to solve a component task by merely tailoring a powerful LLM. Whereas a fusion of multiple LLMs can be a promising choice when tackling problems of this kind. β’ For passage ranking, we propose four different strategies for personalized retrieval, which enables us to well investigate the impact of utterance rewriting and the way of incorporating personalized context. Through result analysis and comparison, we found that: Though our proposed method for selecting PTKB statements is relatively reliable, how to incorporate the selected PTKB statements to formulate the input for personalized retrieval matters a lot. A direct concatenation is not suggested according to the inferior performance of our proposed strategies. β’ For response generation, we resort to zero-short
LLM-based answer generation by incorporating personalized context. Our method is able to generate grounded and natural personalized responses, and is comparable to the top-tier LLM-based baseline.
Figure 1 describes our focused framework for CIS that accounts for users' personas. It assumes that there is a personal text knowledge base (PTKB), which consists narrative sentences providing personal information about the users.
A system following this framework consists of the following key modules. (1) Statement ranking: given the context of the conversation and the current user utterance, this module returns a ranked list of PTKB statements based on their relevance, which reflects the user's persona; (2) Passage ranking: given the context of the conversation, the current user utterance, and the PTKB statements, this module is responsible for retrieving a ranked list of passages from the document collection;
(3) Response generation: this module returns the answer text as a response to the user. In particular, the response should be a generative or abstractive summary of the relevant passages. We recognize that the gap exists between our focused framework for CIS and the real-world search scenarios. Since this topic is still in its infancy, we leave it as a future work to explore more complex frameworks.
Given the target paradigm for CIS in section 2, we elaborate on the proposed methods for addressing the key module as below.
The key idea of our method (denoted as SR_FML) for tackling statement ranking is to effectively fuse multiple LLMs through a cascade of four steps. At the first step, we rewrite each conversation turn's utterance. Specifically, the T5-CANARD model [15] fine-tuned with the testing topics of TREC CAsT 2022 [16] is used, and the preceding turns' conversations (3 turns at most) are used as the context. At the second step, given the candidate PTKB statements, we perform binary logistic regression based on the BERT [17] model. The candidate PTKB statements with a true label are kept for later steps, and the statements with a false label are filtered out. At the third step, we perform binary logistic regression again over the remaining PTKB statements based on MonoT5 [18] in the same way as the second step. In addition, we use RankGPT [19] to sort the PTKB statements, and assign the top half statements with a true label, and a false label for the remaining bottom statements. At the fourth step, we mange to unify the ranking information and binary classification results of the previous two steps via a scoring function and an indicator function. The scoring function assigns a weight for each remaining statement in the 2nd step as follows:
where πΌ ππ π ππππΊππ (π ) and πΌ ππ πππππ 5 (π ) represent the rank positions according to the regression scores by MonoT5 and RankGPT, respectively. |π| represents the number of remaining PTKB statements in the second step. The indicator function builds upon π€(π ) and a voting mechanism as follows:
and π€(π ) > 0.65 0 otherwise (2) πππ π΅πΈπ π (π ), πππ πππππ 5 (π ), and πππ π ππππΊππ (π ) respectively represent the binary classification result by each adopted LLM, where an output of 1 denotes a true label, and 0 for a false label.
The final result list of PTKB statements is generated by selecting statements with a positive output via the indicator function and ranking them via the scoring function in a decreasing order.
To cope with passage ranking, we resort to the typical pipeline of retrieve-then-rank. Firstly, we use BM25 with the default setting in Pyserini to retrieve the top 5 passages. Then we design 4 strategies (denoted as PR_S1, PR_S2, PR_S3 and PR_S4, respectively) to re-rank the top 5 passages using multiple specifically selected LLMs in a zero-shot manner.
To formulate the input, PR_S1, PR_S3, and PR_S4 concatenate the rewritten utterance and the top 2 relevant PTKB statements returned by the module of statement ranking. PR_S2 directly uses the rewritten utterance as the input.
During the ranking process, the differences among the four strategies are as follows: (1) PR_S1 and PR_S2 assemble the results by multiple LLMs (i.e., "stabilityai/stablelm-tuned-alpha-7b", "eachadea/vicuna-13b-1.1", "jondurbin/airoboros-7b", "TheBloke/koala-13B-HF") [20,21,22,23,24] in a voting manner. Specifically, given the information need represented by the input, we ask each LLM to compare the candidate passages in a pairwise manner. The passage that is identified to be more relevant than the other gets a vote. Finally, we rank the passages based on the cumulative number of votes in a decreasing order; (3) PR_S3 merely relies on MonoT5 with the default setting in PyGaggle to rank the passages; (4) PR_S4 relies on the idea of RankGPT to rank the passages, where the GPT-3.5 API is used.
For tackling response generation, we aim to generate personalized response. Specifically, for each conversation turn, the top-1 passage and the top-2 PTKB statements representing the personalized context are used as the input. For the base LLM, we resort to T5 [25], which is specifically fine-tuned for the summarization task.
We use the dataset released by TREC iKAT 2023 for evaluating the effectiveness with 25 testing topics. Each topic has 1 βΌ 3 subtree conversations that represent different personas. For each personalized conversation, there is a list of around 10 PTKB statements. Moreover, the passage collection has 116, 838, 987 passages, which is derived from a subset of ClueWeb22-B [26].
In order to make a fair and thorough analysis, we perform a module-specific comparison by selecting the most competitive and representative baseline methods from TREC iKAT 2023's participants. We add a prefix of BS to each baseline method for a better clarity.
For statement ranking, BS_zs_Llama and BS_ft_Llama use zero-shot and fine-tuned Llama-2-7b-chat [27] for rewriting the utterance, respectively. Then they use MiniLM12 [28] to rank PTKB statements based on the rewritten utterance.
For passage ranking, BS_Llama2 initially instructs Llama-2-7b-chat to reformulate the current utterance considering previous conversation turns' context. Then, the revised conversation, along with a specific passage, are provided to the model to assess the passage's relevance.
For response generation, BS_FastChatT5andLlama creates a summarization for each of the top passages retrieved by BM25 using FastChatT5 [29], then it generates the response to current utterance based on the summaries in a retrieval-generate loop. A final response is summarized by BS_DenseMonoT5 using different engines including conventional language models and Llama2 based on top passages.
Besides the above module-specific baseline methods, BS_GPT-4 is compared across three modules, which represents the method using the most powerful LLM (i.e., GPT-4 [30]). For statement ranking, BS_GPT-4 casts it as a binary classification problem. The prompt includes the instruction, context of the conversation, PTKB statements of the user, and current user utterance. The output is a ranked list of relevant statements. For passage ranking, BS_GPT-4 initially generates an answer for each turn. Subsequently, GPT-4 is employed to produce five queries for each answer. These generated queries are used via BM25 to retrieve passages, then the pre-trained MiniLM12 is deployed for ranking the passages. For response generation, GPT-4 is prompted to generate the answer, using the top-10 retrieved passages, the top-3 PTKB statements, the context of the conversation and the user utterance.
All experiments were conducted on a server with two A100 (40GB) GPUs. The CUDA version is 12.2. For fine-tuning T5-CANARD, the configuration is: training epochs: 5, batch size: 4, learning rate: 1π β 5. For SR_FML, bert-base-uncased with default parameter settings is used as the backbone model, which comes from transformers library provided by HuggingFace [31]. We iterate its predictions five times and compute the average relevance scores for each statement. For RankGPT, the configuration is: window size: 4, step size: 1. The MonoT5 with default parameter settings in Pygaggle is used. In PR_S3, the window size of RankGPT is adjusted to 3. In PR_S1 and PR_S2, we set the prompt_max_length of the four zero-shot LLMs to 2048. Additionally, we set the decoding method to beam_search, output_max_length to 512, and temperature to 1.0 by default [32]. For RG_SumT5, t5-base-finetuned-summarize-news is employed with configuration: input max_length: 512, output min_length: 50, output max_length: 150, length_penalty: 2.0, num_beams: 4.
In Table 1, Table 2 and Table 3, we show the overall performance of the baseline approaches, and the proposed methods for statement ranking, passage ranking and response generation, respectively. Within each table, the best result in terms of each metric is indicated in bold, and the secondbest result is underlined.
For statement ranking, we note that there are two sets of assessments which were created by the iKAT organizers and NIST assessors, respectively. The key differences are that: During topic generation, the organizers annotated each turn in terms of their provenance to PTKB statements and included their labels in the released topic files. During the assessment of passage relevance, the NIST assessors were also asked to judge the relevance of PTKB statements to each turn. The assessment pool is smaller than the one done by the organizers. The organizers judged all of the turns, while the NIST assessors only judged the turns that were selected for passage relevance [14]. From Table 1, we can observe that BS_zs_Llama outperforms the other methods in terms of nDCG@3, P@3 and Recall@3. Though BS_ft_Llama relies on the same LLM, its performance is impacted due to the rewritten utterances in a fine-tune setting. On the contrary, BS_GPT-4 relying on the powerful GPT-4 shows inferior performance across two sets of assessments. This indicates that the usage of GPT-4 for statement ranking is not straightforward, further exploration is needed for a better performance. Over the set of iKAT organizers' assessments, our proposed method (i.e., SR_FML) shows competitive performance as BS_zs_Llama, and achieves the best performance in terms of MRR. This indicates the benefit of fusing multiple LLMs, which enables us to leverage on the advantages of different LLMs. In view of the fact that the set of iKAT organizers' assessments bases on a larger assessment pool, it is reasonable to say that the evaluation over this set is more reliable.
For passage ranking, the results in Table 2 show that BS_GPT-4 significantly outperform BS_Llama2 and our pro- posed methods by a large margin. This echoes the findings in prior studies [19,33,34,35] which have shown the leading capability of GPT-4 in the passage ranking task. One probable reason is that the pipeline of generate-retrieve-generate adopted by BS_GPT-4 is more suitable for passage ranking than our adopted pipeline of retrieve-generate. Among our proposed strategies for passage ranking, PR_S2 shows the best performance, and also outperforms BS_Llama2. Compared with BS_Llama2, a possible reason for the inferior performance of the other three strategies is the way of formulating the input. We directly concatenate the utterance and related PTKB statements as the input, while BS_Llama2 rewrites the utterance with the statements using LLM. Another possible reason for our inferior performance is that we focus on the earlier positions and only re-rank the top-5 passages returned by BM25. As a result, this setting would become a bottleneck for us to get relevant passages given the limited retrieval ability of BM25. For response generation, the results are evaluated in terms of groudedness and naturalness. Groudedness measures whether the generated response can be attributed to the passages that it is supposed to be generated from. Naturalness measures the extent to which the response sounds humanlike, such as the general fluency and understandability of the generated response. GPT-4 is used to evaluate both the groundedness and naturalness of the responses in each turn. Finally, the mean of groundedness and naturalness over all turns is reported. From Table 3, we can observe that BS_GPT-4 again outperforms the other methods by a large margin. Our proposed method (i.e., RG_SumT5) outperforms BS_DenseMonoT5 and shows competitive performance as BS_FastChatT5andLlama.
It is noticeable that the evaluation results are likely to be somewhat biased towards BS_GPT-4, since the evaluation is conducted by GPT-4. We leave it as a future work to further test the effectiveness of these methods for response generation through human evaluation results.
A joint look across Table 1, Table 2 and Table 3 reveals that: First, we do not observe a clear correlation between statement ranking and passage ranking, which seems counterintuitive. For instance, though BS_GPT-4 shows inferior performance in statement ranking, it outperforms the other methods by a large margin in passage ranking. This counterintuitiveness may arise from a number of possible reasons, such as the strong zero-shot capability of GPT-4 and the precise understanding of persona information underlying selected PTKB statements. This is also worthy to be investigated as a future work. Second, for both personalized retrieval and response generation in the context of CIS, there is still a large room to improve the performance.
In this study, we focus on CIS that accounts for personalized retrieval and response generation. By following the CIS paradigm presented in the TREC iKAT track, we propose different methods to tackle three core tasks, namely personal textual knowledge base (PTKB) statement ranking, passage ranking and response generation. We have shown that fusing multiple LLMs is a promising way for addressing PTKB statement ranking. Also, our analysis indicates that an effective way of injecting the selected PTKB statements is quite important for personalized retrieval. Since conversational systems arise in a variety of applications, such as recommender systems and question answering, we believe that our work provides insights for developing conversational systems that account for personalized retrieval and response generation.
This research has been supported by JSPS KAKENHI Grant Number 19H04215.