Despite the impressive capabilities of recent Large Language Models (LLMs) to generate human-like text, their ability to produce contextually appropriate content for specific communicative situations is still a matter of debate. This issue is particularly crucial when LLMs are employed as assistants to help solve tasks or achieve goals within a given conversational domain. In such scenarios, the assistant is expected to access specific knowledge (e.g., a database of restaurants, a calendar of appointments) that is not directly accessible to the user and must be consistently utilised to accomplish the task. In this paper, we conduct experiments to evaluate the trustworthiness of automatic assistants in task-oriented dialogues. Our findings indicate that state-of-the-art open-source LLMs still face significant challenges in maintaining logical consistency with a knowledge base of facts, highlighting the need for further advancements in this area.
Conversational assistants [
While in traditional approaches [
Our aim is to shed new light on the trustworthiness of an LLM playing the role of an assistant in a task-oriented conversational domain while interacting with a user. We aim to answer the following research questions: (i) How can we operationally define the consistency between a task-oriented dialogue and the domain database behind the dialogue? (ii) How can we quantify the degree of trustworthiness of an assistant-LLM? (iii) Can we collect empirical evidence on a suficiently large amount of taskoriented dialogues?
To address these research questions, we set up an
experimental framework allowing large-scale analysis,
where task-oriented dialogues are first automatically
generated by two instances of a state-of-the-art LLM,
LLama3 8B [
preliminary phase, referred to as the Human-Llama Interaction phase (cfr. Section 3), we test the capabilities of an open-source LLM (i.e. LLama-3) to generate adequate task-oriented dialogues through interactive conversations with humans.
In the second phase, referred to as the Llama-Llama Interaction phase (cfr. Section 4), we automate both the generation and evaluation of task-oriented dialogues, creating a Llama-Llama generated MultiWOZ dialogue corpus, The Dining Llamas of Oz1. Following in this section, the description of the MultiWOZ dataset and the metrics used to check and quantify the reliability of the generated dialogs in both phases.
MultiWOZ is a widely known task-oriented dialogue dataset collected via the Wizard of Oz approach. The dataset comprises over 10,000 dialogues between a customer and the Cambridge InfoTown assistant, designed to help customers navigate Cambridge’s amenities. The conversations span over seven diferent domain concepts, including train ticket reservations, tourist attraction searches, and restaurant reservations. For our experiments, we selected data related to the restaurant domain (version 2.3 [9]).
The MultiWOZ dialogues were collected with a system that provides information to the user relying on a specific database, known as the Knowledge Base (KB), describing properties of the Cambridge domain. Each domain concept has its own KB; for our experiments, we consider only the restaurant KB. The restaurant KB holds information about 110 diferent instances (i.e., restaurants), where each instance comprises a series of properties (e.g., Name, Food, Area) and corresponding values (e.g., The Old Cambridge, british, north).
All system turns in the dialogues are expected to consistently rely on the information contained in the KB to provide accurate information to the user.
Knowledge Base, we analysed each system-generated conversational turn referring to any piece of information provided in the KB. Each turn was assessed based on two separate binary metrics: • KB-Alignment: Assesses whether the system turn is consistent with the KB, meaning that does not contradict any information provided in the KB. • KB-Grounding: Assesses whether the system turn refrains from hallucinating and introducing information not present in the KB, ensuring all mentioned details are grounded in the existing KB.
https://github.com/tLabruna/The-Dining-Llamas-of-Oz at:
Figure 1 would be as follows: T4 (KB-Alignment = 0, KB2.1. The MultiWOZ 2.3 Dataset Grounding = 1), T6 (KB-Alignment = 0, KB-Grounding Since the primary focus of this work is about task- = 0). In addition to this, we used two evaluation metrics oriented dialogues, we used the MultiWOZ (Multi- to assess the overall quality of each turn and provide a Domain Wizard-Of-Oz) dataset [8], one of the most global evaluation of the whole corpus: prominent datasets in this area. MultiWOZ has been extensively employed to develop and test models for natural language understanding, dialogue management, and natural language generation. • Correct Turns: Indicates the percentage of turns that have both KB-Alignment and KB
Grounding annotated as 1. • Correct Dialogues: Indicates the percentage of dialogues that have all turns with both KBAlignment and KB-Grounding annotated as 1.
proach of the MultiWOZ dataset through the humanLlama interactive generation of novel dialogues. Although this phase required substantial human efort, it was crucial for obtaining an initial high-quality set of dialogues.
We aimed to generate dialogues where a human interacts with a system played by Llama-3 8B in two languages: English and Italian. The model was prompted to play the role of the Cambridge InfoTown system. The system’s goal was to guide the user towards reserving a restaurant in Cambridge. For each dialogue, we utilised 10 restaurant instances taken from the MultiWOZ KB. We selected 6 distinct sets of instances, which had the following characteristics: 1. All with the same Food; 2. All with diferent Food (or as diferent as possible); 3. All with the same Price; 4. All with diferent Price (or as diferent as possible); 5. All with the same Area; 6. All with diferent Area (or as diferent as possible).
tiate the sets since they are the informable slots within the Restaurant concept.
The human users were instructed to follow a scenario that involved reserving a restaurant, providing a realistic context for the dialogues. Five distinct instructions were employed for the interactive generation of a human-LLM dialogue, each paired with the 6 sets of KB instances, resulting in a total of 30 dialogue scenarios. The process was repeated in both English and Italian, leading to the creation of 30 dialogues in each language, for a total of 60 dialogues.
The manual evaluations were conducted by three annotators who assessed the dialogues based on the binary metrics KB-Alignment and KB-Grounding. Each of the 60 dialogues was annotated by at least two diferent annotators to ensure reliability. The inter-annotator agreement between human evaluators was measured using Cohen’s in both metrics and languages that indicates substantial agreement on Landis and Koch’s agreement scale [10]. We instructed GPT-4o2 to perform the same evaluations as the human annotators. This consisted in feeding the model with a given KB/dialogue pair, asking it to output two lists of turn assessments: one for the KB-Grounding and another for the KB-Alignment. Then we computed the agreement between GPT-4o’s evaluations and the human evaluations. The precise prompt used to instruct GPT-4o can be found in Appendix B. Although the agreement with GPT-4o (see Table 1) was slightly lower than the substantial agreement observed between human annotators, it was still classified as moderate on Landis and Koch’s agreement scale [10]. Due to these results we assumed GPT-4o to be a valuable automatic judge and deployed it the same way for the LLama-LLama evaluation phase (cfr. Section 4).
After recognising the ability of Llama-3 to generate dialogues and the evaluation skills of GPT-4o (cfr. Section 3.2), we conducted further experiments by generating 1,311 dialogues using Llama-3 8B and following the MultiWOZ dataset. For each dialogue of the original dataset, we utilised the instructions provided to the human user in the Wizard-of-Oz setting to guide a Llama acting as the user, interacting with a Llama acting as the system.
During the dialogue generation phase, we randomly selected 70 instances from the entire Knowledge Base for each simulated dialogue, ensuring that each dialogue was staged in a varied KB scenario. This approach, a.k.a LLama-Llama phase, allowed us to create a large set of automatically generated dialogues, each based on a diferent subset of the KB. We call this generated dataset "The Dining Llamas of Oz," which comprises 1,049 training instances, with 131 instances each for the validation and test sets.
Kappa ( ) to provide a measure of the inter-rater reliabil- 2GPT-4o was used via the Microsoft Azure APIs. The API version ity (IRR) level. As per Table 1, we obtained an average was 2024-02-01. The cost for the API interactions was about $400.
Table 2 presents statistics for the dataset, including approach significantly improved the agreement: we obthe average number of turns per dialogue, the average tained a of 0.68 for KB-Alignment and 0.49 for KBlength in number of tokens for user and system turns, Grounding (moderate/substantial agreement). Conseand the Standardized Type-Token Ratio (STTR) [11] for quently, we decided to use this technique for automated user and system turns. The STTR is calculated by merg- evaluation. ing all turns, segmenting them into chunks (we used a Using this approach, we assessed 262 dialogues (from segmentation size of 1000), and computing the average the evaluation and test splits) using GPT-4o. This proTTR for all chunks. vided a broader understanding of the KB consistency of Llama-generated dialogues across a larger dataset. The Table 2 KB consistency evaluation is summarised in Table 3. The Statistics of the Llama-Llama dialogues dataset. turns were filtered by removing those that were judged to have no reference to the KB. In addition to evaluating Statistic Value the metrics for all 262 dialogues, we further analysed the Number of Dialogues 1311 dataset by dividing it based on two criteria: the success Average Dialogue Length 6.21 of the dialogues and the dialogue length. For the success Average User Turns Length 25.69 criterion, we distinguished between dialogues with a user Average System Turns Length 124.52 instruction that, in the original MultiWOZ dataset, led User Turns STTR 0.29 to a successful restaurant booking (successful dialogues) System Turns STTR 0.41 and those that did not lead to any restaurant reservation (unsuccessful dialogues). For the dialogue length criterion, we distinguished between dialogues that had three 4.1. Turn-by-Turn Evaluation or fewer turns (a maximum of three user utterances and three system utterances) and those that had four or more To assess the quality of the Dining Llamas of Oz dataset, turns. we employed GPT-4o, as in our previous experiments.
Using the same approach as in Section 3.2, we obtained a KB-Alignment score of 49.73% and a KB-Grounding score 5. Discussion of 38.59% for the entire dataset. To verify the annotation quality of these new dialogues, we manually annotated 30 dialogues from the evaluation split and compared these annotations with GPT-4o’s evaluations on the same dialogues. This initial comparison resulted in a not ideal of 0.15 for KB-Alignment and 0.06 for KB-Grounding (slight agreement). To enhance these performance metrics and establish a reliable evaluation pipeline, we revised our approach: instead of passing the entire dialogue to GPT-4o, we evaluated one turn at a time. The detailed methodology was as follows: Our investigation into the performance of state-of-theart Large Language Models (LLMs) like Llama-3 in taskoriented dialogue systems reveals several critical insights about their current limitations. The central finding is that while these models exhibit advanced capabilities in generating text, their quality in managing task-oriented dialogues remains unsatisfactory.
Initially, we compared human evaluations with GPT4o’s evaluations to assess its efectiveness in evaluating dialogue quality. This comparison was instrumental in determining that GPT-4o could be useful for dialogue 1. Provide GPT-4o with a user utterance and the evaluation, but it highlighted that the model’s perforcorresponding system response, and prompt it to mance degrades significantly when scaled from a smaller determine if the system’s response references the to a larger Knowledge Base. The annotation agreement KB. dropped notably as the number of KB instances increased 2. If GPT-4o indicates a reference to the KB: from 10 to 70, indicating that GPT-4o struggles with a) Prompt GPT-4o with the same user-system larger, more complex datasets.
turn and the KB to determine if the sys- To address this, we shifted our approach to a turn-bytem’s turn shows KB-Alignment. turn evaluation method. After extensive experimentation b) Prompt GPT-4o with the same user-system and prompt engineering, this method yielded improved turn and the KB to determine if the sys- results in terms of annotation agreement. However, this tem’s turn shows KB-Grounding. approach proved to be highly resource-intensive, pushing up costs significantly due to increased OpenAI API usage.
The full prompt is available at Appendix B. This Our automated evaluations on 262 dialogues provided method allows for a more precise scoring of each turn, some revealing observations, as shown in Table 3. Nothough it increases OpenAI API usage and associated tably, only around 40% of system turns demonstrated costs. We discovered that this turn-by-turn evaluation KB-Alignment and KB-Grounding. When considering both metrics together for Correct Turns and Correct Dia- to acknowledge certain limitations that may afect the logues, the results were even more concerning: just 26% generalizability and scalability of our findings. The turnof turns and less than 9% of dialogues met the criteria for by-turn evaluation approach, while efective in enhancboth metrics. These numbers underscore the inadequacy ing evaluation accuracy, proved to be computationally exof current systems, indicating that a system producing pensive. The quality of GPT-4o’s evaluations was highly such a low percentage of correct dialogues is not practical dependent on efective prompt engineering. Crafting the for real-world applications. right prompts to ensure accurate evaluation results was
Further analysis showed that dialogues with successful challenging and time-consuming. Additionally, employbookings performed better than those with failed book- ing a diverse set of models for generating and evaluating ings. Specifically, dialogues with successful bookings had dialogues could provide more comprehensive findings. 28.59% of correct turns and 11.29% of correct dialogues, Using multiple models might help in understanding the compared to dialogues with failed bookings, which had strengths and limitations of diferent approaches, poten9 percentage points fewer correct turns and only 0.5% tially ofering a more robust analysis of dialogue quality correct dialogues. This discrepancy likely arises because and consistency. This could also help in mitigating the when no suitable restaurants are available, the Llama limitations inherent in any single model or evaluation model tends to hallucinate, providing restaurants not approach. present in the KB. While these restaurants may exist in Cambridge, they are absent from the provided dataset, highlighting the model’s failure to adhere to the instruc- 7. Conclusions and Future Work tions given in the prompt.
We also explored the impact of dialogue length on In this study, we explored the capabilities of state-ofperformance. Shorter dialogues achieved nearly 30% cor- the-art LLMs in generating task-oriented dialogues, forect turns and 11.23% correct dialogues, while longer cusing on maintaining consistency with a provided KB dialogues showed a significant drop: 7 percentage points and avoiding hallucinations. Our experiments demonfewer correct turns and only 3.17% correct dialogues. strated that Llama-3, despite its advancements, struggles This suggests that as the conversation progresses, the to perform reliably in these settings. The model showed likelihood of errors increases, possibly due to the model’s significant limitations, especially in dialogues that led dificulty in managing and integrating information from to failed outcomes (where the desired restaurant was previous turns. not in the KB) and longer interactions. As a side contri
Overall, our findings highlight that current state-of- bution, we release The Dining Llamas of Oz, a corpus the-art open-source LLMs, such as Llama-3, are still un- of 1,311 dialogues generated through user-Llama and able to efectively serve as task-oriented dialogue systems system-Llama interactions, to aid future research. Our while maintaining consistency with a provided KB. This ifndings highlight the need for further development to underscores the need for further advancements in LLM improve LLM reliability and accuracy in task-oriented capabilities and evaluation methodologies before such dialogue applications. systems can be reliably used in practical applications.
C. Carlson, R. Carmichael, B. Chan, C. Chang, F. Chantzis, D. Chen, S. Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Chung, D. Cummings, J. Currier, Y. Dai, C. Decareaux, T. Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecofet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. P.
Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. GontijoLopes, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S. S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, J. Huizinga, S. Jain, S. Jain, J. Jang, A. Jiang, R. Jiang, H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, Łukasz Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, L. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, J. H. Kirchner, J. Kiros, M. Knight, D. Kokotajlo, Łukasz Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Leung, D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, S. M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, J. Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. Mossing, T. Mu, M. Murati, O. Murk, D. Mély, A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, L. Ouyang, C. O’Keefe, J. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Parascandolo, J. Parish, E. Parparita, A. Passos, M. Pavlov, A. Peng, A. Perelman, F. de Avila Belbute Peres, M. Petrov, H. P. de Oliveira Pinto, Michael, Pokorny, M. Pokrass, V. H. Pong, T. Powell, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. Saltarelli, T. Sanders, S. Santurkar, G. Sastry, H. Schmidt, D. Schnurr, J. Schulman, D. Selsam, K. Sheppard, T. Sherbakov, J. Shieh, S. Shoker, P. Shyam, S. Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, I. Sohl, B. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. Tezak, M. B. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek, J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. Wainwright, J. J.
Wang, A. Wang, B. Wang, J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng, M. Wiethof, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, W. Zaremba, R. Zellers, C. Zhang, M. Zhang, S. Zhao, T. Zheng, J. Zhuang, W. Zhuk, B. Zoph, Gpt-4 technical re- città di Cambridge. Usa un tono amichevole e port, 2024. URL: https://arxiv.org/abs/2303.08774. onversazionale, fornendo risposte arXiv:2303.08774. informative e utili. Tutte le informazioni [8] P. Budzianowski, T.-H. Wen, B.-H. Tseng, che fornisci devono basarsi strettamente I. Casanueva, S. Ultes, O. Ramadan, M. Gašić, sulla Knowledge Base che ti è stata data. MultiWOZ - a large-scale multi-domain wizard-of- Assicurati che le tue risposte siano accurate, Oz dataset for task-oriented dialogue modelling, pertinenti, e mirate ai bisogni dell’utente. in: Proceedings of the 2018 Conference on Sii breve." Empirical Methods in Natural Language Processing, Association for Computational Linguistics, The following prompt has been used to instruct a Llama Brussels, Belgium, 2018, pp. 5016–5026. URL: to play the role of a user looking for a restaurant in https://www.aclweb.org/anthology/D18-1547. Cambridge, in English: [9] dTo.iH:1a0n.,1X8.65L3iu/,vR1./TDa1k8a-n1a5b4u7,.Y. Lian, C. Huang, "You are a turist in the city of Cambridge D. Wan, W. Peng, M. Huang, Multiwoz 2.3: A multi- and you are looking for a restaurant to dine domain task-oriented dialogue dataset enhanced in. Strictly follow the instructions given to with annotation corrections and co-reference an- you on the criteria by which looking for the notation, in: Natural Language Processing and restaurant. You don’t need to follow all the Chinese Computing: 10th CCF International Con- instructions at once, instead follow them as ference, NLPCC 2021, Qingdao, China, October 13– the conversation continues. Be very brief, 17, 2021, Proceedings, Part II 10, Springer, 2021, pp. and go straight to the point. At the end, 206–218. thank the system and say goodbye. When the [10] J. R. Landis, G. G. Koch, The measurement of ob- conversation is over, after the farewell, server agreement for categorical data, biometrics return \"END\" (in caps lock)." (1977). The following prompt has been used to instruct a Llama [11] B. Richards, Type/token ratios: What do they really to play the role of a user looking for a restaurant in tell us?, Journal of child language 14 (1987) 201–209. Cambridge, in Italian:
"Sei un turista nella città di Cambridge e stai cercando un ristorante dove cenare.
Basati strettamente sulle istruzioni che ti vengono fornite riguardo i criteri in base ai quali cercare il ristorante. Non seguire tutte le istruzioni subito, invece seguile passo passo durante la conversazione. Sii molto breve e vai subito al punto." "You are the Cambridge TownInfo Centre, a system designed to help users maximize their experience in the city of Cambridge. Use a B. GPT Prompts friendly and conversational tone while providing helpful and informative responses. The following system prompt has been used has genAll the information you provide must eral instruction for telling GPT to behave like a dialogue strictly rely on the Knowledge Base that you evaluator: have been provided with. Ensure that your answers are accurate, relevant, and tailored "You are a dialogue evaluator. Given a to the user’s needs. When you find the dialogue you have to return a list of symbols restaurant to reserve, give a random separated by commas, where each symbol is an reservation number to the user. Be brief." evaluation of each turn in the dialogue. Only system turns must be considered." The following prompt has been used to instruct a Llama The following prompt has been used to instruct GPT to play the role of a Cambridge InfoTown system, in to determine if a system turn talks about information Italian: contained in a KB: "Sei l’assistente Cambridge InfoCittà, un "Given the following user and system turns, sistema progettato per aiutare gli utenti a return 1 if the system turn contains trarre il meglio dalla loro esperienza nella information that requires verification from an external source to ensure its accuracy, 0 otherwise." The following prompt has been used to instruct GPT to determine if a system turn constitute a KB-Error: "Given the following user turn, system turn, and Knowledge Base (KB), return 0 if the system contradicts the KB (e.g. says that a restaurant is at north, but it’s actually at south), 1 otherwise."
"Given the following user turn, system turn, and Knowledge Base, return 1 if the system doesn’t mention properties outside of the Knowledge Base, 0 otherwise (e.g. says that the restaurant serves british and indian, but only indian is present in the KB)."