Task-Oriented Dialog Systems (TODS) assist users in completing a task within a conversation [1], for instance, in the context of customer information and bookings (train/restaurant). In an applied setting with real users, it is important that those systems provide correct answers, tasks can be quickly solved, and lead ideally to high user satisfaction. To ensure this, TODS often provide a high level of control over its dialog management and answer behavior for system developers. Existing solutions normally either manually implement a dialog manager to control the complete interaction, or train it on large amounts of dialog interactions [2,3,4,5].
In contrast, Large Language Models (LLMs) are very good at open-domain dialog and provide fluent and convincing messages in different styles. However, those answers might be misleading and even false (hallucination) [6,7,8]. In task-oriented dialog, the model could possibly 'break out' of the given dialog task.
Using LLMs for task-oriented dialog is still in its infancy. Madotto et al. [9] used LLMs for the whole pipeline of Natural Language Understanding, Dialog State Tracking, Dialog Policy and Natural Language Generation. Hudeček and Dusek [10] expand on this idea by evaluating the abilities of LLMs to generate complete taskoriented multi-turn dialogs. They used LLMs for NLU and DST also but, unlike our work, they used a static prompt. Other approaches to LLMs for task-oriented dialog are presented by Cao [11], Hu et al. [12], Wei et al. [13], Li et al. [14].
To address those limitations and concerns, we propose Dynamic Prompting, a technique to combine a traditional task-oriented dialog system pipeline with the benefits of LLMs. Showcased and tested in the context of restaurant booking, we present the advantages and limitations of our approach.
For our experiments, we use the Multi-domain Wizard-of-Oz 2.2 (MultiWOZ) dataset [15,16]. The dataset contains conversations between users and task-oriented chatbots that were collected using the Wizard-of-Oz [17] method. For our setup, we only use conversations about the restaurant domain. In the conversations, users ask chatbots about restaurants in Cambridge, England. The systems can recommend various restaurants based on attributes such as the type of food, location or price range. Also, the system can perform bookings. Table 1 shows an example dialog from the dataset. The restaurant domain encompasses two intents (find, book) and ten slots. For each user utterance, the dataset contains annotations for intent, slots in the utterance and dialog state (detected slots over the whole dialog).
Task-Oriented Dialog Systems (TODS) consist, traditionally, of the following main components [1]: Natural Language Understanding (NLU), Dialog State Tracking (DST), Dialog Manager (DM) and Response Generation (RG). The NLU performs intent detection and slot filling on the current user utterance, whereas DST keeps track of the slots over the whole dialog. The DM selects the next action of the bot based on the results of NLU and DST. Lastly, RG constructs the response to the user.
In the following, we introduce dynamic prompting, a TODS architecture, extended by the capabilities of an LLM. Figure 1 shows the architecture. We chose to use a trained model for the NLU component to handle intent recognition and entity extraction, as Hudeček and Dusek [10] highlighted the limited performance of LLMs in these tasks. For NLU, we use the RASA NLU component, powered by the DIET classifier [18], while for DST, we use a simple hashmap that stores the most recent NLU results. We trained the NLU component on user utterances only from the restaurant domain of the train split of the dataset, focusing on the find and book intents. During training, we also provided use case-specific entities, including categorical (pricerange, area, bookday, bookpeople), as well as non-categorical (food, name, booktime, address, phone, postcode, and reference). We replace/extend the Dialog Manager and Response Generation with a Prompt Generation and an LLM. The prompt generation uses input from the DST and NLU and uses a series of rules, similar to a dialog manager. Also, it fetches data from a database and generates a prompt. It uses prompting templates that consist of three parts: 1) A general task description, 2) content from the database, NLU states in JSON format, and 3) the previous conversation in a theater script style. Table 2 shows an example prompt of our system in the restaurant domain with the detected intent 'book restaurant'. The prompt is then sent to a LLM to generate a corresponding user reply. In our experiment, we use GPT-3.5-Turbo (ChatGPT) by accessing the model via API calls, as it has demonstrated leading performance in the results presented by Hudeček and Dusek [10].
Appendix A shows the prompting templates and the rules that we developed for our prompt generator. Our prompt construction approach involved multiple iterations of 'trial and error' process on the on training set, evaluating their effectiveness based on the system's task completion and relevance to the conversation. Initially, we introduced single instructions in the prompts. However, scenarios such as having no available restaurants, multiple options, or booking a restaurant required more specific instructions. This led us to implement dynamic prompts with tailored rules for each situation.
Assist the user in booking a restaurant. Always assume the restaurant is available to confirm a successful booking. Provide a reference number when the restaurant name, bookday, bookpeople, and booktime are given. Prompt if these details are missing. Omit information about fictional bookings.
Dialog State: {"food": "mediterranean", "pricerange": "expensive", bookday": "wednesday, bookpeople": "7", booktime": "15:15"} Conversation History User: Can you find me an expensive place serves panasian food? Bot: i haven't found any expensive panasian restaurants. should i try another price range? User: Are there restaurants that serve Mediterranean food? Bot: I have a couple. Both are in the centre. la mimosa, shiraz restaurant User: I am interested in La Mimosa. Please book a table for 7 at 15:15 on Wednesday. Thanks for your help!
Example prompt of Dynamic Prompting, which is sent to ChatGPT.
In our experiment, the chatbot generates a response using our dynamic prompting system for each dialog turn. We evaluated its performance on the test split of the dataset's restaurant domain. To evaluate the responses on different levels, we label them by two human annotators, given the following criteria. The annotation guidelines can be found in the supplementary materials.
• Task Success Rate describes the ratio of successful dialogs to the total number of dialogs. Following the definition of Wen et al. [19] and Nekvinda and Dušek [20], we mark a dialog as successful if 1) the system's recommendation aligns with the user's requests (such as price range, location, and cuisine) and 2) the system adequately addressed subsequent requests by the user, such as providing the telephone number or confirming a successful booking. • Prompt instruction performance, a binary scale to assess whether responses aligned with the prompt instructions. • Information Extraction Performance, a binary scale, if the system is able to fetch the relevant address from the JSON information. • Response slot accuracy, the ratio of correctly predicted slot values and the number of slot values in the response. It measures if our system is able to return all desired slots to the user. We compute ratios across all annotated turns from these metrics. • Sensibleness describes if the utterance makes sense given the context [21,22].
• Specificity describes if the utterance is specific regarding the context [21,22]. LMs are used to generate unspecific answers such as "this is great", which are sensible but not desired. • Interestingness describes if the utterance captures someone's attention, arouses curiosity or exhibits traits such as unexpectedness, wit, or insightfulness [22]. Interestingness contributes to a compelling and engaging user experience.
Table 3 shows the task success rate of our system compared to other TODS on the MultiWOZ 2.2 dataset. Although the other systems use the whole dataset and, thus, are not perfectly comparable to ours, it still shows that Dynamic Prompting has a similar performance compared to SOTA systems. This is remarkable, particularly as we use a relatively simple NLU component, which by itself might produce errors. However, if we do not use the NLU system of our pipeline but instead use the entity annotations from the dataset, we get a 'perfect' NLU without any errors. In this case, our Dynamic Prompting achieves a task Success Rate of 0.94 -which highlights the efficiency of the LLM solution.
Table 5 shows further performance metrics. The dialog success rate is supported by the high sensibility and specificity scores, which indicate that the system answers on point and does not deviate from the dialog's goal. However, the response slot accuracy is only 80% and needs to be improved -but this is not the focus of this work.
Task Success Yang et al. [23] 0.83 Lee [3] 0.80 Su et al. [24] 0.85 Dynamic Prompting 0.81 perfect NLU + Dynamic Prompting 0.94
Comparison of Task Success Rates on MultiWOZ 2.2 data, with an inter-annotator agreement of 1 for Dynamic Prompting.
perfectly (Information Extraction Performance=0.98). Although the system does not always follow all instructions from the prompt (Prompt Instruction Performance=0.82), the task success is still quite high, so we assume that only minor errors cause the relatively low Prompt Instruction Performance.
In the following, we analyze the conversations and, particularly, the generated responses of our Dynamic Prompting in more detail.
In one situation the user asked to send the information via email, which the designers of the original dataset did not anticipate. In those situations, traditional dialog systems then can only answer with "I did not understand". Our approach instead was able to produce a sensible response, although it has never been trained for this case (see Table 4).
Similar to our findings in Section 3.1.4, the responses of our system are not only longer but also more engaging compared to the ground truth. For example, in one situation, our system produced an answer such as "You're welcome! If you have any more questions or need further assistance, feel free to ask. Have a great day too!" while the crowd worker wrote only "Thank you. Goodbye". Overall, we counted 'polite' phrases in the responses and found out that dynamic prompting uses them more often than the ground truth, such as "enjoy your meal" (15.5 more often), "have a great day" (2.2), "you're welcome" (4.8), "certainly!" (61.0), "great!" (20.0). Table 9 in the appendix shows more detailed examples.
The database entries are formulated in a different format. Names are often lowercase, and the crowd workers did not correct this issue when they wrote the system responses. Also, postcodes are stored in the format "cb17aa" in the database, although the correct format would be "CB1 7AA" in the Cambridge area. Our approach consistently fixes these errors out of the box.
Dynamic Prompting produces responses that are, on average, 2.41 times longer and more diverse than the responses of the crowd workers in the WOZ dataset, with lexical diversity measured by an MTLD score [25] of 80.41 compared to 72.26 for the WOZ dataset. We assume that the crowd workers were interested in providing fast and minimalistic answers. However, while diverse replies might be considered as positive as they make the interaction more enjoyable, it needs to be determined if long answers are preferable over short answers containing all necessary information. Table 7 in the appendix shows examples of this behavior.
We detected a user dialog in which the NLU failed to detect the correct area. The user asked for a restaurant in the center of the city, but due to the NLU error, the prompt generator wrote two restaurants in the prompt, one in the center and one in the south. The LLM was able to detect that the user wanted a restaurant in the center and presented only the correct restaurant in its response (see Table 8 in the Appendix).
Our system produced two types of problems. First, it was not able to display the total count of available restaurants correctly. Although the total count of available restaurants is encoded in the JSON prompt, the LLM opted to count the number of elements within the JSON options array instead.
The second common problem was the generation of fake booking reference numbers. At the end of the dialog, users often ask for a booking reference number. We do not include booking reference numbers in the prompt. Therefore, the system hallucinated booking reference numbers. We evaluated including reference numbers in the JSON format in the database section of the prompt on ten samples, and we eliminated all occurrences of fake reference number generation in our small sample set. Still, we did not include this in the whole dataset, so our results serve as an indicator only that this approach could solve the problem of hypothetical booking reference number generation.
Prompt: Assist the user in finding a restaurant. A single restaurant matches the user's criteria. Provide requested details of the restaurant. If no specific request is made, offer general information like food type, area, and price range. Respond in a coherent, non-numbered, and fluent text format based only on the conversation history, state, and database information. State: restaurant {"area": "centre", "food": "italian", "name": "stazione restaurant and coffee bar", "bookday": "saturday", "bookpeople": "1", "booktime": "12:45"} Database: restaurant {Count:1, Options: {"address": "Market Hill City Centre", "area": "centre", "food": "italian", "id": "19240", "name": "stazione restaurant and coffee bar", "phone": "01223352607", "postcode": "cb23nj", "pricerange": "expensive", "type": "restaurant"}} User:
Terrific! You are great, please send me all this information to my e-mail, thanks.
I apologize, but as a language model AI, I don't have the capability to send emails. However, you can easily save or take a screenshot of this conversation for future reference. Remember to have a great time at stazione restaurant and coffee bar on Saturday at 12:45! Truth:
You're welcome, was there anything else you need?
Showcasing the ability of the system to handle unusual requests.
The table shows the scores and the interannotator agreement (IA, Cohen Kappa) of the quantitative analysis.
We In addition, our system shows various ways to correct errors, such as NLU errors, user requests not anticipated by the designer of DS, and errors in the format of the database entries. Moreover, the generated system answers are more diverse (Section 3.1.4) and more polite (Section 3.1.2) than the human-generated responses in the dataset. We would like to examine these qualitative results in future research in a more quantitative way.
Overall, we find that the widespread problem of hallucinations in LLMs is not an issue in our system as long as we present the correct information to the LLM. As soon as the user asks the system for information that is not present in the prompt, such as the booking reference numbers, the LLM starts to hallucinate.
Although we assess the system's performance solely on the restaurant domain, the dynamic prompting method can be extended to other domains in the Multi-WOZ 2.2 dataset, such as hotel, taxi, and train. Expanding to new domains will require updating the prompt generation module to accommodate new intents and state values, ensuring smooth integration with these additional domains. I am looking at a place to eat that serves venetian food in the centre.
I apologize, but it seems that there are no restaurants in our database that serve Venetian food in the centre at the moment. However, I can suggest a few alternatives for you. You might consider trying some other types of cuisine such as Italian, seafood, or Mediterranean, which are commonly found in the centre. Alternatively, you could try searching for Venetian restaurants in nearby areas or expanding your search to include restaurants that are not strictly in the centre.
Unfortunately, I don't see any restaurants that serve Venetian in the centre of town. Would you like to try a different area or type of cuisine?
Showcasing the ability of the system to generate diverse responses for a similar scenario.
Prompt: Assist the user in finding a restaurant. Numerous restaurants match the user's criteria. Ask for missing details of food type, area, or price range to narrow down the search before providing the restaurant options. Respond in a coherent, non-numbered, and fluent text format based only on the conversation history, state, and database information. State: restaurant {"food": "modern european", "pricerange": "moderate"} Database: restaurant {Count:5, Options: {"address": "152 -154 Hills Road", "area": "south", "food": "modern european", "id": "14731", "name": "restaurant alimentum", "phone": "01223413000", "postcode": "cb28pb", "pricerange": "moderate", "signature": "slowroast sirloin of beef red onion celeriac and garlic", "type": "restaurant"}, {"address": "83 Regent Street", "area": "centre", "food": "modern european", "id": "3697", "name": "de luca cucina and bar", "phone": "01223356666", "postcode": "cb21aw", "pricerange": "moderate", "signature": "roasted barbary duck breast served with sweet potato wedges and mange tout with a red wine sauce", "type": "restaurant"} User: Doesn't really matter. Is there one in the centre? TOD:
Yes, there is a modern European restaurant called "De Luca Cucina and Bar" located in the centre at 83 Regent Street. They offer a variety of dishes including their signature dish of roasted Barbary duck breast served with sweet potato wedges and mange tout with a red wine sauce. The price range is moderate. Would you like more information about this restaurant? Truth:
Yes, there are 2: De Luca Cucina and Bar & Riverside Brasserie. Would you like further information? Thank you. Goodbye.
Showcasing the ability of system to generate polite and engaging responses.
This work has been supported by the Federal Joint Committee of Germany (Gemeinsamer Bundesausschuss) as part of the project smartNTX (01NVF21116).
Table 6 shows some prompting templates that have been developed for our system.
Tables 7 -9 show different examples conversations.