While current Large Language Models (LLMs) achieve excellent performance in information seeking tasks, their conversational abilities when participants need to collaborate to jointly achieve a communicative goals (e.g., booking a restaurant, fixing an appointment, etc.) are still far from those exhibited by humans. In the paper, we specifically focus on proactivity [2,3,4,5], a collaborative behaviour investigated in the context of dialogue pragmatics [6]. Proactivity refers to the act of taking initiative, anticipating potential problems, and actively providing information and contributing to the conversation with ideas, suggestions or solutions. Proactivity involves participants actively participating in the dialogue, addressing concerns, and promoting a collaborative environment. The following is an example of a dialogue in which proactive utterances are underlined. After completing my Bachelor's degree in Rome, I would like to move back towards home, to Florence. a:
We currently do not have any offers that fit your needs in the Florence area, however, there are job opportunities in Rome.
In order to model proactivity, we follow a similar approach as Shaikh et al., which focuses on grounding acts, a class of collaborative behaviors investigated in dialogue pragmatics. The main idea is that grounding acts can be: (i) identified and annotated by a Large Language Model; (ii) modeled through appropriate fine tuning of the model itself. In addition, our work is related to recent approaches that use large language models as annotators [9], [10], [11]. In the long-term, our research goal is to instruct LLMs to be as proactive as humans are.
The goal of the paper is to show the feasibility of automatic detection of proactive utterances in taskoriented dialogues. We propose a task, Last Utterance Proactivity Prediction, where a portion of a dialogue (i.e., a dialogue snippet) is given to a model, which has to predict whether the last utterance of the snippet is proactive or not proactive. Using dialogue snippets, instead of full dialogues, brings several benefits: (i) dialogue snippets are much more manageable than full dialogues, allowing us to reduce the complexity of understanding and annotation; (ii) several phenomena in dialogue, including proactivity, depend on a short context, which allows a model to learn from snippets, rather than full dialogues; and (iii) dialogue snippets make it easier to experiment on balanced datasets, overcoming the skew distribution of proactivity in whole dialogues (it has been estimated that about 85% of the utterance in a task-oriented dialogue is not proactive).
We started with D-PRO2 , a corpus of manually annotated task-oriented dialogues, which includes 151 dialogues from different sources, amounting to 2,855 turns and over 6,000 utterances, and carried out the following steps:
• we transformed the whole-dialogue annotation task to a one-utterance annotation task: given a short dialogue context, the model needs to establish whether the final utterance is either 'proactive' or 'not_proactive'; • in order to shorten the provided dialogue context, we collected 4 conversational turns' worth of excerpts (snippets) from each dialogue. We believe 4 turns to be a convenient context for proactivity annotation since statistics in the D-PRO Corpus on turn-adjacency between proactive utterances and the turn that triggers them revealed that an average of 77.7% proactive utterances are a direct response to the previous turn's utterances; • to restore balance among labels we choose the same number of snippets that ended without proactivity as the snippets that ended with proactivity.
A relevant consequence of the reduction of the provided dialogue context, is a significant reduction of the input prompt length for a LLM, and therefore a reduction of computational need.
This section reports the main features of the setting we used to experiment the last utterance proactivity task introduced in Section 2.
Dataset for the experiments. The dataset for the experiments has been derived from D-PRO, a corpus equipped with manually curated proactivity-oriented annotations. D-PRO comprises 151 dialogues from 5 task-oriented dialogue sub-corpora, namely, Italian Whatsapp Corpus ( [12]), the Italian Nespole! Corpus ( [13,14]), Jilda ( [7,15]), the Italian Ubuntu Chat Corpus ( [16]), and Multiwoz 2.2 ( [17]). Most of the dialogues are in Italian, with the only exception of the Multiwoz 2.2 dialogues and some dialogues from the Italian Whatsapp Corpus due to code mixing and code switching employed by the speakers. D-PRO proactivity annotations are performed at the utterance level. The composition of our experimental dataset is as follows: from D-PRO we gathered as many 4-turn proactive dialogue snippets as there were proactive utterances, so that each snippet ended with a different proactive utterance. Then, we extracted as many non-proactive snippets as there were nonproactive utterances, so that each snippet ended with a different non-proactive utterance. Finally, we selected an equal amount of non proactive dialogue snippets as proactive ones at random to restore balance between the two types of snippets.
Data splitting. From each of the 5 corpora in D-PRO we randomly selected 30 dialogue snippets as a train set (to be used as few-shot examples), 50 snippets as a validation set (to be used for parameter optimization), and 100 snippets as a test set 3 .
For the choice of the best model for proactivity prediction we carried out a number of experiments, reported in Section 4.1. The model selected is Openai's GPT-4o-2024-08-06, used with temperature = 0. Prompt optimisation. A prompt engineering phase took pace, so that various prompt proposals were tested in the same setting (the same train snippets used as few-shot examples, same validation snippets used to evaluate the model). Figure 1 shows the final prompt used in our experiments. The prompt consists of two main parts: (i) the system prompt, which contains the general task instructions given to the model; (ii) the messages prompt, that is further dividend into alternating user messages and assistant messages: this is the part of the prompt where the model receives few-shot examples (user messages) with answers (assistant message). The final user/assistant pair contains the target dialogue which is being evaluated by the model at current time.
Baseline. A random chance baseline (accuracy = 0.5, see Table 8) is created by eliminating any system and message prompt except for "Output exclusively either "proactive" or "not_proactive. " and providing the target dialogue snippet.
We first present several optimization trials performed on a single corpus (MultiWoz) in order to select the best LLM (4.1), to assess the impact of the number (4.2) and of the order (4.3) of few-shot example snippets. Secondly, we run test on the DEV set of each of the five corpora in order to select the best few-shot snippets order for each corpus (4.4).
Once the best prompt had been established, we tested the APIs of various models to pick the best cost/performance trade-off, reported in Table 1. GPT-4o-2024-08-06 was selected as the best performing model (Accuracy: 0.74, F1: 0.68) with lower fares than GPT-4o-2024-05-13 and better scores than both GPT-4o-mini and GPT-4o-mini-2024-07-18.
This experiment assesses the stability of the model while changing the order of the few-shot examples.
As literature points out [18,19,20], LLMs suffer of a position bias when handling a longer context: we found that this is the case also in our experiments and that there is up to 10 points of a difference in accuracy when testing with different orders of the same set of examples.
In this experiment we select the best order of the few-shot snippets for each of the five dialogue datasets (i.e., Whatsapp, Nespole, Ubuntu, Jilda and MultiWoz) for the last utterance proactivity prediction task. We tested 5 different orders of the dialogue snippets randomly shuffling the same set of snippets selected in section 4.2. We used the following configuration for the experiments: 12 random few-shot snippets; 50 validation (DEV) snippets; 5 random shuffles of the few-shot snippets; average of the performances of the 5 shuffles for each corpus. The selection of the optimal order is given by the highest average accuracy and F1.
In this section we present the results of the last utterance proactivity prediction task in two different configurations: using few-shots from individual corpora (5.1), and mixing few-shots from all corpora (5.3), introducing transfer learning as well. Lastly, we describe an experiment in Section 5.4 that attempts to evaluate the model's stability in corrupted context scenarios.
We tested the model by using the prompt and few-shot configuration that gave the best results on the development set for each corpus individually. As Table 5 shows, while IAA with the ground truth labels is still not optimal, and scores pretty low on both Whatsapp and Ubuntu (fair agreement 4 ), we reach a 4. On average, Nespole achieved the best accuracy (0.86), followed by MultiWoz (0.77), Jilda (0.75), Whatsapp and Ubuntu (both 0.64). For all dateset the results are largely above the baseline (i.e., 0.50 accuracy, equivalent to chance: see also Table 8, Baselines -Full Context column.), showing that the model has correctly learned our definition of proactivity. The fact that Nespole has obtained the best results is somehow surprising, given that this corpus is quite complex: utterances are longer than in the other corpora, and so are dependencies between a proactive utterance and its own trigger utterance. Longer utterances, on the other hand, mean that the model is given a slightly richer context on which to base its judgments, which may help with the annotation process. As far as lowest scores are concerned, the poor results for Whatsapp and especially Ubuntu were expected, since these are less structured (both syntactically and grammatically), more chaotic, and multi-party dialogue corpora, where proactivity is much more difficult to be unanimously detected also by humans and where the human-human IAA scores the lowest (0.63 and 0.41 respectively).
Secondly, we run some in-context learning [22] experiments to check whether few-shot examples from different corpora could improve the performance on one individual target corpus. The idea is drawn from works on multi-task learning [23], where more than one task is learned simultaneously by the model, and transfer learning, where improvement is obtained in a new task through the transfer of knowledge from a related task that has already been learned [24,25]. Our intuition is that example variety on very similar tasks may be the key to improvement on single target task. Following this line, we combine dialogue snippets from all the five corpora as few-shot examples, so that we have 5 sets of 12 snippets each. Given the position bias hold by the model, we decided to keep the intra-corpus examples order the same as the one used in 5.1, and to randomly shuffle the inter-corpus order 5 times to select the optimal few-shot prompt (same methodology in 4.4 and 5.1). The outcomes of the tests with the optimal prompt are reported Table 6, that is directly comparable to Table 5. We found out that the only corpus in our experiment that suffered the mixed few-shot prompting is Jilda, with a significant drop in performance, while every other corpus has very similar or slightly higher scores. Ablation tests on the Jilda corpus led to an accuracy of 0.71 while removing the most chaotic corpora (Ubuntu and Whatsapp), proving still that the individual corpus few-shot approach works best for this one corpus.
We finally tested on the cumulative test set of all corpora, with mixed few-shot examples. We experimented with two configurations of few-shot snippets: i. 60 snippets in optimal order as in Table 6; ii. 15 snippets in total, with 3 random snippets per corpus. Outcomes in Table 7 show best and average results over 5 runs for both the 60 and the 15 few-shot examples setting.
We investigated the performance of the LLM in a corrupted context situation. According to definition, the two key characteristics of proactivity are not being solicited and being beneficial to the dialogue goals, hence proactivity can only be defined in terms of the previous context. When we corrupt the context before the snippet's final utterance, we may be compromising the data required for the proactivity annotation task. We implemented two corrupted context situations: triggering utterance removed, where the text of the utterance that triggers the final utterance in the dialogue snippet is removed from the snippet, and triggering utterance masked, where the text of the trigger utterance is masked by a placeholder. In both circumstances, the presence of a corrupted utterance is indicated by the utterance number, whereas only the content is erased or masked. Since we need to test the effect of the context corruption, the model still learns the original full context task from the few-shot examples. We anticipate that the LLM's performance will suffer since a critical component of the dialogue (the triggering utterance) has been compromised. Also, we expect the number of false positives to increase significantly. This is due to the possibility that by eliminating the trigger utterance, we will also eliminate the element that renders the final utterance either proactive or non-proactive. Specifically, we are deleting the element of the context that allows us to determine whether the content of the last utterance is novel (i.e., proactive) or unrequested by the trigger. Since the request is missing from the corrupted setting, the triggered response seems to be proactive rather than solicited, resulting in an increase in "proactive" labels. Results, presented in Table 8, confirm our intuition, showing that the model accuracy moves from 0.80 of the full context to 0.66 and 0.64 when the triggering utterance is removed and masked, respectively. The majority of the performance drop is attributable to an increase in false positives, from 2 in the full context to 8 in the corrupted context, as well as a drop in true negatives, which supports our hypothesis. On the other hand, our experiments show that even with insufficiently task-specific few-shot examples, the model can perform significantly better than the random chance baselines (see also 3) through a solid instruction prompt.
We introduced a new task, namely, last utterance proactivity prediction, aiming at assessing the capacity of Large Language Models to detect and annotate proactive behaviours in task-oriented dialogues.
The task allows us to shorten the context from a whole dialogue to a dialogue snippet, simplify the annotation process, and balance the dataset for positive and negative labels. We showed that a few-shot approach with GPT-4o achieves encouraging performance on a test set composed of dialogue snippets collected from five different corpora, and that in particular for the Nespole corpus the agreement between the model labels and the human-annotated gold labels is nearly equivalent to the agreement between humans. As for future work, there are several ongoing activities. First, we are still investigating techniques to further improve the performance on the task, especially in testing on the combined dialogues from all corpora. Then, we plan to use the GPT-4o model to automatically annotate a large amount (i.e., about 100K) of dialogue snippets, in order to create a training corpus, which, in turn, will be used to instruct an open source model (e.g., Llama 3 8B) to detect proactivity.