The rapid growth in number and complexity of conversational agents has highlighted the need for suitable evaluation tools to describe their performance. Current ofline conversational evaluation approaches rely on collections composed of multiturn conversations, each including a sequence of utterances. Such sequences represent a snapshot of reality: a single dialog between the user and a hypothetical system on a specific topic. We argue that this paradigm is not realistic enough: multiple users will ask diverse questions in variable order, even for a conversation on the same topic. In this work1 we propose a dependency-aware utterances sampling strategy to augment data available in conversational collections while maintaining temporal dependencies within conversations. Using the sampled conversations, we show that the current evaluation framework favours specific systems while penalizing others, leading to biased evaluation. We further show how to exploit dependency-aware utterances permutations in our current evaluation framework and increase the power of statistical evaluation tools such as ANOVA.
The conversational search domain has recently drawn increasing attention from the Information
Retrieval (IR) community. A conversational agent is expected to interact seamlessly with the
user through natural language, either written (i.e. text chat-bots) or spoken (i.e. vocal assistants).
Following the development of conversational systems, also their evaluation is receiving a lot of
attention. According to the best practices proposed by TREC CAsT [
By answering the first question, we obtain a sound process to permute utterances of a
conversation, producing new conversations to test conversational systems. We, therefore, use
such conversations to compare models under the current evaluation paradigm, highlighting and
measuring its flaws. Finally, we propose a new strategy to include the permuted conversations
in the evaluation methodology. We do not propose a new evaluation measure – as done for
example in [
In this work, we focus on the evaluation of Multi-turn Task-driven Conversational search systems.
One of the most peculiar aspects related to the multi-turn conversational task is the role played by
the concept of “context” [
Several works [
In our experimental analysis, we consider the Conversational Assistance Track (CAsT) 2019 [
As commonly done [
Concatenation-Based Models A simple approach to enrich utterances with context to address
the multi-turn conversational challenges consists in concatenating them with one (or more)
of the previous ones. We propose three concatenation-based strategies, previously adopted as
baselines in the literature [
Pseudo-Relevance Feedback Based Models We consider two approaches based on pseudorelevance feedback (PRF) that account for the “multi-turn” aspect. RM3-previous (RM3p): it concatenates the current utterance and the RM3 expansion of the previous one (using BM25 as ifrst stage retrieval model); RM3-sequential ( RM3s): it takes the relevance feedback considering the ranked list retrieved for the previous utterance, and uses it to expand the current. The diference between the two models is that, for RM3p, the ranked list depends only on the previous utterance and the one at hand. Conversely, the latter considers the sequence of utterances observed up to the current one.
Language Model-Based Models Among the neural language models, we consider coref-spanBERT
(anCB). This method relies on the Higher-order Coreference Resolution model, as defined in
[
Table 1 reports the nDCG@3 observed for the diferent archetypal conversational retrieval baselines either by considering only the original order of the utterances as defined in CAsT 2019 or considering the average over multiple permutations for each conversation. To grant a fair comparison between diferent conversations, since they can have a diferent number of valid class-based permutations, we sample only 100 permutations for each of them. An interesting insight that can be drawn by Table 1 is that the best performing system is the “First Utterance” (FU). The first utterance of the original conversation is often the most generic: f we concatenate it with other utterances, it boosts their recall, helping them obtain better results. The FU approach obtains the same results even when we permute conversations. Since we forced the first utterance to remain in its position, the order does not influence this algorithm.If we consider the result achieved with permuted conversations, we observe a general decrease in the average performance, due to the increased variance caused by the permutations. If we consider the maximum performance achievable, interestingly, all the methods can outperform the results achieved with the original order, indicating that there are situations in which diferent orders are preferable. The change in performance occurs due to the diferent information flow. The conversational models selected – as the majority of common conversational strategies – exploit the context to solve the anaphoras and rewrite the utterances. Such context derives from previous turns. By changing the previous turns, we also change the context, and thus the information used by the system. This aims at mimicking a real-world scenario, where we do not know if previous utterances provided good context. Furthermore, such context might change depending on the path followed by the user.
Figure 1 plots, for each CAsT 2019 conversation, the distribution over the permutations of the average performance of all systems. The yellow diamond represents the mean performance using the default order of the utterances. It is insightful noticing that the default order rarely gives the best performance: using a diferent order of utterances strongly influences performance. Such a pattern is also observable for each system singularly2. Notice that, with the new permuted conversations, it is always possible to cherry-pick conversations permutations to make any model the best in a pairwise comparison.
2We do not report the figure for each system, to avoid clutter. 0.10 0.05 0.00
We are now interested in assessing the efect of using utterances permutations in the current
evaluation scenario. We therefore compare diferent retrieval models using ANalysis Of VAriance
(ANOVA). If we were to apply ANOVA in the current evaluation setup, we would likely rely on
the following model:
= .. + + +
(MD0)
Where is the mean performance of all utterances for the conversation , using the retrieval
model . .. is the grand mean, is the contribution to the performance of the -th conversation,
while is the efect of the -th system. Finally, is the error. Table 2 reports the summary
statistics for ANOVA when applied to CAsT 2019 conversations, using the Model MD0. We
observe that the efect of the “conversation” factor is significant and large-sized ( 0.14).
2 ≥
This pattern is often observed in many IR scenarios, such as ad-hoc retrieval [
If we include permutations for each conversation, we can use the following ANOVA model: () = .. + + () + + (MD1) Where, compared to Model MD0, () is the nested factor that represents the efect of the -th permutation of the -th conversation. Table 3 reports the summary statistics for ANOVA with model MD1. By looking at Table 3 we can see the first huge advantage of including permutations in our evaluation framework: the Model factor is now significant - although small (0.01 < 2 < 0.06). As a side note, Tukey’s post-hoc analysis shows that anCB is the best model, followed by RM3s which belong to the same tier. Subsequently, we have RM3p and CU, which again are statistically not diferent from each other, but worse than the previous ones. Finally, LP is the only member of the worst-quality tier. We have moved from having all models equal in Table 2 to a four-tiers sorting of the models in Table 3. The Permutation factor is not significant an this highlights that there is not a single best permutation for every system, but rather there is an interaction between the systems and permutations: distinct models behave diferently according to the permutation at hand. Table 3 shows that, if we use the permutations as additional evidence of the quality of a model, we discriminate better between them. Furthermore, we do not know in which order the user will pose their utterances. Including permutations allows us to model better the reality: what we observe in our ofline experiment is likely to generalize more to a real-world scenario. Permutations allow robust statistical inference, without requiring to gather new conversations, utterances and relevance judgements.
In this work, we showed that traditional evaluation is seldom reliable when applied to the conversational search. We proposed a methodology to permute the utterances of the conversations used to evaluate conversational systems, enlarging conversational collections. We showed that it is hard to determine the best system when considering multiple conversation permutations. Consequently, any system can be deemed the best, according to specific permutations of the conversations. Finally, we showed how to use permutations of the evaluation dialogues, obtaining by far more reliable and trustworthy systems comparisons. As future work, we plan to study how to estimate the distribution of systems performance without actually having the permutations and the models at hand.