English. The task of Dialogue Act (DA) tagging, a crucial component in many conversational agents, is often addressed assuming a single DA per speaker turn in the conversation. However, speakers' turns are often multifunctional, that is they can contain more than one DA (i.e. “I'm Alex. Have we met before?” contains a 'statement', followed by a 'question'). This work focuses on multifunctional DA tagging in Italian. First, we present iLISTEN2ISO, a novel resource with multifunctional DA annotation in Italian, created by annotating the iLISTEN corpus with the ISO standard. We provide an analysis of the corpus showing the importance of multifunctionality for DA tagging. Additionally, we train DA taggers for Italian on iLISTEN (achieving State of the Art results) and iLISTEN2ISO. Our findings indicate the importance of using a multifunctional approach for DA tagging.
Dialogue Acts (DAs), a linguistically motivated
model of speakers’ intentions in a conversation,
play a crucial role for several conversational AI
tasks. DAs have been successfully used as part
of conversational agents components, for example
for Spoken Language Understanding
One of the bottlenecks for current research on
DAs is the lack of publicly available resources
with DA annotation. While this is true also for
English, it is even more important for languages with
fewer resources, such as Italian. For Italian, the
only publicly available resource with DA
annotation is currently the iLISTEN corpus
While useful, this resource relies on an
annotation scheme which assumes only one single DA
per conversational turn (see Figure 1). However,
ISO 24617-2
In this work, we investigate the task of multifunctional DA tagging in Italian. The contributions of this paper are: (1) we create iLISTEN2ISO, to the best of our knowledge the first publicly available resource with DA annotation in Italian which uses a multifunctional approach and is ISO-standard compliant; (2) we present an analysis of iLISTEN2ISO showing the importance of multifunctional DA annotation; (3) we propose baseline DA tagging models for Italian trained on iLISTEN (achieving, to the best of our knowledge, SOTA results) and iLISTEN2ISO.1 2
Dialogue act corpora Most publicly available
resources with DA annotation are hardly
compatible, given that each resource is typically tagged
with its own different scheme tailored for a given
domain
Dialogue Act tagging DA tagging is the task of
assigning a DA tag to a given utterance in a
dialogue. The definition of utterance depends on the
schema used: in some schemes (Dinarelli et al.,
2009), the utterance corresponds to a turn, while
in others
The iLISTEN corpus
In iLISTEN the turn DA annotation is not multifunctional, i.e. each turn is assigned one single DA. However, not tackling the turn DAs with a multifunctional approach could result in loss of information, with the DA tag capturing only the most dominant function of a turn. In Figure 1, for example, tagging the entire turn with one DA would prevent the system from understanding that two different questions are asked.
In order to create the iLISTEN2ISO annotation,
each turn from iLISTEN was annotated with a
multifunctional approach following the ISO
standard. This process involved first segmenting turns
into functional units (FUs), defined as minimal
stretches of communicative behaviour that have a
communicative function
Because of the lack of resources, the segmentation and mapping process of iLISTEN was conducted by one single annotator, under the supervision of a second annotator with previous training in ISO standard annotation. In order to ensure a reliable annotation process, after the creation of the guidelines, the second annotator repeatedly assessed a sample (100 utterances) of the annotated data. This sample was built through a stratified random sample, where for each DA tag, 20% of examples of that class was randomly sampled. This evaluation and reassessment was performed twice. In the first round, performed after the first annotation of the data, some issues regarding the usage of some DAs arose and were discussed; in the second examination, performed after the second phase of annotation of the data, no problem was found. 4
The annotation layer of iLISTEN2ISO does not change only the legacy iLISTEN schema, but also the internal structure of turns due to the segmentation process. On average in iLISTEN2ISO we have 1.61 FUs per turn, which become 1.81 on system side and 1.5 on user side if we consider them separately (this difference is justified by the fact that the system turns are on average longer than user turns). In Figure 2, we report for each legacy DA (user side), the number of segments per turn on average. Furthermore, inside the bars we list the 3 most common sequences of ISO DAs to which each legacy DA is mapped. In Table 1, we compare the number of DA tags between iLISTEN and iLISTEN2ISO. We notice that iLISTEN2ISO has a larger number of DAs in total, compared to iLISTEN. Additionally, while in iLISTEN the number of DAs in common (2) is much lower compared to either user or system DAs, in iLISTEN2ISO the common DAs between system and user are larger than the independent ones, with the advantage of potential better generalization across the two. Looking at the distribution of the ISO DAs regarding user turns, it can be noticed that the four most common DAs are: inform 24.5%, quesiLISTEN iLISTEN2ISO tion 21.3%, answer 15.3% and auto-positive 7%. Moreover, the DA distribution has a tail composed of 19 DAs with a frequency below the 5%. However, this is not a drawback of the scheme since it gives us a fine-grained representation of the actions performed by the user. Additionally, iLISTEN2ISO can be used in conjunction with any other corpus annotated with ISO standard thus, giving the possibility of augmenting the samples for a specific low-represented class. 5
In this section, we describe the two baseline
models for Dialogue Act (DA) classification used in
our experiments. The first model is a Support
Vector Machine (SVM)
Our second model is a Convolutional Neural
Network (CNN), following
In this section, we report the results of Dialogue Act (DA) tagging experiments, using our proposed baselines on both legacy (using iLISTEN) and multifunctional ISO standard DA schemes (using iLISTEN2ISO).
Experimental setup For comparison with
previous work, we follow the competition rules and
report results considering only user DAs, using
official splits. Additionally, we do not assume
gold DAs for the context for testing (which might
not be available at inference time), rather we use
predicted ones. In order to do this we train
a separate model for tagging system DAs used
only during inference. The performances of
system models are: Micro F1 96.1% and Macro F1
96.6% on iLISTEN; Micro F1 97.5% and Macro
F1 96.3% on iLISTEN2ISO. For iLISTEN, the
obtained classification results are compared with
Unitor, the winner of the EVALITA competition
Model Unitor SVM CNN SVM CNN we group the least frequent DA tags to the label “Other”. The final DA scheme for iLISTEN2ISO consists of 7 DAs. In iLISTEN the number of examples in training and testing is 1097 and 479 respectively; in iLISTEN2ISO we have 1609 and 777 respectively.
Results As shown in Table 2, our proposed models yield comparable results on both nonmultifunctional (iLISTEN) and multifunctional (iLISTEN2ISO) DA tagging. On iLISTEN, our models even overcome previous SOTA performances (Unitor) on both Micro and Macro F1. We observe that while in terms of Micro F1 our models achieve very similar results on both corpora, in terms of Macro F1 they perform better on multifunctional DA tagging.
Error analysis To better understand the performance of our models on iLISTEN and iLISTEN2ISO, we look at the confusion matrices depicted in Figure 3 and Tables 3 and 4 reporting the performances computed for each DA. (a) iLISTEN (b) iLISTEN2ISO Considering the CNN performance, looking at confusion matrices in Figure 3, we notice that on iLISTEN the worst class is reject where 48.7% of examples are predicted as statement. This is probably due to the similar structure of reject utterances to statement ones, while the discriminant is the semantic content that model fails to detect. This problem can be seen also in Table 3, where the reject DA is predicted with the worst performances among other tags. An example of error is given by the following interaction: the system says “Mangiare ad orari fissi e’ un modo per evitare di saltare i pasti e di trascurare sostanze che spesso non vengono compensate nei pasti successivi.” and the user responds “purtroppo spesso il lavoro limita la possibilita` di fare una dieta sana e regolare.”. This user’s turn is tagged with reject but it is predicted by the model as statement. As it can be seen, the structure of the user’s turn is similar to a statement because the user expresses her or his opinion, in this case regarding the difficulty to follow an healthy diet.
Another interesting mismatch in iLISTEN regards info-request, 11.6% of which are predicted as statement. This is interesting because the class info-request is usually composed of questions, however analyzing heuristically the examples we notice that some of them contains other tags, such as answers or statements, which are hidden in the legacy annotation. In this regard, another potential source of error is the lack of punctuation as it can be seen in the utterance “e` necessario fare sport per mantenersi in forma”. This utterance can be interpreted as a statement, but if a question mark is added at the end of the utterance it can be interpreted as a question. This also highlights the importance of punctuation or prosodic features in order to detect the right DA.
Another problem, that can be identified looking at the iLISTEN confusion matrix in Figure 3, is that the kind-attitude-smalltalk DA is confused with many different others DAs. This is due to lack of segmentation since analysing the ISO DAs distribution of the turns tagged with this tag, it emerged there is not a predominant DA. In fact, the four most common ISO DAs are: inform 21.3%, question 20.9%, thanking 13.5% and autopositive 10.8%.
Regarding the iLISTEN2ISO confusion matrix, it can be seen that request is the most confused class. Indeed, 48.8% of examples are predicted as question, 16.3% as other and only 20.9% are predicted correctly. The reason behind this performance is that the model fails to distinguish a request from a question since both of them are in a question style.
Another frequently mispredicted DA in iLISTEN2ISO is answer, often confused with inform. This is due to the fact that the model has difficulties in representing and then distinguishing the semantic content. Moreover, as it can be noticed in Table 4 this problem is more highlighted in the CNN’s rather than in SVM’s performances.
Finally, comparing the iLISTEN2ISO results presented in Table 4 with iLISTEN results presented in Table 3, it can be seen that the question DA is better predicted than info-request. In this case, only 4.4% of question examples are confused with inform. The reason of this improvement is probably the segmentation process that highlighted the multifunctionality of the utterances augmenting the specificity of the classes.
Interestingly, if we compare confusion matrices for SVM (which we decided not to include in the paper for lack of space) and CNN, shown in figure 3, we notice that the most confused classes are the same for both models across both datasets. 7
We presented iLISTEN2ISO, a resource for Italian multifunctional DA tagging using ISO 246172. We argued the importance to consider turns as a composition of multiple communicative functions, in order to preserve important semantic information. Moreover, we presented different baseline DA tagging models, on both iLISTEN and iLISTEN2ISO.
We believe the presented resource could be useful to the research community for experimenting with multifunctional DA tagging in Italian, as well as cross-corpora DA tagging. As future work, we plan to explore joint DA segmentation and classification in Italian, for example taking inspiration from the work presented by Zhao and Kawahara (2019).
The research leading to these results has received funding from the European Union – H2020 Programme under grant agreement 826266: COADAPT.