English. We describe the first edition of the “ itaLIan Speech acT labEliNg” (iLISTEN) task at the EVALITA 2018 campaign (Caselli et al., 2018). The task consists in automatically annotating dialogue turns with speech act labels, i.e. with the communicative intention of the speaker, such as statement, request for information, agreement, opinion expression, or general answer. The task is justified by the large number of applications that could benefit from automatic speech act annotation of natural language interactions such as tools for the intelligent information access, that is by relying on natural dialogues. We received two runs from two teams, one from academia and the other one from industry. In spite of the inherent complexity of the tasks, both systems largely outperformed the baseline.
Italiano. Descriviamo la prima edizione del task di “itaLIan Speech acT labEliNg” (iLISTEN) organizzato nell’ambito della campagna di valutazione EVALITA 2018. Il task consiste nell’annotazione automatica di turni di dialogo con la label di speech act corrispondente. Ciascuna categoria di speech act denota l’intenzione comunicativa del parlante, ossia l’intenzione di formulare un’affermazione oggettiva, l’espressione di un’opinione, la richiesta di informazioni, una risposta, un’espressione di consenso. Riteniamo che il task sia rilevante per la il dominio della linguistica computazionale e non solo, alla luce del recente interesse da parte della comunità scentifica nei confronti dei paradigmi di interazione e accesso intelligente all’informazione basati su dialogo. Il task ha visto la partecipazione di due team, uno accademico e uno industriale. Nonostante la complessità del task proposto, entrabi i team hanno ampiamente superato la baseline.
Speech acts have been extensively investigated in
linguistics
In particular, a recent research trend has
emerged to investigate methodologies to enable
intelligent access to information, that is by
relying on natural dialogues as interaction metaphor.
In this perspective, chat-oriented dialogue systems
are attracting the increasing attention of both
research and practitioners interested in the
simulation of natural dialogues with embodied
conversational agents (Klüwer, 2011), conversational
interfaces for smart devices
While not representing any deep understanding
of the interaction dynamics, speech acts can be
successfully employed as a coding standard for
natural dialogues tasks. In this report, we describe
the first edition of the “itaLIan Speech acT
labEliNg” (iLISTEN) task at the EVALITA 2018
campaign
The remainder of the paper is organized as follows. We start by explaining the task in Section 2. In Section 3, we provide a detailed description of the dataset of dialogues, the annotation schema, and the data format and distribution protocol. Then, we report about the evaluation methodology (see Section 4) and describe the participating systems and their performance (see Section 5). We provide final remarks in Section 6. 2
The task consists in automatically annotating dialogue turns with speech act labels, i.e. with the communicative intention of the speaker, such as statement, request for information, agreement, opinion expression, general answer, etc. Table 1 reports the full set of speech act labels used for the classification task, with definition, examples, and distribution in our corpus. Regarding the evaluation procedure, we assess the ability of each system to issue the correct speech act label among those included in the taxonomy used for annotation, described in the Section 3. Please, note that the participating systems are requested to issue labels only for the speech act used for labeling the user’s dialogue turns, as futher detailed in the following. 3 3.1
We leverage the corpus of natural language
dialogues collected in the scope of previous research
about interaction with Embodied Conversational
Agents (ECAs)
The dialogues were collected using a Wizard of Oz tool as dialogue manager. Sixty subjects (aged between 21–28) were involved in the study, in two interaction mode conditions: thirty of them interacted with the system in a written-input setting, using keyboard and mouse; the remaining thirty dialogues were collected with users interacting with the ECA in a spoken-input condition. The dialogues collected using the spoken interaction mode were manually transcribed based on audiorecording of the dialogue sessions.
During the interaction, the ECA played the role of an artificial therapist and the users were free to interact with it in natural language, without any particular constraint: they could simply answer the question of the agent or taking the initiative and ask questions in their turn, make comments about the agent behavior or competence, argument in favor or against the agent’s suggestion or persuasion attempts. The Wizard, on his behalf, had to choose among a set of about 80 predefined possible system moves. As such, the system moves (see Table 2) are provided only as a context information but are not subject to evaluation and do not contribute to the final ranking of the participant systems. Conversely, the participating systems are evaluated on the basis of the performance observed for the user dialogue turns (see Table 1). 3.2
Speech acts can be identified with the
communicative goal of a given utterance, i.e. it
represents its meaning at the level of its
illocutionary force
Defining a speech act markup language is out of the scope of the present study. Therefore, we adopt the original annotation of the Italian advicegiving dialogues. Table 1 shows the set of nine labels employed for the purpose of this study, with definitions and examples. These labels are used for the annotation of the users’ dialogue turns and are the object of classification for this task. In addition, in Table 1 we report the speech act labels used for the dialogue moves of the system, i.e. the conversational agent playing the role of the artificial therapist. The speech act taxonomy refines the DAMSL categories to allow appropriate tagging of the communicative intention with respect to the application domain, i.e. persuasion dialogues in the healthy eating domain.
In Table 3 we provide an excerpt from a dialogue from our gold standard. The system moves (dialogue moves and corresponding speech act labels) are chosen from a set of predefined dialogue moves that can be played by the ECA. As such, they are not interesting for the evaluation and ranking of participating systems and are provided only as contextual information. Conversely, the final ranking of the participating systems is based on the performance observed only on the prediction of speech acts for the users’ move, with respect to the set of labels provided in Table 1. Please, note that the two sets of speech act labels for the user and the system moves, in Table 1 and Table 2, respectively, only partially overlap. This is due to the fact that the set of agent’s moves includes also speech acts (such as persuasion attempts) that are observed only for the agent, given its caregiver role in the dialogue systems. Vice versa, some speech act labels (such as clarification questions) are relevant only for the user moves. 3.3
We provide both the training and testing dialogues in the XML format following the structure proposed in Figure 1. Each participating initially had access to the training data only. Later, the unlabeled test data were released during the evaluation period. The development and test data set contain 40 and 20 dialogues, respectively, equally distributed with respect to the interaction mode (textvs. speech-based interaction). 4
Regarding the evaluation procedure, we assess the ability of each system to issue the correct speech act label for the user moves. The speech act label used for annotation of the user moves are reported in Table 1.
Specifically, we compute precision, recall and F1-score (macroaveraging) with respect to our gold standard. This approach, while more verbose than a simple accuracy test, arise from the need to correctly address the unbalanced distribution of labels in the dataset. Furthermore, by providing detailed performance metrics, we intend to enhance interesting discussion on the nature of the problem and the data, as they might emerge from the participants’ final reports. As a baseline, we use the most frequent label for the user speech acts (i.e., STATEMENT).
OPENING
CLOSING
INFO-REQUEST
SOLICITATION-REQ-CLARIF
STATEMENT
GENERIC-ANSWER
AGREE-ACCEPT
REJECT
KIND-ATT-SMALLTALK
The task was open to everyone from industry and
academia. Sixteen participants registered, but only
two teams actually submitted the results for the
evaluation. A short description of each system
follows:
UNITOR - The system described in
X2Check - The team did not submit the report.
The performance of the participating systems is
evaluated based on the macro (and micro)
precision and recall
The best performance (0.6531) is provided by the UNITOR system. Both systems are able to overcome the baseline also for micro-F. The baseline has a low macro-F since it predicts always the same class (STATEMENT) and for the other classes the F-measure is zero. As expected, the micro-F overcomes the macro-F since some classes are hard to predict due to the low number of examples in the training data, such as AGREE, SOLICITATION-REQ-CLARIF and REJECT. Precision, Recall, and F-score values by speech act labels are showed in Table 5.
We also provide the confusion matrix for each
system, respectively Table 6 for UNITOR and
Table 7 for X2Check. We observe that, for both
systems, the class REJECT is the most difficult
to classify. This evidence is consistent with the
findings from previous research on the same
corpus of dialogues
We received two runs from two teams, one from
academia and the other one from industry. In
spite of the inherent complexity of the tasks, both
systems largely outperformed the baseline,
represented by the trivial classifier always predicting
the majority class for users’ moves. The best
performing system leverages syntactic features and
relies on a Structured Kernel-based Support
Vector Machine. Follow-up editions might involve
extending the benchmark with dialogues from
different domains. Similarly, dialogues in different
languages might be also included in the gold
standard, as done for Automatic Misogyny
Identification task at EVALITA 2018