English. Domain and language shift are still major bottlenecks for a vast range of task-oriented dialogue systems. This paper focuses on data-driven models for dialogue state tracking, and builds on top of recent work on dialogue domain adaptation, showing that state-of-the-art models are very sensible to language shift obtained through automatic translation. Experiments show that combining training data for the two languages (English and Italian) is always beneficial, while combining domains does not increase performance. As a relevant side effect of our work, we present a new dataset for dialogue state tracking available for Italian, derived from MultiWOZ 2.3.
Italiano. I cambiamenti di dominio e di lingua sono ancora uno dei maggiori ostacoli per una ampia classe di sistemi di dialogo task-oriented. Questo lavoro si focalizza su modelli derivati da dati per tracciare gli stati del dialogo, e prosegue lavori recenti su adattamento del dialogo al dominio, mostrando che i modelli allo stato dell’arte sono molto sensibili ai cambiamenti di lingua ottenuti tramite traduzione automatica. Gli esperimenti mostrano che combinando i dati di addestranmento per due lingue (inglese e italiano) e’ sempre vantaggioso, mentre la combinazione di dominii non migliora le prestazioni. Come importante conseguenza del lavoro, presentiamo il primo dataset per il tracciamento degli stati del dialogo disponile per l’italiano, derivato da MultiWOZ 2.3.
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
This paper is mainly motivated by the interest
of exploring, and improving, the capacity of
current data-driven task-oriented conversational
systems to address shifts of domain and changes of
language. Our starting point is the dialogue
domain adaptation (DDA) approach proposed by
A further motivation for our work is related to
the scarcity of annotated data for task-oriented
dialogues systems for the Italian language. Among
the dialogic resources originally collected in
Italian, we mention the recent JILDA dataset
A second type of data is derived through
automatic translation from English. This is the case
of the Italian version of the SNIPS dataset
We are particularly interested in data-driven
approach of dialogue state tracking (DST)
Starting from MultiWOZ 2.3, a popular dataset in English developed for booking traveling facilities (e.g., restaurants, hotels, trains, attractions) in the area of Cambridge, we incrementally operated both language and domain shifts. We provide three experimental configurations: (i) a translation of the Cambridge data set into Italian; (ii) a domain shift from Cambridge to Pisa, maintaining English as language; and, finally, (3) a configuration where both the initial domain and the language are changed. As a relevant side effect, the datasets for the three configurations are now available for further research on dialogue state tracking for Italian1.
In the paper we first introduce the relavant background in dialogue domain adaptation (Section 2), then we explain how dialogue domain adaptation is concretely applied to domain changes, and ifnally we report the experiments we have conducted (Section 4 and 5). 2
In the Dialogue Domain Adaptation setting
Given a slot-value occurring in a source
dialogue D-SOURCE, the dialogue domain adaptation
process consists of choosing the best slot-value
in KB-TARGET to substitute the slot-value in the
D-SOURCE utterance. The first step is to check
whether the slot-value is known in KB-SOURCE.
If it is known, we look for a correspondence in
KB-TARGET, otherwise we directly keep it in
DTARGET (or, in case of different languages,
translate it into target language). In order to decide if
the slot-value is in the KB-TARGET, we use a
similarity function based on a variation of the Gestalt
Pattern Matching algorithm
For the experiments of this paper we have used
FREQUENCY-KB, an adaptation strategy based
that obtained the best performance in
We broke down the problem of adapting a conversational dataset to a new language and a new domain into three different steps: first we performed delexicalization by inserting some placeholders in the place of the slot values; then we automatically translated the dataset, leaving the placeholders unchanged; finally, we substituted the placeholders with the new domain slot-values. Each one of these steps is discussed in the following sub-sections. 3.1
The setting that we are presenting involves the annotations being specifically slot-name slot-value pairs. Both the slot-values contained in the utterances, and those in the annotations, can not be translated the same way as the rest of the text, but need to undergo a Domain Adaptation process (e.g., we don’t want I need a taxi to The Old Castle to be translated into Ho bisogno di un taxi per Il Vecchio Castello).
For this reason, the first step is to delexicalize a D-SOURCE dialogue, i.e., substituting all the slotvalues in the utterances with placeholders. The example above shows this placeholder insertion, for moving from the following original sentence: “I need a restaurant in the north that has Caribbean food and a moderate price range please .” to the utterance: “I need a restaurant in <#0#> that has <#1#> food and a <#2#> price range please .” 3.2
The second step is to perform the translation from the source language to the target language without considering the placeholders. According to our example, we will produce the following Italian utterance: “Ho bisogno di un ristorante a <#0#> che abbia <#1#> cibo e un <#2#> fascia di prezzo per favore .” 3.3
As a third step, the placeholders need to be substituted back with slot-values of the target domain KB-TARGET. Which slot-values to substitute depends on the Dialogue Domain Adaptation strategy and will be discussed later.
Finally, all the slot-values - both from utterances and annotations - that could not be substituted through DDA, need to be automatically translated, which will result in the following: “Ho bisogno di un ristorante a est che abbia caraibico cibo e un economico fascia di prezzo per favore .”
As can be noted, a downside of using placeholders is that this method does not consider the subject-verb agreement, nor the order of the words to be different between the original and the translated text. It should also be observed that in the cases of north and moderate, the slot substitution selects different values from the KB, while in the case of Caribbean it could not find a correspondence in the KB, hence it got translated directly from the original. 4
We started from the public available dataset
MultiWOZ 2.3
Pisa KB-TARGET. We manually created a KBTARGET for Pisa, mirroring the instance distribution of the KB-SOURCE for Cambridge. For every entity instance of the Cambridge KB, a corresponding Pisa instance was created, keeping the slot-names as they were in the original, and changing only the slot-values. The specific instances were chosen by analysing the frequency distribution in the Cambridge KB and finding a similar correlation in the Pisa domain. For example, all the Cambridge restaurants with INDIAN food type, which is the most common in Cambridge, were substituted with Pisa restaurants with ITALIAN food type, which is the most common in Pisa. All the Pisa instances were taken from publicly available datasets containing real information on Pisa entities 2.
English to Italian, we used the automatic
transla2http://www.datiopen.it/
tor available at FBK. 3 The MT engine is built on
the ModernMT framework4 which features
neural machine translation implementing the
Transformer architecture
Test data used in the experiments were manually checked, correcting a number of translation issues, including, for instance, wrong prepositions used for time expressions (from di 13:00 to delle 13:00), and wrong agreements (from prezzi medio to prezzi medi). Training data were not corrected. Datasets. We run experiments over the following four datasets: • CAM-ENG. This is the original MultiWOZ 2.3 dataset, with Cambridge as domain and English as language. It is used as referent for the other experiments. • CAM-ITA. This is the translation to Italian of the original MultiWOZ 2.3 dataset, with Cambridge as domain. • PISA-ENG. This is the original MultiWOZ 2.3 dataset adapted to the new Pisa knowledge base, using dialogue domain adaptation, as described in Section 3. • PISA-ITA. This is the MultiWOZ 2.3 dataset, ifrst translated into Italian and then domain adapted to the Pisa knowledge base.
For all the datasets we kept the same training/test split of dialogues as in the original MultiWOZ 2.3. In addition, we have experimented the following combinations: • CAM-ITA + CAM-ENG. This combination provides all the available data for the Cambridge domain, mixing the two languages. • PISA-ENG + CAM-ENG. This combination provides all the available data for English, mixing the two domains. • CAM-ITA + PISA-ITA. This combination provides all the available data for Italian, mixing the two domains.
3We would like to thank the Machine Translation Research Unit of FBK, and in particular Mauro Cettolo, for the kind support in the generation of automatic translations. 4http://github.com/modernmt/modernmt 5http://opus.nlpl.eu Test 0.52 0.48
Dialogue State Tracking Model. The goal of
the experiments is to assess the robustness of a
dialogue state tracking model when domain and
language are changed. As for DST model, we have
used TRADE
Results of the experiments are presented in Table 1. The first column indicates which dataset the model was trained on; the second column reports the dataset used for testing the model; the last four columns report measures on the model performance. Training Accuracy refers to the Joint Accuracy obtained at training time; Turn Accuracy indicates how many single predictions were actually correct; the Joint F1 score reflects the accuracy of the model, considering both precision and recall; finally, the Joint Accuracy, measures the percentage of correct predictions of dialogue states for every dialogue turn, where a prediction is considered correct if all the slot values in the dialogue turn are correctly predicted. Results are reported into four groups depending on the dataset that has been used for testing. For every group we have three configurations: the first experiment reports the performance with the initial dataset, the second considers the extension of the initial dataset with the second language, and finally, the third experiment considers the extension of the initial dataset with the second domain. 6
Results reveal several interesting aspects. First, we register a decrease in performance between the datasets in English and those automatically translated to Italian. This can be due to the process of placeholder insertion and subsequent substitution of slot-values, along with the translation itself, which can be source of errors. On the other side, the domain adaptation from CAM-ENG to PISAENG and from CAM-ITA to PISA-ITA did not show the same decrease of performance, rather it resulted even in a small increase for the first case.
The central part of our work, however, focused on generating adapted dialogues and investigating the performance variations derived from them. Train-departure Train-destination Restaurant-area
Hotel-area
Cam-ITA Accuracy With regards to this aspect, it should be noted that the addition of a second language resulted in a significant improvement almost in all cases, with an increase of 5% for CAM-ITA, 4% for PISAENG and 13.5% for CAM-ITA. On the other side, the addition of the second domain does not bring much advantage, resulting in zero change for CAM-ENG and PISA-ITA, a small decrease for CAM-ITA and a small increase for PISA-ENG. 6.1
In order to better understand the factors that affect the variation of Joint Accuracy performances between the datasets of each group, we have analysed the overlaps among the training datasets. We estimated such overlap as the proportion of slotvalues in two datasets for every domain that are exactly the same .
We have observed that in most of the cases adding a dataset with high overlap for a certain domain produces an improvement in DST performance for that domain. As an example, the domain with highest overlap between the Cam-ITA dataset and the Pisa-ITA dataset is Taxi (86.11% of overlap). On the other side, the domain with lowest overlap between the same datasets is Attraction (44.45% of overlap). These overlaps have strong correlation with the DST performances on the two domains: the Cam-ITA + Cam-ENG dataset produces an improvement of 1.5 points with respect to the Cam-ITA dataset on the Taxi domain, and shows a decrease of 1 point on the Attraction domain.
This correlation can also be verified if we look at a slot-name level. Table 2 shows some examples of slot-names with corresponding overlaps between dialogues and slot-name prediction accuracy, taken from the Cam-ITA setting with domain and language additions. As it can be noted, when the slot-name overlap between the aggregated dialogue and Cam-ITA is higher, the respective prediction accuracy also tends to be higher. 7
We have investigated domain and language shift for data-driven task-oriented dialogue systems. We have extended recent work on dialogue domain adaptation to a cross-language setting, where both the domain and the language are changed. We showed that: (i) state-of-the-art models are very sensible to language shift obtained through automatic translation; (ii) combining training data for the two languages is always beneficial; on the contrary, combining data of different domains does not produce any improvement in all of our settings. Finally, as a relevant side effect of our work, we present a new dataset for dialogue state tracking available for Italian, derived from MultiWOZ 2.3. All the data are made available for further research on dialogue domain adaptation.