In recent years, cross-lingual approaches have been successfully applied to a variety of tasks and have shown potential when available data is scarce. That potential can be e ectively leveraged when building a retrieval-based dialogue system. In this paper we investigated di erent methods of cross-lingual training of retrieval-based dialogue systems. We compare several cross-lingual approaches, including adversarial pretraining on resource-rich language dataset and further ne-tuning of pretrained models with low-resource language dialogue data. This adversarial architecture extends Dual Encoder network with language discriminator. Other approaches are based on di erent training strategies, such as mixing data in di erent languages, adversarial learning and pretraining on big data. Experiments show that adversarial learning performs competitively, which is also true for the data mixing strategy.
Retrieval-based dialogue systems have received a great amount of attention recently, mainly due to their predictability and more reasonable data requirements compared to their generative counterparts. However, retrieval-based models are also becoming larger, needing more data in order to be successfully trained. It is quite di cult to get a large dataset with human-human dialogues, su cient for training a retrieval-based system. Especially when it is not English language because most languages have limited resources.
The lack of labeled data in most languages remains an open question and
attracts more and more attention from researchers. For example, the work of Chen
et al. [
In our work we focus on the possibility of pretraining models on large dataset
and ne-tuning on the target case. One of the proposed approaches uses an
adversarial component as language discriminator for transfer semantic relations
from one language to another just as it was in the work of Chen et al. [
Retrieval-based dialogue systems. Over the last decade, various new
architectures have been proposed in attempt to maximize performance on response
retrieval datasets. Most of these use a recurrent network to construct either a at
[
Pretraining and transfer learning. Pretraining a deep neural model using
data from a di erent domain is a common method for nding a good
initialization point for successful further training, or ne-tuning, on target data. A large
number of works from various elds prove the e ectiveness of this procedure
and explore the principle behind it. For example, the work of Erhan et al. [
In the current work, we investigate the potential of pretraining a retrievalbased dialogue model using data in another language for improving performance in target language. Conceptually, we attempt to perform a sort of transfer learning by using weights of the model trained on the source dataset as a starting point for ne-tuning on the target dataset.
Adversarial learning. To extend our base model to a scenario where the
available data is bilingual, we add the language discriminator, Q, which aims to
identify the language of an input text. Speci cally, the inputs of Q are
embeddings given by E of pairs of contexts and responses in both languages. Similarly
to ADAN [
Jq( e) max q E(x)EP src [Q(E (x))]
E E(x0)E P tgt [Q(E (x0))]
E (1) Response matching loss is binary cross-entropy, denoted as Lm( ; ), between prediction from M and the ground truth label:
Jm( e) min E [Lm(M(E (x)); y)]
M x;y Adversarial loss is therefore given by:
Je min Jm( e) + Jq( e) e (2)
Ubuntu RU Customer support DE, ru 0.13 / 0.24 / 0.55 0.23 / 0.40 / 0.75 DE, en+ru 0.30 / 0.48 / 0.80 0.40 / 0.62 / 0.91 DE, pretr en 0.24 / 0.39 / 0.72 0.37 / 0.60 / 0.90 ADE, pretr en 0.27 / 0.42 / 0.75 0.40 / 0.65 / 0.92
ADE, pretr en+ru 0.17 / 0.32 / 0.66 0.27 / 0.49 / 0.82
We hypothesize that such training will allow E and M to learn languageindependent knowledge to improve response retrieval performance on our target dataset compared to the baseline, the monolingual Dual Encoder. 3
3.1
Our experiments include target data of 2 types. First is Ubuntu Corpus [
The Ubuntu Corpus. The English data set is the Ubuntu Corpus which contains multi-turn dialogues collected from chat logs of the Ubuntu Forum. The data set consists of 1 million context-response pairs for training, 20 thousand pairs for validation, and 20 thousand pairs for testing. Positive responses are true responses from humans, and negative ones are randomly chosen sampled from other responses in the training set. The ratio of the positive and the negative is 1: 1 in training, and 1: 9 in validation and testing.
The Ubuntu Corpus, Machine Translated to Russian. Some of our experiments also used the Ubuntu corpus in Russian. The data was obtained using a machine translation of the full Ubuntu Corpus into Russian. Our goal was to keep data intersection to minimum. Therefore, in the Russian version, we transliterated all the English words, thus received zero intersection between dictionaries. In experiments using this corpus, we used the rst 80 percent of the dialogues from the English corpus and 20 percent of the translated.
cludes dialogues of customer support services of a large Russian mobile network operator. It consists of 200 thousand context-response pairs in training set and 20 thousand in validation and testing. Similar to Ubuntu corpus, negative examples were randomly selected from other parts of the training set. The ratio of the positive and the negative is 1: 1 in training, and 1: 9 in validation and testing. Dictionaries of current corpus and English Ubuntu intersect on 1 percent. 3.2
For evaluation we use Recall@k (R@k) metric. The test dataset was prepared that for each context-response pair another 9 responses were selected from elsewhere in the test data. The 10 options for response were ranked, and the result was agged as positive if the correct response was included in the top-k of ranked utterances. The percentage of positive results yields Recall@k, a conventional metric for evaluating retrieval-based models. 3.3
Our main results are shown in Table 1, which aggregates performance of the retrieval process for di erent pretraining and training methods. We conduct our experiments with random embeddings initialization (experiments with pretrained bilingual word embeddings (BWEs) didn't show any other correlations). For calculating R@k, k 2 f1; 2; 5g is being used. Whenever pretraining is conducted in SOURCE language, we save the best model with best recalls on SOURCE validation set, and the same is for TARGET. All results in the Table 1 have been evaluated with test set in TARGET language.
We conducted a series of experiments with classical Dual Encoder network to learn about generalization ability of this siamese RNN for the case of crosslingual learning. First of all, we made straightforward training of DE with TARGET dataset. Following the idea of training with only low-resource data available, we consider TARGET-only training as baseline. Evaluation (Row 1) gives the lowest recalls (TARGET test set).
After that, we conduct two cross-lingual experiments. For that, we utilize SOURCE dataset, which volume is 4x larger than TARGET dataset. We exploit Ubuntu corpus to minimize domain divergence. The rst cross-lingual setup implies SOURCE pretraining and subsequent ne-tuning with TARGET data. We observe recall increase during evaluation on the test set. The essence of the next experiment is blending languages during training. Without any pretraining , we train dual encoder on mixed bilingual data, so that for every 4 SOURCE-language documents there is one TARGET-language. Row (3) represents TARGET-test and rise of recalls compared with TARGET-only and pretraining methods.
To investigate e ectiveness of adding language discriminator to DE architecture, we conduct two series of experiments, one with pretraining on SOURCE-only data and another one with blending SOURCE and TARGET languages. Following the design scheme of baselines, we prepare pretrained model and then ne-tune it with TARGET language, then evaluate with calculating recalls. Language discriminator is always trained on both SOURCE and TARGET languages, while encoder part of the system may be trained with SOURCE-only and SOURCE-TARGET joint mix, which led to two separate results. We compare ADE performance with original DE performance and observe that adversarial learning gives performance increase compared to train on TARGET language, thus it gives competitive results comparing to DE trained on the blended data.
For the basic experiments we exploit initialization of random word embeddings
by the embedding layer of the encoder. This allows us to track relative change
of the evaluation metrics. For the experiment we use pre-trained cross-lingual
word embeddings by Conneau et al. [
For all our experiments on both languages, the encoder E is implemented as
one-directional LSTM, with 300 hidden units. We set the 300-dimensional
embedding layer to be trainable. Language discriminator is implemented with 2
linear layers and ReLU non-linearities. Dual Encoder and Language
Discriminator are optimized using separate Adam optimizers [
According to the results of Table 1, it can be noted that DE model trained on blended data (DE, en+ru) and the model pretrained by the ADE method on English set (ADE, pretr en) outperform the DE model, which simply sequentially learns on English set and then ne-tune on Russian set (DE, pretr en). This e ect can be explained by the fact that model (DE, pretr en) can have over tting on target data. In this case, adding a mixture of data in (DE, en+ru) and discriminator (ADE, pretr en) serves as a kind of regularization.
In Table 2 we present the absolute values of the loss function on training data during the training model English and Russian Ubuntu with Random embeddings in order to study this issue in more detail. The results also con rm that the absolute values of the loss function in model (DE pr(en)) on the TARGET language are lower than the others, and therefore the model in this case can be over tted.
The results obtained with adversarial training on machine translation corpus make us think that the discriminator does not su ciently bridges the gaps between languages. This comes from the fact that ADE performs no better than joint DE trained on the blend of English and Russian data. Therefore, it seems that the generalization ability of the encoder itself is higher when it gradually fed with bilingual data. Our main hypothesis was that adversarial component would make encoder to generate language-independent features. However, it can be seen that adversarial training su ce more when the target data is not directly matching source data. We speculate that the reason of this may lie into the hidden latent factors of the sentence representations. Along with language latent component, a representation contains semantic latent component. When we feed the discriminator with parallel data, it does not know a di erence between these latent factors and strives to use all possible dissimilarities to predict whether the sentence is from SOURCE or TARGET. So, when the semantic factor becomes diminished, representations become meaningless | and correct response can hardly be matched to context. However, when datasets are not aligned perfectly, this e ect does not occur, maybe due to correct consideration of latent factors by discriminator (or, at least, in a more balanced way). This side e ect makes language discriminator, as we constructed it, not very e ective in the terms of cross-lingual adaptation, but helps to bridge the gaps between di erent domains, which is also important. 5
In this work we investigated di erent methods of cross-lingual training of retrievalbased dialogue systems. These methods include pretraining on resource-rich language dataset and further ne-tuning of pretrained models with low-resource language dataset. Also, we experimented with blending data in di erent languages for improving training. We exploited LSTM-based Dual Encoder network as basic retrieval-based model. We also introduce adversarial cross-lingual architecture, ADE, which is based on Dual Encoder model and exploits language discriminator. We validate e ectiveness of these methods conducting experiments on English-language Ubuntu Dialogue Corpus and two Russian-language corpora | machine translation of the Ubuntu Corpus and Russian Customer Support Corpus. Experiments show e ectiveness of mixing low- and rich-resource data and training improvement, compared to regular pretraining. Moreover, we observe competitive performance of ADE on cross-lingual tasks.
In future work we plan on adapting a more advanced retrieval-based model to an adversarial setting. We may also explore the impact of using a transformer model as the encoder of the proposed adversarial model.