Introduction

With the success of existing dialogue models on generating human-like responses, dialogue models are required to express their own personality. Zhang et al. [1] introduce a personaconditioned dialogue dataset PersonaChat to build persona consistent dialogue models. However, the best performing generative models trained on PersonaChat such as GPT2 [2] still generate fluent but inconsistent responses. The reason is that dialogue models are trained with standard maximum likelihood loss, which lacks the constraint of persona consistency.

Unlikelihood training is a technique developed for removal of repetition in language model completions. Li et al. [3] use unlikelihood to solve the persona consistent issue of dialogue models. However, they just consider the whole sentence and ignore the keywords. We observed that most inconsistent responses are caused by similar but inconsistent words. As shown in fig. 1, the generated response of GPT2 model is inconsistent with persona due to the word "20". The word "20" is similar to "26" but is inconsistent considering the fourth persona description.

In this poster, we construct negative word set to separate semantics of similar but inconsistent words. Firstly, we obtain coarse negative word set by Word2Vec. Secondly, we use ConceptNet [4] to remove synonyms and add antonyms. Thirdly, we use the unlikelihood loss to compute the loss of negative word set, which assigns a low probability to inconsistent words. The experiments show that our method can generate more consistent responses.

Persona

Query: how old are you?

Resonse: i am 26 years old. 3. i have three dogs. 2. i am a man.

4. i am 26 years old.

1. i am a doctor. inconsistent GPT2: i am 20 years old.

Figure 1:

The response generated by GPT2 is inconsistent with the persona.

Approach

Problem Definition

Our task is to train a generative model to generate a persona consistent response. Formally, given a set of persona texts 𝑃 = {𝑃 1 , 𝑃 2 , … , 𝑃 𝑚 }, and an query 𝑄, to generate a response 𝑅 which should consistent with persona. Here 𝑃 𝑖 , 𝑄 and 𝑅 are sentences, which are consist of some words, such as 𝑅 = {𝑟 1 , 𝑟 2 , … , 𝑟 𝑛 }, where 𝑛 is the length of 𝑅.

Training Loss

Likelihood Loss

Likelihood training is commonly used in text generation models. Some pre-trained generative models trained with likelihood can generate fluent and meaningful responses. For a sample {𝑃, 𝑄, 𝑅}, the likelihood uses maximum likelihood estimation (MLE) to compute loss:

𝐿 𝑀𝐿𝐸 = − log (𝑝 𝜃 (𝑅 | 𝑃, 𝑄)) = − |𝑅| ∑ 𝑖=1 log (𝑝 𝜃 (𝑟 𝑖 | 𝑃, 𝑄, 𝑅 <𝑖 ))(1)

where 𝑟 𝑖 is current word needed to be predicted, 𝑅 <𝑖 are previous words before 𝑟 𝑖 and 𝑝 𝜃 (𝑟 𝑖 | 𝑃, 𝑄, 𝑅 <𝑖 ) represents the probability of 𝑟 𝑖 predicted by model conditional on 𝑃, 𝑄, 𝑅 <𝑖 .

Unlikelihood Loss

Likelihood training increases the probability of true word and decreases the probability of all other words. On the contrary, unlikelihood training decreases the probability of negative words. The unlikelihood (UL) loss can be defined as:

𝐿 𝑈 𝐿 = − log (1 − 𝑝 𝜃 (𝑅 | 𝑃, 𝑄)) = − |𝑅| ∑ 𝑖=1 ∑ 𝑐∈𝐶 𝑖 log (1 − 𝑝 𝜃 (𝑐 | 𝑃, 𝑄, 𝑅 <𝑖 ))(2)

where 𝐶 𝑖 is the negative word set of current word 𝑟 𝑖 .

Constructing Negative Word Set

Negative Word Set

MLE leverages the previous context of the word to predict the current word, which results in words with similar contexts having similar semantics. Therefore, the model might generate similar but inconsistent words. One solution is separating the semantics of similar words, which can be done by UL training. The core of UL training is to construct negative word set. Unlike Welleck et al. [5], our negative word set contains inconsistent words with current word conditional on persona. As shown in fig. 1, the generated word "20" is inconsistent with "26" in persona. The negative word set of "26" maybe contains "20", "25" and "30" et al.

Word2Vec

The challenge is how to construct negative word set used for UL loss. Word2Vec, a method of learning word embedding, follow the distributional Hypothesis: words that occur in the same contexts tend to have similar meanings. Word2Vec leverages the previous and following context to predict current word, which is similar to MLE. We learning word embedding of all words on dataset by Word2Vec. we approximatively regard similar words computed by cosine similarity of word embedding as negative word set. For example, the negative word set of "man" computed by Word2Vec contains "male", "boy", and "girl" et al.

ConceptNet

We also observed that there are some noise words in the negative word set constructed by Word2Vec. The synonym, hyponym, and hypernym have similar context but are consistent with the word, so we should remove them from the coarse negative word set. Fortunately, ConceptNet [4], a knowledge graph containing common sense knowledge, provides these three relations of one word. For example, "male" is a synonym of "man" and "dog" is a hyponym for "pet". Besides, ConceptNet also provides antonyms that can be added to negative word set. For example, "man" and "woman" is a pair of antonym, and they have similar context but opposite semantics.

Training Model

We use GPT2 [2] as our basic model because it shows strong performance in dialogue generation.

During the training phase, we combine the UL loss with the MLE loss as follows:

𝐿 = 𝐿 𝑀𝐿𝐸 + 𝐿 𝑈 𝐿(3)

where 𝐿 𝑀𝐿𝐸 aims to promote true words, training model to assign the highest probabilities to such words. On the other hand, 𝐿 𝑈 𝐿 focuses on negative words, so that the model can learn to rank negative words lower than true words effectively.

Experiments

We verify our method on PersonaChat [1]. For automatic evaluation, we employ a classification model to evaluate the persona-consistency of the generated responses. The results contain three categories: consistent (Consi.), contradictory (Contr.), and neutral. We use perplexity (PPL) to measure the fluency of responses. For human evaluation, we randomly select 100 samples per method and ask three professional annotators to evaluate the quality of these samples. Annotators also label generated responses as consistent (Consi.), contradictory (Contr.), and neutral with persona. The fluency (Flue.) of responses is rated on a 3-scale, with higher scores indicating better fluency.

Table 1 shows that the model trained by UL loss achieves better results on all metrics than the base model. Higher consistent ratio and lower contradictory ratio indicate our method can separate semantics of similar but inconsistent words. The consistency of responses is further improved after using ConceptNet, which means the knowledge such as synonyms provided by ConceptNet is useful to construct higher-quality negative word set.

Conclusion

In this poster, we propose a method to construct negative word set for unlikelihood training to separate semantics of similar but inconsistent words. Experiments demonstrate that our method can improve persona consistency of dialogue generation. In future work, we are interested in leveraging more efficient loss and constructing more appropriate data to improve the persona consistency of dialogue generation.