Recently, it has been reported that the quality of chats can be improved by utilizing information about what kind of persona the chat interlocutor and what kind of personality the system itself mimics in order to generate responses using LLMs. In these studies, personality data is stored in a form of statements describing interlocutor's profile. In order to control LLM using profile describing sentences in actual chats, it is not enough to prepare them in advance, but it is necessary to actively add them through the chats. However, not enough research has been conducted on generating the profiling sentences from the utterances in the chat dialogs. In particular, there are no studies on the generation of profiling sentences that can only be generated with context the context. Therefore, in this study, we propose a task to generate profiling sentences from the target utterances based on context in a chat dialog in Japanese, and create a dataset for this task. Experiments on the created dataset and analysis show that LLM can generate profiling sentences while taking the context into account.
In recent years, the performance of large language models (LLMs) has dramatically improved, leading
to significant advancements in dialog systems. However, in case of chat, dialog systems based solely on
language models sufer from limitations due to the inability to remember users in the long term and
inconsistent utterances in simulating personality. Related research [
In order to utilize profiling sentences for response generation in actual conversations, it is essential
to prepare profiling sentences that express the personality of the interlocutor (hereinafter referred to as
user profiling sentences ) and profiling sentences that represent the personality that the system should
assume as its own personality (hereinafter referred to as system profiling sentences ), which have been
collected during previous dialogs. Previous studies have either prepared profiling sentences in advance
[
In previous research on improving the consistency of the system’s own dialog by using system
profiling sentences, about five profiling sentences are prepared as a part of the personalities commonly used
in chats before starting a chat and are added to the input for dialog generation [
CEUR Workshop Proceedings
ceur-ws.org ISSN1613-0073 beyond these sentences, and thus cannot maintain consistency over a long period of time. It is not realistic to prepare in advance information about all aspects of a person’s personality as system profiling sentences to maintain consistency, hence the system needs to make sure that the utterance it is about to generate is consistent with past utterances.
In addition, it is desirable for the system to be able to recognize user profiling sentences from utterances during chats, since it would be a great burden to have the users themselves prepare their own profiling sentences in advance.
Thus, both long-term memory and utterance coherence research require the recognition of individuality from utterances. We address this issue by generating profiling sentences from utterances as a task of personality recognition.
We propose a task to generate profiling sentences with LLMs that take into account the context of chats, and construct a dataset by LLM generation and human consistency checks. Table 1 shows examples of sentences inferred by human from utterances and their implicit context. In addition, we propose a method to train LLM and generate profiling sentences on that dataset, and report on the automatic and manual evaluation of the generated results. Furthermore, we confirm the importance of considering the context of the chat by manually checking the dataset itself and the output of the trained profiling sentence generation model are context-sensitive.
The contributions of this paper are as follows: • Proposed a task to generate profiling sentences considering the implicit chat context • Proposed method for training profiling sentences generation with LLM • Identified diferences between automatic and manual evaluation in profiling sentences generation • Confirmed that profiling sentences generation by LLM is dificult to take into account
the chat context
Currently, there are three main methods for linking utterances to profiling sentences, but the existing methods are not suficient for the task of recognizing profiling sentences in chit-chat utterances.
First, there is an approach that uses classification to recognize whether an utterance corresponds to a
profiling sentence. For example, some studies [
Second, there is an approach to extract profiling sentences from dialog by rules. For instance, [
Third, there is a method to generate profiling sentences from dialogs using LLM. However, existing
research [
During a chat, humans recognize and update information about the personality of the person they are talking to each time they receive the other person’s utterance. Regarding the recognition part, humans can to some extent write down the information of the other party as short sentences for each utterance. It is important to note that the profiling sentences written out by the human are not extracted solely from the target utterance, as shown in Table 1. Our goal is to enable the system to extract and record the interlocutor’s information as profiling sentences from the utterance, similarly to human recognition. As in previous studies, we do not define the exact nature of a profiling sentence. Instead, we treat all sentences that describe personality or personal facts as profiling sentences, consistent with their usage and validation in prior research. It would be preferable generate human-like profiles in the system, but the annotation becomes very complex when trying to rigorously create such tagged dialog data. Therefore, this paper focuses on demonstrating the importance of generating profiling sentences by LLM using context. In this section, we summarize the diferences between the ideal task we originally wanted to perform, the conventional task, and the proposed task.
First, we describe the ideal task. Let Δ be one of the profile sentences of the interlocutor’s profiling sentence can be inferred from the th utterance of the dialogue, and let Δ = {Δ 1, Δ 2, Δ 3...} be all the profile sentences from utterance . In addition, = { 1, 2, 3...} be all interlocutor’s profiling sentences collected from the previous dialogs, updated by Δ after receiving .
Note that some profiling sentences are updated, including the place of residence or hobbies, and that is not the union set of Δ from = 1 to = .
Let = { 1, 2, 3...} be the context other than the target utterance necessary to generate the information about another person known from the utterance, then the profiling sentence generated by model ℎ , which has (theoretically) human-like capabilities, can be Δ = ℎ ( , ). The ideal profiling sentence generation is to obtain ℎ and generate human-like profiling sentences. However, can assume a variety of elements, which is problematic when creating a dataset. In the case of a human predicting an interlocutor’s profiling sentence from utterance, the can be assumed to be the profile of a dialog interlocutor already known, the dialog history, or any other concept that is shared with the interlocutor. The elements of in this paper are the interlocutor’s profiling sentence −1 held before receiving an utterance and the history of utterances = { 1, 2, 3..., −1 }. When starting to talk, 1, 1, and 0 are empty sets. In this study, we do not update the profile, but for the sake of explanation, we denote the profile update model as . The formula for profile update is = (Δ , −1 ). In summary, ideally the profiling sentence should be extracted and updated as follows: • Extraction: Δ = ( • Update: = (Δ , ) = (
, −1 , ) , −1 )
It is most desirable to create a dataset by storing and Δ for each , but in reality, problems such as the burden and cost of annotators appear. The longer the conversation is, the more profiling sentences are collected for each chat, and it is not easy to annotate each utterance while keeping track of all of profiling sentences.
On the other hand, the task setting of the profiling sentence generation in the previous study [
Δ _ We propose a task to generate profiling sentences from a target utterance while considering the two previous utterances as context. Here, the profiling sentence is an important element, but we ignore it in this study. In our task setting, it is crucial to efectively utilize the two preceding utterances as context for generation. In other words, with the utterance history = { −1 , −2 }, our proposed task can be expressed as follows.
Δ = ( , ) = ( , ) = ( , −1 , −2 )
This task aims to generate zero or more known profiling sentences Δ from the target utterance using contextual information −1 , −2 . Table 1 shows examples of utterances with 4 profiling sentences, Table 2 shows examples of utterances with 0 and 1 profiling sentences. In the previous study, utterances that humans inferred to be without profiling sentences were not included in the target utterances. However, in this study, we included them to simulate a more realistic chatting scenario. Since humans can also infer implicit contextual information −1 and −2 to predict profiling sentences from target utterances , the results are compared with the results of a human performing Δ _ = ( , −1 , −2 ) in the same task.
We create a dataset for this task and provide a benchmark. We also compare and analyze Δ and in the proposed task to confirm the importance of context in profiling sentence generation.
In this section, we first describe the chat data we use to build our dataset, and then explain how we create profiling sentences from the chat data and ensure their quality.
We extend an existing chat dataset by creating a dataset that links target utterances with
contextsensitive profiling sentences. The original dataset for the extension is the JPersonaChat dataset [
As described in the task setup section, we add the two most recent utterances in the utterance history,
including the interlocutor utterance, to a single target utterance as contextual information. We use
a subset of the original dataset divided by dialogue. The target dialogs for extraction are 500 dialogs
randomly selected from the JPersonaChat dataset, which contains 5,448 utterances. All utterances in
the obtained subset dialogs are considered as target utterances, and each target utterance and the two
utterances immediately preceding it are considered as one case of data. Here, we counted the number of
utterances required by one of the authors to infer profiling sentences for 100 target utterances randomly
selected from the original chat data, and found that there was only one target utterance that required
three or more utterances of context, so we set the number of utterances used for context to two. If the
target utterance is within two utterances from the start of the conversation, the entire dialogue history
is added. We want to obtain corresponding profiling sentences for each target utterance. However, it
takes a lot of efort to obtain a comprehensive and accurate profiling sentences all by hand. The method
in which a single annotator infers a profiling sentence for a single target utterance is unreliable. The
method in which multiple people check the profiling sentences inferred by one person is considered
to be more accurate, but it lacks comprehensiveness because it misses profiling sentences that were
not recognized by the annotator who made the inference. Ideally, the profiling sentences written by
several people should be merged, and the merged profiling sentences should be checked by several
people, but this is very costly. Therefeore, we use LLM to create profiling sentence dataset to decrease
the costs. This method of generating data using a language model and manually checking it is often
used in recent years [
This section describes the process of writing profiling sentences from utterances using LLM. We use gpt-4-0613 as the LLM when creating the dataset. As a preliminary experiment, we asked LLM to infer profiling sentences using several prompts, and the results with instructions only (zero-shot) were more comprehensive than the results generated with examples of correct answers (few-shot). Therefore, we use a zero-shot setup for LLM inference. the LLM is given a prompt that is a concatenation of the instruction, two prior utterances from the dialog history, and the target utterance, and is asked to generate as many profiling sentences as possible for all the target utterances. The profiling sentences include utterances for which no profiling sentences exist, but if they do not exist, they are output as none. As noted in the task description section (2.2), the target utterances for this study include those in which humans do not infer any profiling information. However, adding target utterances without profiling sentences to the dataset creates a situation where the linkage between them is no longer 1-to-many, rendering some existing automatic evaluation methods unusable. For a more accurate evaluation, the dataset is checked manually for validity. The human evaluation also compares the results of human inferences Δ _ = ( , −1 , −2 ) on a small number of utterances (100).
Next, to ensure the accuracy of the profiling sentences generated by the LLM, the correspondence between the target utterance and the profiling sentences is manually checked. This annotation work is done by a crowdworker. In order to check whether each profiling sentence can be inferred from the target utterance, we first divide a “target utterance, two prior utternaces” (hereafter referred to as “utterance group”) and a “profiling sentence” to achieve one-to-one alignment. Annotators are assigned to evaluate pairs of utterance groups and profiling sentences. The information in the profiling sentences, derive from the target utterance, is assessed by three raters using three categories: correct, possibly correct, and incorrect. An example of an annotator’s decision is shown in Table 3. A majority vote is used as the final decision, and in the case of a split decision by all workers, an intermediate label, possibly correct, is adopted. Note that our goal is to generate profiling sentences derived from the target utterance, thus any profiling sentences that are mentioned only in the utterance history and are unrelated to the target utterance are judged as incorrect. profiling sentences with incorrect Japanese or profiling sentences that can be applied to any utterance (e.g., “I can speak the language”) are also judged to be incorrect. The total number of profiling statements inferred by the LLM is 16,971, of which 9,475 (55.83%) are judged correct, 1,763 (10.39%) are judged possibly correct, and 5,733 (33.78%) are judged incorrect. Data annotated as incorrect are not used in this experiment. The inter-annotator agreement (three-way average of weighted kappa coeficients) is 0.557, indicating moderate agreement. This suggests that there are some individual diferences in what is perceived as a profiling sentence. After the annotation is completed, the dataset is created by reverting to data for each target utterance for correct and potentially correct “utterance groups” paired with “profiling sentences”. When processing the data for each target utterance, the profiling sentences are left blank for target utterances that don’t have any profiling sentences associated with them, as shown in the Table 2. The dataset we created is publicly available 1 . 1https://github.com/shingetsu-ak/generation-of-interlocutor-profiling
We train models and conduct experiments to validate the datasets we create. In this section, we describe the details of model construction and training, evaluation method, experimental results, and an analysis of contextual influences.
For training, the dataset is randomly shufled for each target utterance and divided into subsets of training, validation, evaluation = (8:1:1) by the number of target utterances. Although it is possible for a target utterance to appear in the utterance history of other target utterance, since our task is to map target utterances to profiling sentences, we have separated them in this way because we believe that training, validation, and evaluation should be done on unique target utterances.
Although the previous study [
In both the method for training simple concatenations (concat) and the method for training utterances
and profiling sentences on a one-to-one basis (profile-wise), profiling sentences are generated using
a Transformer-based decoder, following the approach in the previous study [
We propose an one-to-one training method that learns a profiling sentence from a target utterance,
but with this method the model can only generate one profiling sentence per inference. Therefore,
in order to obtain multiple profiling sentences in one utterance, it is necessary to let the model infer
multiple times and remove the same profiling sentence from the generated results. Therefore, to ensure
that the diversity of the learned profile sentences is reflected in the generation of the profile sentences,
they are generated 10 times. Since many of the 10 generated profiling sentences are semantically similar,
the semantic similarity is measured by sentenceBERT [
We compare models that learn profiling sentences concatenated together, and models that learn utterance groups and profiling sentences one-to-one. profiling sentences that are confusing to humans may confuse the model during training, thus we created a model that uses profiling sentences that the crowdworker judged to be “possibly correct” during dataset construction and a model that does not use those profiling sentences for training, and made comparisons. Therefore, there are four models to compare: concat (correct), concat (correct+possibly correct), profile-wise (correct), and profile-wise (correct+possibly correct).
We use not only the same automatic utterance-level metrics as in previous studies, but also human
ratings of the profiling sentences. In previous studies [
For automatic evaluation of each utterance, a test set of the created dataset is used. BLEU [
For the human evaluation at the profiling sentence level, we use 100 target utterances randomly selected from the test set of the created dataset. In this case, the target utterance without a profiling sentence is also evaluated as a single case of data, with the case where no profiling sentence is generated as the correct answer. The generated “profiling sentences” are mapped one-to-one to “target utterance, two preceding utterances” and judged manually as correct or incorrect by annotator. Since our goal is to generate all the profiling sentences that a human would infer from the target utterance, we include profiling sentences that are possibly correct in the correct answer and make a binary decision. Three annotators are hired for the human evaluation, and the decision is made by majority vote. We also create a set of profiling sentences Δ _ , all of which are manually extracted for the 100 test sets used in the manual evaluation. For Δ _ , three annotators are asked to write out profiling sentences, and one of the authors checks for duplicates and removes them.
The experimental results of the automatic evaluation are shown in Table 4. Comparing the combined method and our one-to-one training method, the combined method scored higher overall in the automatic evaluation. This indicates that training and outputting the combined profiling sentences is advantageous for automatic evaluation against the combined correct sentences. Next, when comparing the model trained on only correct answers with the model trained on both correct and possibly correct answers, the model trained solely on correct answers obtained a lower score. This may be because the correct sentence, referred to as the correct answer in the automatic evaluation, included profiling sentences that could potentially be correct in the training. This likely favored the model with a more diverse output.
The experimental results of the human evaluation are shown in Table 5. Here, LLM+GOLD is the result of the dataset itself. The total number of Δ _ is used to calculate the recall and F1 scores. Overall, the results were diferent from those of the automatic evaluation. In particular, profile-wise (correct), which had the lowest score in the automatic evaluation, resulted in the highest F1 score in the human evaluation. This suggests that automatic evaluation at the utterance level is less correlated with human evaluation at the profiling sentence level.
The profiling sentences generated by the human evaluation are further analyzed to check whether the model trained on the created dataset is able to generate profiling sentences while being influenced by the context. First, for comparison, we manually count the profiling sentences that could not be predicted without context in the Δ _ , a set of profiling sentences that are all extracted manually. The results show that 20.92% (32/153) of the profiling sentences required context. Thus, we see that there are a certain number of profiling sentences in the test set that require context. Next, we analyze the profiling sentences included in the dataset we have created. The results show that 3.40% (5/147) of the profiling sentences referred to the context. This confirms that the proportion of contextual references in LLM profile writing is much lower than that in human profile writing. This is a limitation of using LLMs to extend the dataset, but could be improved as the capabilities of the LLMs increase. The percentage of profiling sentences generated by the learned model is as follows: • concat (correct): 5.06% (4/79) • concat (correct + possibly correct): 7.14% (6/84) • profile-wise (correct): 3.95% (3/76) • profile-wise (correct + possibly correct): 7.14% (6/84) We see that the profiling sentence generation model is able to generate profiling sentences with reference to the context at about the same rate as the dataset, although less than the human profiling sentences. Comparing the model trained with only the correct answer and the model trained with both the correct answer and the possibility of the correct answer, it is found that the model trained with both the correct answer and the possibility of the correct answer is more context-referenced. On the other hand, as can be seen from Table 5, this model has lower prediction accuracy, hence achieving both is an issue to be solved in the future.
In this study, we proposed a task to generate profiling sentences from target utterances by adding two utterances as context, and created and evaluated a dataset linking profiling sentences that can be inferred from a single utterance. For the generation of profiling sentences, we proposed a one-to-one training method in which profiling sentences are learned from target utterances, in addition to the conventional method of training profiling sentences that can be inferred from a single utterance by concatenating them. For the evaluation, automatic evaluation at the utterance level and human evaluation at the profiling sentence level were performed. The experimental results show distinct discrepancy between automated and human evaluations for inferring profiling sentences. We also confirmed that the proposed training method is as efective as, or even more efective than, existing approaches. Analysis of the generated profiling sentences confirmed that the profiling sentence generation model is able to generate implicit profiling sentences at approximately the same rate as the dataset. In the future, we plan to work on creating a dataset that includes more contextually referenced profiling sentences. We also intend to analyze the relationship between profiling sentences for each interlocutor by converting the utterances in the created dataset into dialogue-by-dialogue data.
We conducted our experiments in Japanese, but further verification is needed because the results may change if the experiment is conducted in English or any other language. This paper has shown the diferences between the automatic evaluation methods used in existing research and human evaluation, but it is very costly. It is important to study more context-based one-to-many automatic evaluation metrics. Although this research focuses only on the chat domain, the method itself may be applicable to other domains such as opinion extraction. Experiments in other domains is desired in the future. Our research is based on dialogue data created by crowdworkers who pretended to be non-existent people. Therefore, our dataset also does not include profiles of real people. However, when considering actual applications, collecting profiles of existing interlocutors may be a problem from the perspective of privacy. We do not recommend collecting user information without their permission. To ensure smooth and reliable interactions between users and systems, sentence profiling is essential, and this research focuses on achieving that goal.
This research was supported by JST Next Generation Challenging Researchers Program JPMJSP2119 and JST CREST Grant Number JPMJCR20D2, Japan.