The development of dialogue systems benefits from the study of the communication strategies used by human speakers. In the context of recommendation dialogue systems some researchers have investigated the sociable recommendation strategies employed by the Recommenders in natural settings to make successful and persuasive recommendations (Hayati et al., 2020, INSPIRED corpus). However, the Seeker's contribution, as well as the Recommender's, shapes the development of the communicative exchange, in that the Seekers may use specific strategies to disclose their preferences and reach their goal. So, modelling the Seeker's communicative strategies along with the ones used by the Recommender may improve the efficiency of recommendation dialogue systems. In this work, we provide a reliable tagset for the Seekers utterances present in the Inspired dataset, defining a set of communicative strategies coherent with the already existing one for the Recommenders.
Nowadays conversational recommendation
systems seem to be acquiring a fundamental role in
information seeking and retrieval. In a recent
paper, Hayati and her colleagues
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). a dataset of recommendation dialogues collected in a realistic setting, enriched with a detailed annotation of the sociable recommendation strategies employed by the Recommender.
However, as in any interaction, these
dialogues are the result of the cooperation between
the interlocutors, who actively partake in both
the construction of meaning and of the
relationship among each other
In this work, we aim to fill this gap proposing
a tagset for the Seekers communicative strategies
that is coherent with the one previously provided
for the Recommenders by Hayati and colleagues.
The paper is structured as follows:
recommendation dialogue systems are considered in relation
to the Argumentation Theory ( § 2) and the
Inspired tagset
Recommendation dialogues are characterized by
two or more participants who disclose their
preference and make recommendation in order to
select a certain item that should satisfy the
re1The present study is the result of a collaborative work of
all the authors. Paragraphs 2 and 2.1 have been written by
Martina Di Bratto, paragraph 3 by Marta Maffia and Ancuta
Budeanu, 3.1 and 4 by Riccardo Orrico and, finally, sections
1 and 5 by Loredana Schettino.
quirements retrieved during the communicative
exchange. Conversational Recommendation
Systems (CoRS), in the same way, aim at
finding or recommending the most relevant
information (e.g., web pages, answers, movies, products)
for users based on textual- or spoken-dialogues,
through which users can communicate with the
system more efficiently using natural language
conversations
The Inspired corpus
Also the collected conversations present the two-phase pattern typical of the recommendation 2Dataset and code are freely available online. task. In the exploration phase preference elicitation strategies are used by the Recommender in order to collect sufficient information regarding the seeker’s preferences and tastes about the movie domain. They are divided in experience inquiry and opinion inquiry.
In the exploitation phase, on the other hand, eight different strategies have been recognized. During this phase, thus, the Recommenders can start the interaction by offering help to find the recommendation. They can also express their personal opinion or personal experience in order to convince the Seekers basing the recommendation on their own experience. Moreover, they can opt for other persuasive strategies such as credibility, similarity, encouragement, preference confirmation or self-modeling which are mainly used to built rapport with the Seekers, also establishing and improving their role as domain experts. 3
Taking into account the Recommender’s
annotation scheme proposed by Hayati and colleagues
We divided Seekers’ strategies into four categories.3. The first category corresponds to a single strategy, labeled as recommendation request and used by the Seeker to generically ask for a candidate item: ex. Do you have any recommendations? 3In this pilot stage of the research, we decided to work on the labelling of communicative strategies used by the Seekers in the above mentioned ”user information gathering” and ”movie recommendation” phases of dialogues. Other strategies, located at the beginning (greetings) and at the end of the dialogues (intentionality, acceptance, refusal) were also identified but they will not be discussed in this paper The second category (henceforth called get movie) includes global requesting strategies, by which the Seeker can direct the recommendation process on the basis of specific attributes of the movies. They are divided as follows: • get from genre, used to ask for a candidate item according to its genre; ex. What kind of comedy movies do you have to recommend? • get by actor, used to ask for a candidate item featuring a specific actor/actress; ex. Do you have another movie with Tom Hanks? • get similar to, used to ask for a candidate item with analogous attributes to another specified item; I would love to see a remake or something similar to Notting Hill. • get by year, used to ask for a candidate item according to its release date; Do you know anything more recent?
The third category corresponds to the giving preference strategies usually uttered by the Seeker to reply to the Recommender’s inquiries: • personal opinion used to specify personal preferences over candidate items or one/some of their attributes. Also, it can express a positive or negative value towards them; ex. I liked the acting and the movie itself; I didn’t like that movie. • personal experience, used to tell about experiences that could be present or not in the past, thus defining if the Seeker have or have not watched that movie; ex. I saw the trailer for For v Ferrari; No, I haven’t seen it.
Finally, the get info category includes local requesting strategies uttered by the Seeker to require information about a specific, recommended movie. This category includes: • get genre, used to asks about the value of the attribute ”genre” for a specified item; ex. Is it an action movie? • get acted in, used to ask about the movie’s cast; Do you know who else is in the cast? • get score, used to request information about the quality evaluation of the movie; ex. How about the new Rambo? • get plot, used to ask about the storyline of a movie; ex. Could you tell me what the general plot is?
In order to test the validity of the annotation system, we proceeded to annotate Seekers’ utterances taken from the first 20 dialogues between Recommenders and Seekers (331 utterances produced by Seekers) which were annotated by 5 annotators (the authors of this contribution). Each Seeker’s utterance could be given one or two labels: a second label was added in those cases in which two strategies were expressed by the Seeker in the same utterance. In most of these cases the assignment of a first and a second label was facilitated by the sequentiality of information in the utterance (ex.the utterance I recently watched John Wick 3, very good movie, in my opinion and fully action packed was given personal experience as ifrst label and personal opinion as second label by all the annotators); on the contrary, other cases could present a higher level of ambiguity (for example, in case an annotator intended the utterance i like the sci-fi movies to express both personal opinion and get from genre. In these cases, there was not a unique criterion to identify which one was the first and which one was the second label). Data about annotators agreement and preliminary results of Seekers’ strategies based on our annotation are presented in the following sections. 3.1
Since the annotation system accounts for the possibility of having two different strategies within the same utterance, the agreement among the 5 annotators could have 3 possible outcomes: for each utterance there could be i) agreement (A), all 5 annotators agreed on both first and second label (type and presence); ii) partial agreement (PA), at least one annotator disagreed on one strategy, though all 5 agreed on the other (e.g. all annotators agree on the first label, but no agreement is reached on the second); iii) disagreement (D), at least one disagreement for both labels.
In most cases (about 85%) the annotators agreed on at least one of the strategies detected. More specifically, A was reached in about 35% of the utterances, while PA in 50% of the cases. D was registered only for 15% of the utterances. The confusion matrix reported below shows more detailed information about the single strategies. Data reported in the matrix are mean percentages of values of the 10 pairs of annotators: label-by-label agreement was first calculated for each pair of annotators and then mean values for all the pairs were extracted and plotted in the matrix to check for which strategies reported, on average, the highest levels of agreement or disagreement across the annotators (Figure 1).
It is clear from the matrix that most cases of disagreement refer to get from genre. More generally, the matrix shows that among the cases of disagreement, the annotators failed to agree on the assignment of labels relative to global and local requesting strategies, which were often annotated as not representing a specific strategy at all. A sounder measure for the agreement (Fleiss’ Kappa) was calculated for those utterance in which all annotators agreed to assign only one label, which amount to about 1/3 of the total of the utterances4. The Fleiss’ Kappa value obtained for these annotation is 0.887, indicating an overall high agreement among the 5 annotators. The inspection of the score obtained for each specific label shows that while all strategies were detected with a high level of agreement, low values are registered for the category get from genre (Kappa = 0.247)
4The measure was not calculated for the whole data set because of the absence of a stable criterion for ordering strategies in case two were present (see section 3).
This section presents a description of the strategies employed by Seekers in the subset that we analyzed. The data reported here refer to those utterances in which all 5 annotators agreed on the type of strategy detected.
The most frequent strategy is the expression of personal opinions, which alone accounts for almost 50% of the total of the strategies. Of these, the great majority (around 90%) is represented by the strategy ’personal opinion pos’. The strategy ’personal experience’ is also quite frequent, amounting to around 20% of the strategies; among these, the expression of absence of experience (ex. No, I haven’t seen that movie) is more frequent, accounting for more that 60%. Recommendation requests account for 10% of the strategies, while the remainder is made up of those strategies aiming at either collecting specific information about a movie (i.e., get info) or eliciting a title given a specific preference (i.e., get movie). Of the former set of strategies, the information that is more frequently asked concerns the plot of the movie, while for the latter, Seekers appear to be most interested in the release date. Although annotated data about the Seekers’ turns are referred to a small subset of the whole corpus, it is possible to draw some preliminary strategies on the co-construction of the dialogue by the two participants, by considering the by-turn distribution of the strategies in both participants. As for the Seeker, the different strategies employed are not evenly distributed across the dialogue turns, as shown in Fig.2. The plot shows that recommendation requests are almost the only strategy employed at the beginning of the dialogue, after which their occurrence drops dramatically. On the contrary, the occurrence of get info and get movie increase as the dialogue unfolds. Personal opinion and experience, on the other hand, are more evenly distributed, with a drop of their occurrence in the median turns. As for the Recommender, the by-turn distribution of strategies is shown in Fig.3.
The plot shows that, on the Recommender side, the use of the strategy offer help mirrors the use by the Seeker of a request for recommendation, being employed almost exclusively in the first turn. More generally, the first part of the dialogue is characterized by inquiries, by the Recommender to the Seeker, about his/her opinions and experiences. While the use of these strategies decreases as the dialogue unfolds, strategies aimed at overcoming conflicts (e.g. preference confirmation) or persuading/informing (e.g. encouragement or similarity) are more frequent in the second half of the dialogue. This is mirrored, on the Seeker side, by the use of strategies linked to personal opinions/experiences and global and local requesting strategies. 5
This work is supported by the idea that studying
communication strategies used by human
speakers is fundamental to improve the performances
of dialogue systems. This was already supported
by Hayati and her colleagues (2020) who
analyzed the Recommenders’ sociable strategies in
recommendation dialogues to develop successful
and persuasive recommendation dialogue systems.
However, considering the cooperative nature of
dialogues, we argue that annotating the Seeker’s
move may be pivotal in the training phase of
recommendation dialogue systems. Hence, we
propose an annotation scheme for the Seeker’s
utterances that is coherent with the annotation of
Recommender’s utterances. Considering the Seeker’s
role and main moves, we have drawn four
categories: recommendation requests, global
requesting strategies, giving preference strategies and
local requesting strategies. Results on the
reliability of the annotation scheme show that the
agreement between the 5 annotators ranges from
substantial to almost perfect
Concerning the general distribution of the
Seekers’ strategies, positive personal opinion and
nonpresent personal experience seem to be more
frequent than the global and local requesting
strategies. The strategies distribution along with the
dialogue turns, on the other hand, shows that the
ifrst turns are mainly characterized by the
occurrence of recommendation requests, reflecting
the Recommender’s strategy of offering help. In
the middle of the conversation, requests for
getting information or movie titles increase together
with personal opinion and personal experience,
even if the latter seems to be more equally
distributed. This distribution could reflect the
fundamental role of the Seeker in modelling the
conversation. In the first phase of exploration the
Seekers’ personal opinions are explicitly elicited
by the Recommenders’ inquiries. Instead, in the
exploitation phase, the Seeker could also provide
soft evidence of their preferences, which may be
used by the Recommender to help the Seeker
ifnd a suitable item. This attitude is very
common in human-human dialogue with respect to the
human-machine interaction, since it follows the
principles of cooperative dialogue
The authors would like to thank Franco Cutugno, who, within his interdisciplinary course on Natural Language Processing, provided a fruitful environment for linguists and computer scientist to join their competences and inspired this work. Also, the authors would like to thank Antonio Origlia for the always ready advice, constructive discussion and his insightful comments on this work.