This study is set in the framework of taskoriented conversational agents in which dialogue management is obtained via Reinforcement Learning. The aim is to explore the possibility to overcome the typical end-to-end training approach through the integration of a quantitative model developed in the field of persuasion psychology. Such integration is expected to accelerate the training phase and improve the quality of the dialogue obtained. In this way, the resulting agent would take advantage of some subtle psychological aspects of the interaction that would be difficult to elicit via end-to-end training. We propose a theoretical architecture in which the psychological model above is translated into a probabilistic predictor and then integrated in the reinforcement learning process, intended in its partially observable variant. The experimental validation of the architecture proposed is currently ongoing.
A typical conversational agent has a multi-stage
architecture: spoken language, written language
and dialogue management, see
Given that the task considered involves psychosocial aspects that are difficult to program directly, the idea of achieving an effective dialogue
Copyright c 2019 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
manager via machine learning techniques,
reinforcement learning (RL) in particular, may seem
attractive. At present, many RL-based approaches
involve training an agent end-to-end from a dataset
of recorded dialogues, see for instance
Our study relies on a quantitative, causal model
of human behavior being studied in the field of
social psychology
Three relevant psychosocial antecedents of behaviour change are the following: Self-Efficacy (the individual perception of being able to eat healthy), Attitude (the individual evaluation of the pros and cons) and Intention Change (the individual willingness of adhering to a healthy diet). These psychosocial dimensions cannot be directly observed and need to be measured as latent variables. To this purpose, questionnaires are used, each composed by a set of questions or items (i.e. observed variables). Self-Efficacy is measured with 8 items, each associated to a set of answers ranging from "not at all confident" (1) to "extremely confident" (7). Attitude is assessed through 8 items associated to a differential scale ranging from 1 to 7 (the higher the score, the more positive the attitude). Intention Change is measured with three items on a Likert scale, ranging from 1 (“definitely do not”) to 7 (“definitely do”). See Carfora et el. (2019).
In our study, the psychosocial model was
assessed experimentally on a group of volunteers.
Each participant was first proposed a
questionnaire (Time 1 – T1) for measuring Self-Efficacy,
Attitude and Intention Change. In a subsequent
phase (i.e. message intervention), participants
were randomly assigned to one of four groups,
each receiving a different type of persuasive
message: gain (i.e. positive behavior leads to
positive outcomes), non-gain (negative behavior
prevents positive outcomes), loss (negative behavior
leads to negative outcomes) and non-loss
(positive behavior prevents negative outcomes)
The overall model is described by the
Structural Equation Model
Note that latent variables in a SEM have different roles: in this case gain/non-gain/loss/nonloss messages are independent variables, Intention Change is a dependent variable, Attitude is a mediator of the relationship between the independent and the dependent variables, and Self-Efficacy is a moderator, namely, it explains the intensity ot the relation it points at. Intention Change was measures at both T1 and T2, Attitude was measured at both T1 and T2, and Self-Efficacy was measured at T1 only. Note that the time transversality (i.e. T1 ! T2) is implicit in the SEM depiction above.
Once the SEM is defined, we aim to translate
it into a probabilistic model, so as to obtain the
probability distributions needed for the learning
process. We resort to a graphical model, and in
particular to a Bayesian Network
N P (X1; : : : ; XN ) = Y P (Xi j parents(Xi)); i=1 where X1; : : : ; XN are the random variables in the model and parents(Xi) indicate all the nodes having an edge oriented towards Xi.
In the case at hand, a temporal description of
the model, accounting for the time steps T1 and
T2, is necessary as well. For this purpose, we use
a Dynamic Bayesian Network
Notice that the messages are only significant at T2, as they have not been sent yet at T1. We gathered message in the one node Message Type, assuming it can take four, mutually exclusive values. The mediator Attitude is measured at both time steps while the moderator Self-Efficacy is constant over time, as suggested in Section 2. Intention Change has relevance at T2 only since, as we will mention in Section 5, it will be used to estimate a reward function once the final time step is reached. 4
The collected data are as follows. The analysis was conducted on 442 interactants, divided in four groups, each one receiving a different type of messages1. The answers to the items of the questionnaire always had a range of 7 values. However, this induces a combinatory esplosion, making it impossible to cover all the subspaces (78 = 5:764:801 different combinations for Attitude, for instance). We thus decide to aggregate: low := 1The original study included also a control group, which we do not consider here. (1 to 2); medium := (3 to 5); high := (6 to 7).
Our aim is to learn the Joint Probability Distribution (JPD) of our model, as that would make us able to answer, through marginalizations and conditional probabilities, any query about the model itself. The conditional probability distributions to be learnt in the case in point are then the following:
P (Item Ai), for i = 1; : : : ; 8; P (Item SEi), for i = 1; : : : ; 8; P (Message Type); P (Attitude T 1 j Item Ai; i = 1; : : : ; 8); P (Self-Efficacy j Item SEi; i = 1; : : : ; 8); P (Attitude T 2 j Item Ai; i = 1; : : : ; 8; Message Type; Self-Efficacy);
P (Intention Change j Attitude T 2;
Self-Efficacy).
The first three can be easily inferred from the raw
data as relative frequencies. As for the following
four, even aggregating the 7 values as mentioned,
a huge amount of data would still be necessary
(38 24 3 = 314:928 subspaces for Attitude T2, for
instance). As conducting a psychological study on
that amount of people would not be feasible, we
address the issue with an appropriate choice of the
method. To allow using Maximum Likelihood
Estimation (MLE) to learn the BN, we resort to the
Noisy-OR approximation
The translation into a tool to be used for
reinforcement learning is obtained in the terms of Markov
Decision Processes (MDPs), see
Roughly speaking, in a MDP there is a finite
number of situations or states of the environment,
at each of which the agent is supposed to select an
action to take, thus inducing a state transition and
obtaining a reward. The objective is to find a
policy determining the sequence of actions that
generates the maximum possible cumulative reward,
over time. However, due to the presence of latents,
in our case the agent is not able to have complete
knowledge about the state of the environment. In
such a situation, the agent must build its own
estimate about the current state based on the memory
of past actions and observations. This entails using
a variant of the MDPs, that is Partially Observable
Markov Decision Processes
A := factionsg = fask A1; : : : ; ask A8g [ fask SE1; : : : ; ask SE8g [ fG; N G; L; N Lg, where Ai denotes the question for Item Ai, SEi denotes the question for Item SEi and G; N G; L; N L denote the action of sending Gain, Non-gain, Loss and Non-loss messages respectively;
:= fobservationsg = fItem A1; : : : ; Item A8; Item SE1; : : : ; Item SE8g.
Starting from an unknown initial state s0 (often taken to be uniform over S, as no information is available), the agent takes an action a0, that brings it, at time step 1, to state s1, unknown as well. There, an observation o1 is made.
The process is then repeated over time, until a goal state of some kind has been reached. Hence, we can define the history as an ordered succession of actions and observations:
ht := fa0; o1; : : : ; at 1; otg ; h0 = ;:
As at all steps there is uncertainty about the actual state, a crucial role is played by the agent’s estimate about the state of the environment, i.e. by the belief state. The agent’s belief at time step t, denoted as bt, is driven by its previous belief bt 1 and by the new information acquired, i.e. the action taken at 1 and observation made ot. We then have: bt+1(st+1) = P (st+1 j bt; at; ot+1): In the POMDP framework, the agent’s choices about how to behave are influenced by its belief state and by the history. Thus, we define the agent’s policy:
= (bt; ht); that we aim to optimize. To complete the picture, we define the following functions to describe the model evolution in time (the notation 0 indicates a reference to the subsequent time step):
state-transition function: T : (s; a) 7! P (s0 j s; a) := T (s0; s; a);
observation function: O : (s; a) 7! P (o0 j a; s0) := O(o0; a; s0);
reward function:
R : (s; a) 7! E [r0 j s; a] := R(s; a).
These functions can be easily adapted to the specifics of the case at hand. It can be seen that, once the JPD derived from the DBN is completely specified, the reward is deterministic. In particular, it is computed by evaluating the changes in the values for the latent Intention Change.
As we are interested in finding an optimal policy, we now need to evaluate the goodness of each state when following a given policy. As there is no certainty about the states, we define the value function as a weighted average over the possible belief states:
V (bt; ht) :=
X bt(st)V (st; bt; ht); st where V (st; bt; ht) is the state value function. The latter depends on the expected reward (and on a discount factor 2 [0; 1] stating the preference for fast solutions):
V (st; bt; ht) :=R(st; (bt; ht)) +
X T (st+1;st; (bt; ht)) st+1 X O(ot+1; (bt; ht);st+1)V (st+1; bt+1; ht+1): ot+1 Finally, we define the target of our seek, namely the optimal value function and the related optimal policy, as: (V (bt; ht) := max V (bt; ht);
(bt; ht) := argmax V (bt; ht): It can be shown that the optimal value function in a POMDP is always piecewise linear and convex, as exemplified in Figure 3. In other words, the optimal policy (in bold in Figure 3) combines different policies depending on their belief state values.
The next step is to use the POMDP to detect the optimal policy, that is the sequence of questions to ask to the interactant, in order to draw her/his profile, hence the message to send, which maximizes the effectiveness of the interaction. To this end, the contribution of the DBN is fundamental. From the JPD associated, in fact, we construct the probability distributions necessary to define the functions T , O, R that compose the value function. 6
It is evident from Figure 4, describing the full expansion of the policy tree for the case in point, that the computational effort and power required for a brute-force exploration of all possible combinations is unaffordable.
Among all the policies that can be considered, we want to select the optimal ones, thus avoiding coinsidering policies that are always underperforming. In other words, with reference to Figure 3, we want to find Vp1 , Vp2 , Vp3 among those of all possible policies, and use them to identify the optimal policy V .
To accomplish this, we select the Monte Carlo
Tree Search (MCTS) approach, see
In this work we explored the possibility of harnessing a complete and experimentally assessed SEM, developed in the field of persuasion psychology, as the basis for the reinforcement learning of a dialogue manager that drives a conversational agent whose task is inducing healthier nutritional habits in the interactant. The fundamental component of the method proposed is a DBN, which is derived from the SEM above and acts like a predictor for the belief state value in a POMDP.
The main expected advantage is that, by doing so, the RL agent will not need a time-consuming period of training, possibly requiring the involvement of human interactants, but can be trained ‘in house’ – at least at the beginning – and be released in production at a later stage, once a first effective strategy has been achieved through the DBN. Such method still requires an experimental validation, which is the current objective of our working group.
The authors are grateful to Cristiano Chesi of IUSS Pavia for his revision of an earlier version of the paper and his precious remarks. We also acknowledge the fundamental help given by Rebecca Rastelli, during her collaboration to this research.