We address the problem of designing a conversational avatar capable of a sequence of casual conversations with older adults. Users at risk of loneliness, social anxiety or a sense of ennui may benefit from practicing such conversations in private, at their convenience. We describe an automatic spoken dialogue manager for LISSA, an on-screen virtual agent that can keep older users involved in conversations over several sessions, each lasting 10-20 minutes. The idea behind LISSA is to improve users' communication skills by providing feedback on their non-verbal behavior at certain points in the course of the conversations. In this paper, we analyze the dialogues collected from the first session between LISSA and each of 8 participants. We examine the quality of the conversations by comparing the transcripts with those collected in a WOZ setting. LISSA's contributions to the conversations were judged by research assistants who rated the extent to which the contributions were “natural", “on track", “encouraging", “understanding", “relevant", and “polite". The results show that the automatic dialogue manager was able to handle conversation with the users smoothly and naturally.
• Human-centered computing → Human computer interaction (HCI); HCI design and evaluation methods; • Artificial intelligence → Natural language processing.
IUI Workshops’19, March 20, 2019, Los Angeles, USA © Copyright 2019 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. 1
The population of senior adults is growing, in part as a result of
advances in healthcare. According to United Nations studies on
world population, the number of people aged 60 years and over is
predicted to rise from 962 million in 2017 to 2.1 billion in 2050 [
However, learning to use digital technology can be dificult and
frustrating, especially for older people, and this negatively impacts
acceptance and use of such technology [
Spoken language interaction can benefit older people in many diferent ways, including the following: • Dialogue systems can support people with their health care needs. Such systems can collect and track health information from older individuals and provide comments and advice to improve user wellness. They can also remind people about their medications and doctor appointments. • Speech-based systems can help users feel less lonely by providing casual chat as well as information on news, events, activities, etc. • They can also provide entertainment such as games, jokes, or music for greater enjoyment of spare time. • Dialogue systems can help older people improve some skills.
For instance, cognitive games can help users maintain mental
acuity. They can also enable practice of communication skills,
as may be desired by seniors experiencing social isolation
after loss of connections to close family and friends [
All of the above types of systems require a powerful dialogue manager, allowing for smooth and natural conversations with users, and taking account of the special needs and limitations of the target user population.
In this paper we describe how we adapted the dialogue
capabilities of a virtual agent, LISSA, for interaction with older adults
who may wish to improve their social communication skills. As
the centerpiece of the "Aging and Engaging Program" (AEP) [
The conversation is in three segments, and at the two break points the system ofers some advice to users on the areas they may need to work on and how to improve in those areas. In addition, a summary of weak and strong aspects of the user’s behavior is presented at the end of the conversation.
The system was designed in close collaboration with an expert advisory panel of professionals (at our afiliated medical research center) working with geriatric patients. A single-session study was conducted with 25 participants where the virtual agent’s contributions to the dialogue were selected by a human wizard. The nonverbal feedback showed an accuracy of 72% on average, in relation to a human-provided corpus of annotations, treated as ground truth. Also, a user survey showed that users found the program useful and easy to use. As the next step, we deployed a fully automatic system where users can interact with the avatar at home. We planned for a 10-session intervention, where in each session the participants have 10-20 minutes of interaction with the avatar. The ifrst and last sessions were held in a lab, where experts collected information on users and rated their communication skills. The remaining sessions were self-initiated by the users at home. We ran the study with 9 participants interacting with the avatar and 10 participants in a control group.
Our framework for multi-topic dialogue management was
introduced in [
We have initiated further data analyses, and three aspects relating to conversation quality are the following. First, we will be looking at the level of verbosity for diferent users in diferent sessions, to determine its dependence on the user’s personality and on the topic under discussion. Second, we will measure self-disclosure and study its correlation with user mood and personality. Third, we will track users’ sentiment over the course of conversations to determine its variability over time, and its dependence on dialogue topics and user personality.
The main contributions of the work reported here are • demonstration of the flexibility of our approach to dialogue design and management, as evidenced by rapid adaptation and scaling up of previous versions of LISSA’s repertoire, suitable for the Aging and Engaging domain; our approach uses modifiable dialogue schemas, hierarchical pattern transductions, and context-independent “gist clause" interpretations of utterances; • integration of an automated dialogue system for multi-topic conversations into a virtual human (LISSA), designed for conversation practice and capable of observing and providing feedback to the user; • an initial demonstration showing that the automated dialogue manager functions as efectively for a sampling of users as the prior wizard-guided version; this signifies an advance of the state of the art in building conversational practice systems for older users, with no constraints on users’ verbal inputs (apart from the conversational context determined by LISSA’s questions).
The rest of this paper is organized as follows. We first comment on extant work on spoken dialogue systems that are designed to help older adults; then we introduce our virtual agent, LISSA, and briefly explain the feedback system. We then discuss the dialogue manager structure and the content design for multi-session interactions with older users. In the next section, we evaluate LISSA’s ifrst-session interactions with users, referred to above. We then mention ongoing and future work, and finally summarize our results and state our conclusions. 2
As noted in the the introduction, thanks in part to recent improvements in ASR and TTS systems, the use of conversational systems to help technically unskilled older adults has become more feasible.
According to [
[
Virtual companions have also been proposed for palliative care
– helping people with terminal illnesses reduce stress by steering
them towards topics such as spirituality, or their life story. In an
exploratory study with older adults reported in [
Other systems have focused on assisting older adults with their
specific daily needs. For instance [
Another approach that has proved efective in ameliorating
loneliness and depression in older people is reminiscence therapy [
Robotic pets are another interesting technology ofered for
alleviating loneliness in older people; such pets react to user speech
and petting by producing sounds and eye and body movements.
Studies [
In almost all applications of virtual agents, the system needs
to somehow motivate users to engage conversationally with it.
(Robotic pets are an exception.) An interesting study of people’s
willingness to disclose themselves in interacting with an open-domain
voice-based chatbot is presented in [
The LISSA (Live Interactive Social Skills Assistance) virtual agent is a human-like avatar (Figure 1) intended to become ubiquitously available for helping people practice their social skills. LISSA can engage users in conversation and provide both real-time and postsession feedback. 3.1
To help users improve their communication skills, LISSA provides
feedback on their nonverbal behavior, including eye contact,
smiling, speaking volume, as well as one verbal feature: the emotional
valence of the word content. In the original implementation of LISSA,
feedback was presented in real time through screen icons that turn
from green to red, indicating that the user should improve the
corresponding nonverbal behavior. Feedback was also presented at the
end of each interaction, via charts and figures providing
information on the user’s performance. However, as both real-time visual
feedback and charts could be cognitively demanding for older users,
the Aging and Engaging version of LISSA ofers feedback through
speech and text during the conversation. Furthermore, at the end of
each conversation session, the current system generates a
speechand text-based summary of the feedback provided during the
dialogue, mentioning users’ strong areas, weaknesses and suggestions
for improvement. More details on how feedback is generated and
presented can be found in [
Two versions of LISSA were previously designed based on the
needs and limitations of the target users. LISSA showed potential
for impacting college students’ communication skills when it was
tested as a speed-dating partner in a WOZ setting [
Social communication deficits among older adults can cause significant problems such as social isolation and depression. Technological interventions are now seen as having potential even for older adults, as computers, cellphones and tablets have become more accessible and easier to use. Our most recent version of LISSA is designed to interact with older users over several sessions, with the goal of helping them improve their communication skills. The system ofers low technological complexity and the feedback is presented in a format imposing minimum cognitive load. The interface was designed with the assistance of experts working with geriatric patients, focusing on 12 older adults.
The program is designed so that each user becomes engaged in casual talk with the system, where two or three times during the conversation, the system suspends the conversation and comments on the user’s weaknesses and strengths, and suggests ways of improving on the weaker aspects. Also at the end of the conversation, the system briefs the user on what they went through and summarizes the previously ofered advice.
To evaluate the program’s potential efectiveness, we conducted
a one-shot study with 25 older adults (all more than 60 years old,
average age 67, 25% male), using a WOZ setting for handling the
conversation. The results showed that the participants’ speaking
times in response to questions, as well as the amount of positive
feedback, increased gradually in the course of conversation. At the
same time, participants found the feedback useful, easy to interpret,
and fairly accurate, and expressed their interest in using the system
at home. More details on the study can be found in [
Based on the outcome of the WOZ studies, we designed a multisession study where participants can interact with the system in their homes at their convenience. The purpose is to study the acceptability of a longer-term interaction and its impact on users’ communication skills. The design enables each participant to engage in ten conversation sessions with the avatar. The first and the last sessions are held in the lab, where users fill out surveys and are evaluated for their communication skills by experts. The rest of the sessions are held in users’ homes where they need to have access to a laptop or a personal computer. Each interaction with LISSA consists of three subsessions where LISSA leads a natural casual conversation with the user. Users receive brief feedback on their behavior, during two breaks between subsessions. At the end, users are provided with a behavioral summary, and some suggestions on how they can improve their communication skills. In our studies so far, nine participants used the avatar for conversation practice, while ten participants were assigned to a control group.
We collected the conversation transcripts from the participants assigned to LISSA in order to assess the quality of the dialogues. We discuss the evaluation process and results in later sections.
In the next section, we provide some details on how the system manages an engaging conversation with the user, along with an evaluation of the quality of the conversations gathered. Figure 2 shows a portion of a conversation between a user and LISSA in the ifrst session of interaction. 4
In order to handle a smooth natural conversation on everyday topics with a user, the dialogue manager (DM) needs to follow a coherent plan around a topic. It needs to extract essential information even from relatively wordy inputs, and produce relevant comments demonstrating its understanding of the user’s turns. The DM also needs a way to respond to user questions, and to update its plan if this is necessitated by a user input; for instance, if a planned query to the user has already been answered by some part of a user’s input, that query should be skipped. The LISSA DM is capable of such plan edits, as well as expansion of steps into subplans. We now explain the DM structure along with the content we provided for the Aging and Engaging program. 4.1
Our approach to dialogue management is based on the hypothesis
that human cognition and behavior rely to a great extent on
dynamically modifiable schemas [
Accordingly the dialogue manager we have designed follows a flexible, modifiable dialogue schema, specifying a sequence of intended and expected interactions with the user, subject to change as the interaction proceeds. The sequence of formal assertions in the body of the schema express either actions intended by the agent, or inputs expected from the user. These events are instantiated in the course of the conversation. This is a simple matter for explicitly specified utterances by the agent, but the expected inputs from the user are usually specified very abstractly, and become particularized as a result of input interpretation. Schemas can be extended allow for specification of participant types, preconditions, concurrent conditions, partial action/event ordering, conditionals, and iteration. These more general features are still in the early stages of implementation, but so far have not been needed for the present application.
Based on the main conversational schema, LISSA leads the main lfow of the conversation by asking the user various questions. Following each user’s response, LISSA might show one of the following behaviors: • Making a relevant comment on the user’s input; • Responding to a question, if the user asked one (typically, at the end of the input); • Instantiating a subdialogue, if the user switched to an “oftrack" topic (an unexpected question may have this efect as well.)
A dialogue segment between LISSA and one user in the Aging program can be seen in Figure 2.
Throughout the conversation, the replies of the user are transduced into simple, explicit, largely context-independent English sentences. We call these gist-clauses, and they provide much of the power of our approach. The DM interprets each user’s input in the context of LISSA’s previous question, using this question to select pattern transduction hierarchies relevant to interpreting the user’s response. It then applies the rules in the selected hierarchies to derive one or more gist-clauses from the user’s input. The transduction trees are designed so that we extract as much information as we can from a user input. The terminal nodes in the transduction trees are templates that can output gist-clauses. As an example of a gist clause, if LISSA asks "Have you seen the new Star Wars movie?" and the user answers "Yes, I have.", the output of the choice tree for interpreting this reply would be something like "I have seen the new Star Wars movie."
The system then applies hierarchical pattern transduction
methods to the gist-clauses derived from the user’s input to generate
a specific verbal reaction. In particular, each of the gist clauses is
matched against a set of relevant gist clause forms in a
transduction tree, and when a match is found, a corresponding reaction is
constructed as output. This reaction could refer to specific phrases
mentioned by the user or to topics abstracted from them, making
the reaction more meaningful. Figure 3 shows an overview of the
dialogue manager. The most important stratagem in our approach
is the use of questions asked by LISSA as context for transducing
a gist-clause interpretation of the user’s answer, and in turn, the
use of those gist-clause interpretations as context for transducing
a meaningful reaction by LISSA to the user’s answer (or multiple
reactions, for instance if the user concludes an answer with a
reciprocal question). The interesting point is that in gist-clause form,
question-answer-reaction triples are virtually independent of the
conversational context, and this leads to “portability" of the
infrastructure for many such triples from one kind of (casual) dialogue
to another. More details can be found in [
To enable a smooth, meaningful conversation with the user, the pattern transduction trees need to be designed carefully and equipped with appropriate patterns at the nodes. For the Aging and Engaging program we needed ten interactions between LISSA and each user. Each interaction consists of three subsessions, and each subsession contains 3-5 questions to the user, along with sporadic self-disclosures by LISSA. In these disclosures LISSA presents herself as a 65-year old widow who moved to the city a few years ago to live with her daughter, and relates relevant information or experiences of hers. LISSA leads the conversation by opening topics, stating some personal thoughts and encouraging user inputs by asking questions. Upon receiving such inputs, LISSA makes relevant comments and expresses her thoughts and emotions.
LISSA’s contributions to the dialogues, including the choice of the character and her background, were meticulously designed by four trained research assistants (RAs), in extensive consultation with gerontologists with expertise in interventions. Details of her character, interests, beliefs and thoughts were designed and inserted into the interaction along the way. The contents of the DM’s transduction trees were designed based on the suggestions provided by our expert collaborators, as well as on the experience gathered from previous LISSA-user interactions in the WOZ study.
Since we planned for ten interactions between LISSA and each
user, we collected a list of 30 topics that could be of interest to
our target group. The gerontological experts divided the topics
into three groups based on their emotional intensity or degree of
intimacy: easy, medium, and hard. All the topics were ones that
older people would encounter in their daily lives. While the “easy"
ones involve little personal disclosure or intimacy and are typical of
conversations they might have at a senior center with new people
or at a dining hall in a senior living community, the harder
conversations contain more emotionally evocative topics. Table 1 shows
a list of topics in each group. Research shows that the social bond
between users and a virtual agent can increase the efectiveness of
the virtual agent in diferent tasks such as tutoring [
It is noteworthy that the design and implementation of the infrastructure for the 30 topical dialogues represented an order-ofmagnitude scale-up from the previous dialogue designs for speeddating and for interaction with autistic teens. This testifies to the efectiveness of our use of schemas, transduction hierarchies, and interpretations in the form of gist clauses for rapid dialogue implementation. The individual topics were mostly implemented by three of the authors, including two undergraduates, and typically took about half a day or a day to complete. 5
To evaluate the eficacy of the DM system, we designed a procedure to assess the quality of the conversations from a third-person point of view. We compared conversation transcripts from the initial WOZ-based study with transcripts from the fully automatic dialogue system. To do this, 16 conversation transcripts were chosen, each consisting of a single session with three subtopics. 8 transcripts (the first 8 that became available) were pulled from the WOZ study, and the other 8 were pulled from the automated sessions. Both sets of 8 were taken from the first-day interaction between users and LISSA. The topics covered in both sets of interactions were very much alike, so that it was fair to compare them. These transcripts were de-identified, assigned random numerical labels, and cleaned so that the transcripts followed a uniform format.
The transcripts were then assigned randomly to 6 RAs who were blind to the study condition, with each RA being responsible for assessing 8 transcripts. Each RA was tasked to rate each assigned transcript on six criteria related to the quality of the conversation, which are shown in Table 2. Each criterion was independently rated on a Likert scale from 1 (not at all) to 5 (completely). No other guidance was provided to the RAs; rather they were directed to make natural judgments for each criterion based on the transcript alone. The rating results showed good internal consistency (Cronbach’s Emotional intensity
Topics alpha = 0.89). Ratings across RAs were then averaged for each transcript to represent the consensus score for that transcript. The average rating and one standard deviation for the WOZ study compared to the automated LISSA system are shown for each criterion in Figure 5.
As can be seen in the evaluation results, the automated LISSA system was able to achieve fairly high ratings for each of the criteria. This suggests that the system was able to hold conversations that were of good overall quality. Compared to the WOZ study, the automated system was able to achieve slightly higher ratings, although none of the gains were suficiently large to be statistically significant. Nonetheless, these results suggest that the automated LISSA system is capable of producing conversations that are approximately of the same quality as the human-operated (i.e., WOZ) system.
The largest diference between the WOZ ratings and the automated LISSA ratings was in the “On Track" metric, where the average LISSA rating was half a point higher than the WOZ average. Again, this improvement is not statistically significant, given the variance and low number of samples, but it is a result consistent with the overall design of the system. Since the DM was designed to follow a well-established conversation plan, while also being equipped with some mechanisms to handle of-topic input and then return to the main track, we expected the system to be particularly efective at remaining on track as compared to the human-operated systems.
The good scores we achieved for the “Encouraging" and “Polite"
metrics strike us as significant. They suggest that the guidance
provided by psychiatric experts on the design of the topics,
questions, and comments was advantageous. Additionally, our strategy
of including self-disclosure and a backstory in the LISSA dialogues
was probably a factor in the successful performance of our system.
As mentioned in the Related Work section, self-disclosure by an
avatar tends to encourage self-disclosure by the users [
The results discussed above indicate that LISSA’s dialogue system is functioning well. Confirmation of LISSA’s potential for improving users’ communication skills will require additional study, including: transcription and analysis of the ASR files from the at-home sessions; expert evaluation of users’ communication behaviors in the initial and final lab interviews and surveys that were conducted; and additional, longer-range studies with participants in the target population. However, we have been able to obtain some insights into people’s feelings about their interaction with LISSA after completing all 10 sessions.
The surveys administered at the end asked participants numerous questions about their assessment of LISSA, including various aspects of the nonverbal feedback they had received throughout the sessions. A subset of questions of interest here concerned the usability of the system. In particular, participants were asked to rate the following statements: • I thought the program was easy to use. • I would imagine that most people would learn to use this program very quickly. • I felt very confident using the program. • Overall, I would rate the user-friendliness of this product as:...
Responses were given on a five-point scale ranging from "strongly disagree" (=1) to "strongly agree" (=5).
The results from nine participants can be seen in Table 3.
The results from nine participants showed an average score of 4.33 (sd = 0.67) for the first statement, meaning that they found the system easy to use. The average score on the third statement indicates that participants felt no anxiety in handling the interaction at home on their own; and along similar lines, the “user friendliness" rating of 4.56 (sd = 0.50) suggests that users had no real dificulties in dealing with the system.
Criterion WOZ Average
WOZ SD
LISSA Average
LISSA SD
We also provided some qualitative questions about users’ opinions about the system and their experience with it. One question asked what they liked about the system. Some dialogue-related responses were the following: - “having someone to talk to, even though it was a computer" - “starting out with a topic that fit in well with lifestyle that allowed responses and conversation, and providing feedback about responses" - “I liked her personality and her calmness" - “questions were relatable and answerable" - “conversational topics were fitting, though at times seemed more elementary" - “simple and straightforward" - “interfacing with LISSA was like having someone in my home visiting" - “the topics seemed general enough and enjoyed talking about them; helped with self-reflection and have deeper conversations rather than just shallow, surface-level conversations"
When asked what they didn’t like about the system, few users registered any complaints about the dialogue per se. One participant wanted longer conversations, commenting: “a bit more time [should be] devoted to the conversations - ask 4-5 questions (rather than just 3) to help participants feel more comfortable with the conversation". One commented that “having humans instead of computers may be more motivating" – a salient reminder that computers remain tools, not surrogate humans. 7
We plan to use the conversational data collected from multi-session interactions between LISSA and older users to gain a deeper understanding of certain key aspects of these interactions. These aspects include the following: • Verbosity: We plan to study the level of verbosity of participants from session to session, and also the variability of verbosity among participants. Questions of interest are whether verbosity is dependent on the user’s personality, the topic and questions asked, and other factors; and also whether verbosity is correlated with a positive view of the system. • Self-disclosure: Self-disclosure is considered a sign of conversational engagement and trust. We intend to use known techniques for measuring self-disclosure in a dialogue to assess whether the extent of self-disclosure is more dependent on the user’s personality or the inputs from the avatar; and, how self-disclosure might correlate with the user’s mood, social skills and evaluation of the system. • Sentiment: As this program targets older adults who are at risk of isolation and depression, we want to know more about users’ moods, based on the emotional valence of their inputs. We plan to track sentiment over the course of successive sessions, taking into account the context of the dialogue (e.g., the topic under discussion). We should then be able to detect any correlations between the users’ moods, and their evaluation of the system at various points. 8
We introduced the design and implementation of a spoken dialogue manager for handling multi-session interactions with older adults. The dialogue manager leads engaging conversations with users on various everyday and personal topics. Users receive feedback on their nonverbal behaviors including eye contact, smiling, and head motion, as well as emotional valence. The feedback is presented at two topical transition points within a conversation as well as at the end. The goal of the system is to help users improve their communication skills.
The DM was implemented based on a framework proposed in previous work, employing flexible schemas for planned and anticipated events, and hierarchical pattern transductions for deriving “gist clause" interpretations and responses. We prepared the dialogue manager for 10 interaction sessions, each covering 3 topics. Our DM framework allowed quite rapid development of the 30 topics. The content was adapted to the needs and limitations of older adults, based on collaboration with an expert advisory panel of professionals at our afiliated medical research center. A broad spectrum of topics was established, and the sessions were designed to progress from emotionally undemanding ones to ones calling for greater emotional disclosure.
We ran a study including 8 participants interacting with the virtual agent. To evaluate the quality of the conversations, we compared the conversations of these participants in the initial in-lab sessions with 8 such conversations where LISSA outputs were selected by a human wizard. The transcripts were randomly assigned to 6 RAs, who rated the transcripts based on 6 features of high-quality conversation. The results showed high ratings for the automatic system, even slightly better than wizard-moderated interactions. The overall usability evaluation of the system by users who completed the series of sessions shows that users found the system easy to use and rated it as definitely user-friendly. Further studies of interaction quality and efects on users, based (among other data) on ASR transcripts of the at-home sessions, are in progress.
The work was supported by DARPA CwC subcontract W911NF-151-0542. We would also like to thank Professor Paul Duberstein and Shuwen Zhang for their helpful contributions.