=Paper=
{{Paper
|id=Vol-1573/Paper_6_BCSS2016
|storemode=property
|title=How Persuasive is a Virtual Coach in Terms of Promoting Web-based Intervention User Engagement? -
Exploring a Potential Match between User Support Needs and a Virtual Coach’s Motivational Capabilities
|pdfUrl=https://ceur-ws.org/Vol-1573/Paper_6_BCSS2016.pdf
|volume=Vol-1573
|authors=Mark Scholten,Saskia Kelders,Lisette van Gemert-Pijnen
|dblpUrl=https://dblp.org/rec/conf/persuasive/ScholtenKG16
}}
==How Persuasive is a Virtual Coach in Terms of Promoting Web-based Intervention User Engagement? -
Exploring a Potential Match between User Support Needs and a Virtual Coach’s Motivational Capabilities==
https://ceur-ws.org/Vol-1573/Paper_6_BCSS2016.pdf
66 Fourth International Workshop on Behavior Change Support Systems (BCSS’16):
How Persuasive is a Virtual Coach in Terms of Promoting Web-based Intervention
User Engagement?
How Persuasive is a Virtual Coach in Terms of
Promoting Web-based Intervention User Engagement?
Exploring a Potential Match between User Support Needs and
a Virtual Coach’s Motivational Capabilities
Mark Scholten, Saskia Kelders, and Lisette van Gemert-Pijnen
Psychology, Health & Technology, University of Twente, Enschede, The Netherlands
{m.r.scholten, s.m.kelders, j.vangemert-pijnen}@utwente.nl
Abstract. A selection of web-based eHealth interventions have evolved from
innovative prototypes to evidence-based and clinically applied solutions for
mental diseases such as depression and anxiety. Open-access, self-guided types
of those solutions hold the promise of reaching and treating a large population
against reasonable costs.
However, a considerable factor that currently hinders the effectiveness of these
self-guided web-based interventions is the high level of attrition. The absence
of a human caregiver apparently has a negative effect on user adherence. Alt-
hough clear evidence on the rationale for non-adherence is currently missing,
recent studies indicate that user support could an important causal factor.
In this paper we describe the protocol of an exploratory user study which aims
to identify the persuasive features of virtual coaches in terms of user support.
Virtual coaches are anthropomorphized instances of software, designed to en-
gage and motivate users.
We will measure user experience, engagement, motivation and perceived per-
suasiveness during usage of the ‘Living to the Full’ intervention. We will com-
pare a condition with and without empathic virtual coach condition to explore
its supportive capabilities.
Keywords: eHealth, web-based interventions, embodied conversational agents,
virtual coaches, virtual humans, affective computing, behavior change support
system, BCSS, Intelligent Tutoring System, ITS
1 Introduction
Meta–analyses have demonstrated that web-based interventions for mental health
have become reasonably successful treatments against common mental health prob-
lems such as depression and anxiety [1,14,15]. However, it is a consistent finding that
human-supported web-based therapeutic interventions outperform self-guided inter-
ventions (in which there is no support from a human). The mere online, sometimes
remote presence of a human being, delivering informational support, emotional sup-
67 Fourth International Workshop on Behavior Change Support Systems (BCSS’16):
How Persuasive is a Virtual Coach in Terms of Promoting Web-based Intervention
User Engagement?
port or a therapeutic service results in significantly higher effect sizes [9]. In addition,
human-supported interventions achieve higher rates of adherence, that is more partic-
ipants use the intervention as intended, e.g. by completing all the lessons of an inter-
vention. [1,8,15]. Non-adherence is an important issue in web-based interventions for
mental health [21] and becomes an even bigger problem when evidence-based thera-
pies are deployed as free to access self-guided web-based therapeutic interventions
[20]. In these interventions, adherence, defined as the percentage of users who com-
plete all lessons, falls to a level as low as 1% [21] or even 0.5% [22].
The higher rates of adherence in human-supported interventions can be explained
in favor of therapists, who prove to do an effective job in motivating clients during
their change process. Interestingly enough, Talbot [12] describes in her meta-study
that a key converging finding is that the involvement of a professional support pro-
vider, a therapist, is not necessary. What is key is a minimal level of non-guiding
human contact. Irrespective of whether this type of contact is provided by a layperson
or a professional, it has equally large positive effects on intervention adherence.
Moreover, scheduling support can already have an effect of itself on treatment effec-
tiveness. A telephone contact scheduled at the start of the treatment to take place as
soon as a self-help book has been read, yields surprisingly large completion rates and
treatment outcomes [13].
This poses the question what this support is that is needed to achieve higher rates
of adherence and effectiveness. A study of Cavanagh and Millings [34] provides evi-
dence of built-in ‘common factors’ such an generating hope, empathy and warmth,
collaboration and feedback, that increase the effectiveness of interventions. However,
there is no common definition of the support or ‘common factors’ that should be in-
cluded in each intervention to be effective.
Furthermore, this opens up the opportunity to look into different ways of providing
this support. There have been studies into the role technology can play in increasing
adherence. Studies have shown positive effects of e.g. reminders, tailored advice and
updates of the interventions website on adherence [21,35,36]. The study of Kelders et
al. [8] has shown that technology in the form of dialogue support (supporting the in-
teraction between the user and the system) has an influence on adherence. In this way,
technology can be seen as persuasive: it tries to persuade the user to keep using the
intervention and to change their behavior according to what is learned within the in-
tervention.
Computers can be equipped with capabilities that can potentially enhance their per-
suasiveness as is shown in the field of affective computing and by using embodied
conversational agents.
Affective Computing as a term was coined in the seminal article of Picard [26]
who defined it as “computing that relates to, arises from, or influences emotion”.
Picard [26, 27] stated that computers should be equipped with at least the basic func-
tionality to notice and respond to emotions expressed by their users, in order to cater
for a more natural form of human-computer interaction.
Cassell [38] defined embodied conversational agents as human-like appearances
that have the same properties as humans in face-to-face conversation (see Figure 1 for
an example), including:
68 Fourth International Workshop on Behavior Change Support Systems (BCSS’16):
How Persuasive is a Virtual Coach in Terms of Promoting Web-based Intervention
User Engagement?
• The ability to recognize and respond to verbal and non-verbal input.
• The ability to generate verbal and non-verbal output.
• The ability to deal with conversational functions such as turn taking, feedback,
and repair mechanisms.
• The ability to give signals that indicate the state of the conversation, as well as to
contribute new propositions to the discourse.
Fig. 1. Example of an embodied conversational agent
Both developments create capabilities to enrich the dialog between computer and
user. This enriched dialog encompasses both verbal and non-verbal information and
aims to acknowledge the role of human emotion. From a coaching perspective, the
computer side of the dialog steers user emotion into productive directions.
Deploying an affective embodied conversational agent as an adjunctive virtual coach
to self-guided web-based interventions can possibly contribute to the need for support
as expressed by users. However, this poses questions such as:
• When should support be delivered by the virtual coach? Can and should recogni-
tion of non-productive user status trigger a supportive action from the virtual
coach?
• If so, what non-productive user states can the computer reliably distinguish?
• What empirical evidence consists for effective recognition and repair of non-
productive user states?
69 Fourth International Workshop on Behavior Change Support Systems (BCSS’16):
How Persuasive is a Virtual Coach in Terms of Promoting Web-based Intervention
User Engagement?
Research of non-productive user states so far has focused on frustration and bore-
dom. Concerning frustration, Klein et al. [39] conducted an experiment during which
the participant was deliberately frustrated during game play, after which the user was
affectively supported by a text-based agent. Their results showed that the users chose
to interact significantly longer when the system provided textual affective support
compared to the condition during which no affective support was given. Based on
these findings Klein et al. suggested that ‘computers are capable of alleviating strong,
negative emotions, even when they are the source of these emotions’. However, frus-
tration may not need remediation as argued by Mentis [40]. According to Mentis’,
frustration among users of information systems is only of concern if it is associated
with events that are outside of the user’s locus of control, such as a program bug. A
frustrating event of this nature interrupts the user’s cognitive flow and ambition to
fulfill his computerized tasks and is therefore detrimental to user adherence..
A second non-productive user state concerns boredom. Based on user state meas-
urements during online learning Baker et al. [41] conclude that boredom is both a
frequent and detrimental state with regards to learning. They propose that next to
frustration detection, boredom detection should trigger computerized motivational
interventions. D’Mello et al [42] point out that bored students have a high risk of
becoming frustrated. They describe predictors of poor learning and increased drop-out
rates such as pupils going off-task, zoning out and intentionally misusing (i.e. gam-
ing) the learning system. Therefore they state that Intelligent Tutoring Systems (ITS)
should include affective loop capabilities. D’Mello et al. [42] developed an ITS on
biology that aimed to promote engagement and learning by dynamically detecting and
responding to student’s boredom and disengagement. The system made use of an eye
tracker to monitor the student’s gaze patterns and identifies when the user is no longer
looking at the screen. If the longitude of the gaze away period exceeds a threshold, an
animated pedagogical agent displays a motivational intervention, such as “please pay
attention” in order to reengage the user. The gaze-reactive tutor resulted in signifi-
cantly more accurate responses to the sub-category of deep-reasoning questions.
However, overall, there were no significant learning gains. It appeared that there were
significant positive overall learning effects for high proficient learners and negative
overall learning effects for less gifted students. This effect shows that computer sup-
port delivery to users remains a challenging task with potentially counter-productive
effects.
2 Our Research Proposal
To study the opportunities for a virtual coach within a web-based intervention for
mental health, we will study user experience, engagement, motivation and perceived
persuasiveness during usage of an intervention with a virtual coach as an adjunct to
the existing web based intervention ‘Living to the Full’ [43]. We will use a between-
subjects design with two experimental conditions (empathic virtual coach and neutral
virtual coach) and a control condition (the existing intervention without a virtual
coach).
70 Fourth International Workshop on Behavior Change Support Systems (BCSS’16):
How Persuasive is a Virtual Coach in Terms of Promoting Web-based Intervention
User Engagement?
For the creation of the virtual coach, we will make use of the Virtual Human envi-
ronment, as kindly offered by the Institute for Creative Technologies, University of
Southern California [23]. As we make use of separate platforms, the users in the ex-
perimental conditions will have two screens, one for ‘Living to the Full’ and on for
the virtual coach. At this stage, we therefore envisage a researcher to start up the Vir-
tual Coach utterances at the right time, i.e. a wizard of Oz solution will be chosen.
As interaction moments between user and virtual coach, we envisage:
• An introduction moment during which the virtual coach presents itself to the
user and indicates that he is there to support the user and is capable of an-
swering user questions, to some extent
• A supportive moment at the middle of the ‘Living to the Full’ intervention
during which the virtual coach expresses its interest in the user (empathic
coach condition) or gives the user the opportunity to ask questions (neutral
coach condition)
• A supportive and evaluative moment at the end of the ‘Living to the Full’ in-
tervention during which both the neutral and empathic virtual coach ask
about the user’s experiences, with the empathic virtual coach doing it in a
empathic manner
After the session, participants will fill-out a questionnaire on user experience, en-
gagement, motivation and perceived persuasiveness. This will be supplemented with
qualitative interview questions. Based on the lessons learned we will prepare a second
study where the intervention will be used in a more real-life setting, e.g. using more
sessions, which enables us to also focus on the effectiveness of adding virtual coach-
support.
3 Future Research
We foresee further Virtual Coach research projects during which we will measure
user boredom and/or frustration and a virtual coach that acts upon these measure-
ments.
References
1. Cuijpers, P., Donker, T., Johansson, R., Mohr, D. C., van Straten, A., Andersson, G.: Self-
guided psychological treatment for depressive symptoms: A meta-analysis. PLoS ONE 6,
e21274.(2011)
2. Androutsopoulos, J.: From variation to heteroglossia in the study of computer-mediated
discourse In: Thurlow C., Mroczek, K.R. (eds.) Digital Discourse: Language in the New
Media. Oxford University Press, Oxford p. 281 (2011)
3. Alley, S., Jennings, C., Persaud, N., Plotnikoff, R.C., Horsley, M., Vandelanotte, C.: Do
Personally Tailored Videos in a Web-Based Physical Activity Intervention Lead to Higher
71 Fourth International Workshop on Behavior Change Support Systems (BCSS’16):
How Persuasive is a Virtual Coach in Terms of Promoting Web-based Intervention
User Engagement?
Attention and Recall? – An Eye-Tracking Study. Emerging Technologies to Promote and
Evaluate Physical Activity (2014).
4. Tanenbaum, J.G., Antle, A.N., Robinson, J.: Three Perspectives on Behavior Change for
Serious Games. A better paradigm could be: learning via the web: e-learning as provided
by ITS Intelligent Tutoring Systems (ITS). In: Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems pp. 3389-3392 (2013)
5. Fogg, B. J. Persuasive Technology: Using Computers to Change What We Think and Do.
Morgan Kaufmann, San Francisco (2002)
6. Bickmore, T., Gruber, A., Picard, R.: Establishing the computer–patient working alliance
in automated health behavior change interventions. Patient education and
counseling, 59(1), 21-30 (2005)
7. Oinas-Kukkonen, H., Harjumaa, M.: Persuasive Systems Design: Key Issues, Process
Model, and System Features Communications of the Association for Information Systems
24, Article 28 (2009)
8. Kelders, S.M., Kok, R.N., Ossebaard, H.C., Van Gemert-Pijnen, J.E.W.C.: Persuasive
System Design Does Matter: A Systematic Review of Adherence to Web-Based
Interventions. Med Internet Res14(6), e152 (2012)
9. Andersson, G., Carlbring, P., Berger, T., Almlo¨v, J., Cuijpers, P.: What makes internet
therapy work? Cognitive Behaviour Therapy, 38, 55–60 (2009)
10. Eysenbach G.: The Law of Attrition. J Med Internet Res. 7(1): e11 (2005)
11. Helgadóttir, F.D., Menzies, R.G., Onslow, M., Packman A., O’Brian, S.: Online CBT I:
Bridging the Gap Between Eliza and Modern Online CBT Treatment Packages. Behaviour
Change 26(04), 245-253 (2009)
12. Talbot, F.: Client Contact in Self-Help Therapy for Anxiety and Depression: Necessary
But Can Take a Variety of Forms Beside Therapist Contact. Behaviour Change 29(02), 63-
76 (2012)
13. Newman, M.G., Szkodny L.E., Llera, S.J., Przeworski, A.: A review of technology-
assisted self-help and minimal contact therapies for anxiety and depression: is human
contact necessary for therapeutic efficacy? Clinical psychology review 31(1), 89–103
(2011)
14. Andrews, G., Cuijpers, P., Craske, M.G., McEnvoy, P.,Ttitov, N.: Computer therapy for
the anxiety and depressive disorders is effective, acceptable and practical health care: A
meta-analysis. PLoS ONE, 5, e13196 (2010)
15. Richards, D., Richardson, T.: Computer-based psychological interventions for depression
treatment: A Systematic review and meta-analysis. Clinical Psychology Review 32, 329-
342 (2012)
16. Farvolden, P., Denisoff, E., Selby, P., Bagby, R.M., Rudy, L.: Usage and longitudinal
effectiveness of a Web-based self-help cognitive behavioral therapy program for panic
disorder. J Med Internet Res. 7(1), e7 (2005)
17. Christensen, H., Griffiths, K.M, Korten, A.E, Brittliffe, K., Groves, C.: A comparison of
changes in anxiety and depression symptoms of spontaneous users and trial participants of
a cognitive behavior therapy website. J Med Internet Res. 6(4), e46 (2004)
18. Christensen, H., Griffiths, K.M., Jorm, A.F.: Delivering interventions for depression by
using the internet: a randomized controlled trial. BMJ 328(7434), 265 (2004)
19. Barak, A., Klein, B., Proudfoot, J.G.: Defining Internet-Supported Therapeutic
Interventions. Ann. Behav. Med. 38, 4–17 (2009)
20. Cugelman, B., Thelwall, M., Dawes, P.: Online Interventions for Social Marketing Health
Behavior Change Campaigns: A Meta-Analysis of Psychological Architectures and
Adherence Factors. J Med Internet Res 13(1), e17 (2011)
72 Fourth International Workshop on Behavior Change Support Systems (BCSS’16):
How Persuasive is a Virtual Coach in Terms of Promoting Web-based Intervention
User Engagement?
21. Schubart, J. R., Stuckey, H. L., Ganeshamoorthy, A., Sciamanna, C. N. Chronic health
conditions and internet behavioral interventions: A review of factors to enhance user en-
gagement. Computers Informatics Nursing 29(2), 81-92 (2011)
22. Andersson, G., Carlbring, P., Bergera, T., Almlöva, J., Cuijpers, P.: What Makes Internet
Therapy Work? Cognitive Behaviour Therapy 38(1), 55-60 (2009)
23. Rizzo, A., Lange, B., Buckwalter, J.G., Forbell, E., Kim, J., Sagae, K., Williams, J.,
Difede, J., Rothbaum, B.O., Reger, G., Parsons, T., Kenny, P.: SimCoach: an intelligent
virtual human system for providing healthcare information and support International
Journal on Disability and Human Development 10(4) (2011)
24. Kennedy, C.M., Powell, J., Payne, T.H., Ainsworth J.: Active Assistance Technology for
Health-Related Behavior Change: An interdisciplinary Review. J Med Internet Res 14(3),
e80 (2012)
25. Norman, G.J., Zabinski, M.G., Adam, M.A., Rosenberg, D.E., Yaroch, A.L., Atienza,
A.A.:. A review of eHealth interventions for physical activity and dietary behavior change.
AM J Prev Med 33(4), 336-345 (2007)
26. Picard, R.W.: Affective Computing. M.I.T. Media Laboratory Perceptual Computing
Section Technical Report No. 321 (1997)
27. Picard, R.W.: Affective computing: challenges. International Journal of Human-Computer
Studies 59(1–2), 55–64 (2003)
28. Damassio, A.: Descartes' Error: Emotion, Reason, and the Human Brain. (1994)
29. Murray, E.: Web-Based Interventions for Behavior Change and Self-Management:
Potential, Pitfalls, and Progress. Med 2.0 1(2), e3 (2012)
30. Kasap, Z., Moussa, M.B. Chaudhuri, P., Magnenat-Thalmann, N.: (2009). Making Them
Remember Emotional Virtual Characters with Memory. Computer Graphics and
Applications 29(2), 20-29 (2009)
31. Amini, R., Lisetti, C., Yasavur, U., Rishe, N.: On-Demand Virtual Health Counselor for
Delivering Behavior-Change Health Interventions. In: Proceedings of the IEEE
International Conference on Healthcare Informatics (ICHI’13). IEEE. pp. 46-55 (2013)
32. Lisetti, C.L.: Embodied Conversational Agents for Psychotherapy. In: Proceedings of the
CHI 2008 Conference Workshop on Technology in Mental Health. ACM, pp. 1-12 (2008)
33. Dixon-Woods, M., Cavers, D., Agarwal, S., Annandale, E., Arthur, A., Harvey, J., Hsu, R.,
Katbamna, S., Olsen, R., Smith, L., Riley R., Sutton, A.J.: Conducting a critical
interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC
Medical Research Methodology 6(35) (2006)
34. Cavanagh, K., Millings, A.: (Inter)Personal Computing: The Role of the Therapeutic
Relationship in E-mental Health. Journal of Contemporary Psychotherapy 43(4), 197-206
(2013)
35. Brouwer, W., Kroeze, W., Crutzen, R., De Nooijer, J., De Vries, N.K., Brug, J., Oenema,
A. Which intervention characteristics are related to more exposure to internet delivered
healthy lifestyle promotion interventions? A systematic review. Journal of Medical
Internet Research, 13(1), e2 (2011)
36. Fry, J.P., Neff, R.A.: Periodic Prompts and Reminders in Health Promotion and Health
Behavior Interventions: Systematic Review. Journal of Medical Internet Research 11(2)
e16 (2009)
37. Eysenbach, G.: The Law of Attrition. J Med Internet Res 7(1):e11 (2006)
38. Cassell, J.: More Than Just Another Pretty Face: Embodied Conversational Interface
Agents. Communications of the ACM 43(4), 70-78 (2000)
39. Klein, J, Moon, Y, Picard, R.W.: This computer responds to user frustration: Theory,
design, and results. Interacting with Computers 14(2), 119–140 (2002)
73 Fourth International Workshop on Behavior Change Support Systems (BCSS’16):
How Persuasive is a Virtual Coach in Terms of Promoting Web-based Intervention
User Engagement?
40. Mentis, H.M.: Memory of frustrating experiences. In: Nahl, D., Bilal, D. (eds.)
Information and Emotion - The Emergent Affective Paradigm in Information Behavior
Research and Theory. Information Today, Medford (2007)
41. Baker, R.S., D'Mello, S.K., Rodrigo, M.M.T., Graesser, A.C.: Better to Be Frustrated than
Bored: The Incidence, Persistence, and Impact of Learners’ Cognitive-Affective States
during Interactions with Three Different Computer-Based Learning Environments.
International Journal of Human-Computer Studies 68(4), 223–241 (2010)
42. D'Mello, S., Olney, A., Williams, C., Hays, P.: Gaze tutor: A gaze-reactive intelligent
tutoring system. International Journal of Human-Computer Studies 70(5), 377–398 (2012)
43. Nijhuis, C.: Gerandomiseerde, gecontroleerde studie naar het effect van Acceptance and
Commitment Therapy & Mindfulness op depressieve klachten : preventieve zelfhulpcursus
"Voluit leven" Master Thesis, University of Twente, Behavioural, Management and Social
Sciences (2010) http://purl.utwente.nl/essays/60478