Measuring Persuasiveness in Behaviour Change Support Systems

Abstract

In order to guide, measure and record progress in BCSS's a framework which outlines relevant aspects and methods by which they may be measured must be agreed. In this paper we introduce relevant criteria from argumentation: argument mode, initial position, belief entrenchment and incremental persuasion and show how these apply to a real world example of behaviour change. We finish by proposing a preliminary evaluation matrix which places our new criteria into context with existing criteria. This has three dimensions: (i) the type of persuasion, (ii) the person being persuaded and (iii) the type of evaluation.

1 Introduction

In order to guide, measure and record progress in BCSS’s a framework which outlines relevant aspects and methods by which they may be measured should be agreed. Evaluation is often by questionnaire, which aims to measure belief and/or behaviour change. For instance, the Perceived Persuasiveness Questionnaire (PPQ) [10] administered after a persuasive interaction with a BCSS, asks users to scale such factors as trustworthiness, influence and relevance; or by an evaluation task [3] in which users rank their preferences before and after a persuasive interaction. A measure of persuasion is constructed and normalised from the difference between the two rankings. In addition to evaluating belief-change, standard quantitative experiments are performed in some contexts to measure behaviour-change, such as measuring the effect that placing different products by a cash register has on consumer behaviour [9]. Here we see an interplay between conscious belief and unconscious choice [6], with the relationship between these two still very unclear.

Both of these types of evaluation are limited in that they cannot explain the processes by which a participant was persuaded, and nor can they be administered during a persuasive interaction, in order to inform real-time strategies employed by a persuader. In this paper we draw on theories from argumentation to propose ways in which existing measures could be enhanced to overcome these two issues. We envisage that future BCSS’s will use the extensive data generated by tracking user behaviour during a persuasive interaction to inform persuasive strategies in real-time.

The paper is organised as follows, in S2 we give a real-world example of persuasion between three participants in a motivational interviewing setting. In S3 we analyse the example from an argumentation perspective and suggest ways in which persuasion could be measured during a persuasive interaction and the processes by which it occurs could be explained. In S4 we use the theories from argumentation to introduce our preliminary evaluation matrix for BCSS, which we plan to expand to include further metrics. We conclude in S5. 2

A Case Study in Motivational Interviewing

We conducted a series of video and audio-recorded interviews between patients and healthcare professionals1 in the context of the Council of Coaches project [1]. Our goal was to see how multiple healthcare professionals might interact with a patient to support them in behaviour change. Here, we present an excerpt from a session in which the patient (Kate) is a lawyer, who has recently been diagnosed with diabetes and is experiencing difficulties with low blood glucose. Kate is talking to a motivational interviewer (Colin) and a dietician (Barbara)2, who suggest that Kate could manage her blood glucose by increasing her intake of carbohydrates. Kate is very reluctant to do this as she is on a low carb diet for weight reasons, preferring instead to change her medication. Kate: Is this your way of trying to get me to take carbs? ( 1 ) Colin: I guess the bottom line is, that nobody can make you do anything. ( 2 ) Barbara: You’re your own person. ( 3 ) Colin: So, it’s up to you what you decide to do. ( 4 ) Kate: Well, to encourage me. ( 5 ) Colin: What I would like to do, is to provide you with information that comes from the same sources you would use for your work, in other words, someone who has spent their career developing their knowledge and their skill, to understand diet and how that relates to diabetes. And that is perhaps what I was trying to get to, not to try to get you to do something you don’t want to do. ( 6 ) Kate: I see. I see where you’re coming from. ( 7 ) Colin: And I just wonder whether it might be useful to have a thought about that, and think whether that might be a conversation worth having, where we can have that open discussion, and you can understand where Barbara, as a dietician, is coming from. ( 8 ) Kate: Well, I’m certainly open to listening. ( 9 ) Colin: Okay. ( 10 ) Kate: I’m not making any promises. ( 11 ) Colin: That would be a useful place to go forward from then? ( 12 ) Kate: Yes, I’ll be open to listening anyway. ( 13 ) Colin: Thank you, okay, yeah. Barbara, what would you add to the conversation around about carbohydrates and the affect on your bodies in relation to diabetes? (14) Barbara: Okay. So maybe more focus should be put on the types of carbohydrates that we’re having, rather than whether we have them in our diet or not. Because it’s absolutely true, I think, as a population, we include more of the less healthy carbohydrates in our diet, so things that are 1 In order to comply with ethical standards the role of the patient was played by an actor, with personas and scenarios developed prior to the interviews. 2 Names have been changed to protect anonymity. really high in refined, added sugar, so cakes and biscuits and sweeties. So yes, they all have carbohydrates, however they’ve also got a lot of fat, a lot of calories, which can contribute to weight gain. But actually there’s lots of healthy types of carbohydrates. Have you heard of something called the glycaemic index before? (15) Kate: No, that’s not something I’ve come across. (16) Barbara: No? So, the glycaemic index is a measurement really, it goes from zero to 100. The highest food you’ll get for a glycaemic index is pure sugar. Because that raises your blood sugar the quickest. Yeah? So, anything lower in the glycaemic index, will raise your blood sugar much slower, it’ll keep you feeling fuller for longer, and those are the types of carbohydrates that we really should be focusing more on, and as a population, we could all do with include more low GI foods in our diet. (17) Kate: Aha. (18) Barbara: So, types of low GI foods are things like wholemeal granary breads, wholemeal pasta, wholemeal rice, beans and pulses as well. (19) Kate: Now, that’s something I would try. I’m really not keen on the bread and the pasta and the rice, but beans and pulses, I could incorporate into a vegetable based meal. (20) Barbara: Beans and pulses are a really good source or soluble fibre, so they’ll help your digestive system working well, also a good source of protein and iron, so they’ve got lots of other nutrients in there. Not only that, they do count as one of your five a day. (21) Kate: Oh do they? (22) Barbara: They do, yeah. So, if you’re having a lovely vegetable based meal, with your beans and pulses, you could potentially have two or three of your five a day in that one meal. (23) Kate: Right okay. (24) Barbara: And also that means, from this conversation, we are including that small amount of carbohydrate, which hopefully will stop your blood sugar coming down, maybe later on, after your meal. (25) Kate: Right okay, that’s interesting. (26) While this is an excerpt from a conversation between people on behaviour change, we are aiming to develop a vitual coaching system in which the healthcare professionals are virtual agents, forming a BCSS. In the following section we discuss how we might evaluate persuasiveness during the conversation, and ways in which argumentation can elucidate how the process of belief change occurs. 3

Case study analysis

Argument Mode. Aristotle identified three modes of persuasion: ethos, concerned with the personal character of the speaker; pathos, eliciting certain emotions from the audience; and logos, using logical arguments and supportive evidence [4]. Here we see all three in evidence: in appealing to Barbara’s expertise in ( 6 ), Colin uses ethos; Barbara uses logos when she builds up a logical argument in (21) that beans and pulses are healthy carbs, shown as an argument map in figure 1; and in (23), Barbara uses pathos when she refers to a “lovely vegetable based meal”. Argument Schemes. Walton et al [12] identify over a hundred patterns of reasoning that are used in everyday arguments. We see two examples in this excerpt: argument from analogy, in which Colin compares the importance of expert sources in the domain of nutrition to those in Kate’s work in ( 6 ): “What I would like to do, is to provide you with information that comes from the same sources you would use for your work”. This uses analogy to move from properties which the persuadee already accepts about a source domain which is familiar to them (knowing about the law), to similar properties in a target domain (knowing about nutrition), in the persuasive step. We also see Colin establishing an argument from authority in ( 8 ), where he says “you can understand where Barbara, as a dietician, is coming from.” For each argument pattern, Walton et al characterise critical questions, such as whether the two domains are similar in a relevant way, or whether the “authority” has really made a certain claim. These provide a useful guide for anticipating what a persuadee might ask. Initial Position. Persuasion may reinforce, shape or change beliefs or behaviour. In order to both evaluate the effect that a persuasive interaction has on a person and tailor it to them, it is important to find out what their initial position is, if they have one. Walton and Krabbe [13] elaborate persuasion dialogues which begin with the participants holding conflicting points of view. The goal of this type of dialogue is to resolve these conflicts, with each participant attempting to persuade the other. In our case study, Kate’s initial position is a belief that carbs are unhealthy and that she should avoid them. This conflict is signalled at the beginning of the excerpt, on ( 1 ), with much of the following conversation a negotiation of which carbs are healthy and which are unhealthy. In other persuasion interactions - and indeed other parts of our excerpt, participants might start from a neutral point of view. We see this in (16), where Kate has no initial position on the glycaemic index, instead moving into an information-seeking dialogue.

Belief Entrenchment and Incremental Persuasion. Belief entrenchment concerns the

degree to which a persuadee holds a belief, and how connected that is to other beliefs. In classic theories of belief revision, such as the AGM model [2] an agent (real or virtual) places a qualitative entrenchment ordering over their beliefs. A persuadee is less likely to give up beliefs that are more entrenched; conversely, less entrenched beliefs are more susceptible to change or revision. Quantitative approaches to entrenchment include the use of argument structure and acceptability to determine measurable effects of giving up or revising a belief [11]. In approaches such as this, a value can be placed on the persuadability of an agent based on the measurable effects of accommodating the new, or conflicting, information in their knowledge base. This can also be used in determining new methods of persuading an agent based on their willingness to accept new information.

Calculating belief entrenchment during a persuasive interaction is useful, since if a belief is very deep then it may be more effective to use incremental methods of persuasion. Calculating belief entrenchment – possibly via longitudinal studies – both before a persuasive interaction to measure depth of initial position, and after, to measure depth of the new persuaded position can be used to evaluate the power of a persuasive system. We see examples of belief entrenchment in our case study in (15-26) where, having realised that Kate’s beliefs about carbohydrates are deeply entrenched, Barbara employs a much more incremental style of persuasion, building up to her overall conclusion. In this case, each of Barbara’s individual sub-arguments represent a small enough change that Kate is willing to accept. 4

A Preliminary Evaluation Matrix for BCSS’s

In the above section we have applied ideas from argumentation to a case study in motivational interviewing, and shown how they would be relevant in an evaluation of a BCSS. One important pre-existing metric is the notion of persuadability – an individuals’ susceptibility to persuasive strategies and principles [8]. This can be measured prior to a persuasive interaction; for instance using the 7-item persuadability questionnaire developed by Kaptein et al [7]. Busch et al [5] developed further methods by which participants’ persuadability using specific strategies can be estimated and utlised to make a BCSS more personalised and persuasive. We have combined this with the argumentation measures and general evaluation metrics to form the following preliminary evaluation matrix for BCSS’s.

Persuasion Persuadee Evaluation

Scheme: [see [12] ] Initial position: [degree] Method [qualitative, quantitative] Mode: [ethos, pathos,logos] Belief Entr: [degree] When performed: [before, during, after] Incremental: [degree] Persuadability: [degree] Who performs: [manually, automatically]

Conclusions

We have shown via a case study that argumentation can suggest useful ways of measuring BCSS’s, and combined these with pre-existing metrics to suggest an evaluation matrix. This is by no means complete or sufficiently detailed: future work will involve developing it to include further metrics and practical guidance.

Acknowledgements. We are indebted to the Council of Coaches team for their assistance in conducting the interviews; in particular to Tessa Beinema and Harm op den Akker for their development of personas and scenarios; to Catherine Pelachaud, Reshmashree Bangalore Kantharaju and Gerwin Huizing for their advice on recording equipment and setup, and to Nicholas Conway for organising and hosting the sessions. We acknowledge the financial support of the European Commission for the H2020 Council of Coaches project, under Grant Agreement No. 769553.

References

1. op den Akker, H., op den Akker, R., Beinema , T. , Banos , O. , Heylen , D. , Bedsted , B. , Pease , A. , Pelachaud , C. , Salcedo , V.T. , Kyriazakos , S. , Hermen , H.: Council of coaches: A novel holistic behavior change coaching approach . In: 4th International Conference on Information and Communication: Technologies for Aging Well and E-Health ( 2018 )

2. Alchourro´n, C. , Ga¨rdenfors, P. , Makinson , D. : On the logic of theory change: Partial meet contraction and revision functions . The Journal of Symbolic Logic 50 ( 2 ), 510 - 530 ( 1985 )

3. Andrews , P. , Manandhar , S. : Measure of belief change as an evaluation of persuasion . In: Persuasive technology and digital behaviour intervention symposium . p. 1 . The Society for the Study of Artificial Intelligence and the Simulation of Behaviour , London, UK ( 2009 )

4. Barnes , J. , et al.: Complete Works of Aristotle , Volume 1 : The Revised Oxford Translation , vol. 1 . Princeton University Press ( 2014 )

5. Busch , M. , Schrammel , J. , Tscheligi , M. : Personalized persuasive technology-development and validation of scales for measuring persuadability . In: International Conference on Persuasive Technology . pp. 33 - 38 . Springer ( 2013 )

6. Custers , R. , Aarts , H.: The unconscious will: How the pursuit of goals operates outside of conscious awareness . Science 329 ( 5987 ), 47 - 50 ( 2010 )

7. Kaptein , M. , Lacroix , J. , Saini , P. : Individual differences in persuadability in the health promotion domain . In: International Conference on Persuasive Technology . pp. 94 - 105 . Springer ( 2010 )

8. Kaptein , M. , Markopoulos , P., de Ruyter , B. , Aarts , E.: Can you be persuaded? individual differences in susceptibility to persuasion . In: IFIP Conference on Human-Computer Interaction . pp. 115 - 118 . Springer ( 2009 )

9. Kroese , M. , Marchiori , D.R. , de Ridder, D.T.D.: Nudging healthy food choices: a field experiment at the train station floor . Journal of Public Health 38 ( 2 ), 133 - 137 ( 2016 )

10. Lehto , T. , Oinas-Kukkonen , H. , Drozd , F. : Factors affecting perceived persuasiveness of a behavior change support system ( 2012 )

11. Snaith , M. , Reed , C. : Argument revision . Journal of Logic and Computation 27 ( 7 ), 2089 - 2134 ( 2016 )

12. Walton , D. , Reed , C. , Macagno , F.: Argumentation Schemes . Cambridge University Press, New York, USA ( 2008 )

13. Walton , D. , Krabbe , E.C. : Commitment in dialogue: Basic concepts of interpersonal reasoning . SUNY press ( 1995 )