=Paper=
{{Paper
|id=Vol-1153/Paper_1
|storemode=property
|title=Combining User-Tests, Log-Data and Expert-Assessments to Evaluate Constructs of the Perceived Persuasiveness Questionnaire
|pdfUrl=https://ceur-ws.org/Vol-1153/Paper_1.pdf
|volume=Vol-1153
|dblpUrl=https://dblp.org/rec/conf/persuasive/JongWKOG14
}}
==Combining User-Tests, Log-Data and Expert-Assessments to Evaluate Constructs of the Perceived Persuasiveness Questionnaire==
https://ceur-ws.org/Vol-1153/Paper_1.pdf
Second
International
Workshop
on
Behavior
Change
Support
Systems
(BCSS
2014)
7
Evaluation of Perceived Persuasiveness Constructs by
Combining User Tests and Expert Assessments
Nienke de Jong1, Jobke Wentzel1, Saskia Kelders1, Harri Oinas-Kukkonen2, Julia
van Gemert-Pijnen1
1
University of Twente, Faculty of Behavioural Science,
Enschede, the Netherlands
{n.dejong, m.j.wentzel, s.m.kelders, j.vangemert-pijnen}@utwente.nl
2
University of Oulu, Department of Information Processing Science,
Oulu, Finland
harri.oinas-kukkonen@oulu.fi
Abstract. To develop effective behaviour change support systems, persuasive
technology can be used. The persuasive systems design model offers a frame-
work to identify and operationalize such elements. In this pilot study, we
evaluate the questionnaire developed to measure perceived persuasiveness of
information technology. We analyzed verbatim user-test transcripts, and per-
formed expert-assessments of the Nurse Antibiotic Information App (NAIA).
These data were compared to questionnaire results on this app.
Expert-assessment identified task support, perceived persuasiveness, unobtru-
siveness, credibility, perceived effort and perceived effectiveness (as defined
in the Persuasive Systems Design model) as being present within the NAIA.
These constructs also scored satisfactory in the questionnaire. User-test tran-
scripts are in line with questionnaire results.
Given the consistent results in this pilot study, our approach seems promising
for evaluating the questionnaire and will be applied to other settings and web-
sites/applications.
Keywords: eHealth, Perceived Persuasiveness, User-tests, Expert-Assessment
1 Introduction
Any interactive computing system, designed to change users’ attitudes and/or
behaviour, is called persuasive technology [1]. Oinas-Kukkonen and Harjumaa
[2] state that the changing of users’ attitudes and/or behaviour should be achieved
without using coercion or deception.
For the development and design of such technology, the Persuasive Systems De-
sign model (PSD) can be used [3]. However, using this model during develop-
8
Second
International
Workshop
on
Behavior
Change
Support
Systems
(BCSS
2014)
ment and design of a Behaviour Change Support System (BCSS), does not neces-
sarily mean that users feel more motivated for behaviour change. Therefore,
Lehto et al. developed the Perceived Persuasiveness Questionnaire (PPQ) to pre-
dict the perceived persuasiveness of a BCSS [4]. However, thus far, the PPQ has
not been fully validated yet. It is of importance that this is done, to be able to
reliably compare different eHealth technologies, or their application within dif-
ferent settings, with each other.
In this study, the University of Twente and the University of Oulu cooperate, to
evaluate the PPQ, with the ultimate goal to have a validated tool to measure per-
ceived persuasiveness available. This is important, since the PPQ offers eHealth
developers an opportunity to measure the perceived persuasiveness of their tech-
nology and to test the assumptions of the PSD-model. For the validation, we
evaluate the results of the PPQ in different settings, aimed at civilians, patients
and professionals [4-8]. We will perform expert-assessments to determine which
elements of the PSD model are actually incorporated in the ICT system under
investigation.
In this paper, we describe an explorative pilot study in which a questionnaire,
usability tests, and expert-evaluations are combined to evaluate PPQ constructs,
applied to the Nurse Antibiotic Information App (NAIA) [5]. Research questions
are:
- Does the users’ perceived persuasiveness relate to expert evaluations of
the presence of categories for persuasive system principles?
- Does the users’ perceived persuasiveness relate to verbalised user-
experiences during user-tests?
1.1 The Nurse Antibiotic Information App
Nurses need easily accessible, centralized information support at the point of
care, especially regarding medication safety [9, 10]. As part of an antimicrobial
stewardship program (promoting prudent use of antimicrobials), the web-based
NAIA [11] was developed. A more detailed description of the NAIA and it’s
participatory development process is given elsewhere [5, 12]. The NAIA includes
information on the preparation and administration, but also optionally provides
additional background information.
2 Methods
2.1 Study Setting
The NAIA was implemented in two lung wards of a local 1000-bed teaching
hospital. A total of 62 nurses (45 FTE) worked at these wards during the pilot
phase. The app was incorporated within the nurses’ personal hospital start-page,
which allowed for easy access [12]. At the time of the study, the app had been
Second
International
Workshop
on
Behavior
Change
Support
Systems
(BCSS
2014)
9
available for >6 months at the ward, offering nurses many occasions to use it and
get familiar with it.
2.2 The Perceived Persuasiveness Questionnaire
The Perceived Persuasiveness Questionnaire (as it was available at the time of
study) was aimed at evaluating a weight loss application. The questionnaire was
used as part of a larger study for summative evaluation of the NAIA, including
other measures for behavior change specifically relevant for antimicrobial stew-
ardship [5]. The PPQ was adapted, to fit the research goals of the evaluation
study [5]. This means that some constructs of the PPQ were omitted (i.e. dialogue
support and social support). Thus, these are also not included in the current study.
Perceived task support, perceived persuasiveness, unobtrusiveness and credi-
bility were included. Two of the credibility items were merged, since no distin-
guishing Dutch translations could be formulated. We only incorporated one task
support item, addressing the overall aim of the behaviour change (appropriate
antimicrobial use), since the behaviour itself (e.g. correct administration, prepara-
tion, recognition of side effects) is too diverse to address with one item. The
questionnaire was translated into Dutch and back-translated into English. Nega-
tive items were conversed and construct scores were calculated as the average
score of its items.
2.3 Expert-Assessment of Persuasive Elements Within the Nurse Antibi-
otic Information App
The expert-assessment was executed by two native Dutch speaking research-
ers, who were both familiar with the app and its purpose. They also had several
years of experience with working with the PSD model. Experts independently
scored the presence of PSD constructs in a demo-version of the app. Only those
constructs that can (as a persuasive strategy) be built into the technology itself, as
features or characteristics of the system, were scored. This means that use contin-
uance (the users’ intention to continue working with the system [8]) was omitted
as this is more an outcome of persuasive strategies than a strategy in itself. Scor-
ing was performed on a 5-point Likert scale, differences were discussed to reach
consensus.
2.4 User-Test Analysis for Reported Persuasiveness
Analysis of the scenario-based user-tests of the NAIA is currently work-in-
progress, whereas here we report on preliminary results of the summative evalua-
tion via user-tests. It should be emphasized that, in this part of the study, nurses
were not specifically asked to comment on persuasiveness elements. Rather the
user-tests were aimed at the more general evaluation of the user friendliness of
the NAIA. Two independent researchers analyzed the verbatim transcripts of 16
of the 34 user-tests that have been performed. This is done by scanning for any
10
Second
International
Workshop
on
Behavior
Change
Support
Systems
(BCSS
2014)
remarks, made by the nurse, about constructs of the PSD model. First, the re-
searchers checked whether they identified the same text fragments for coding
(thus, text excerpts that exemplify a persuasiveness construct). Second, they
checked whether the same code was applied to the fragment. If researchers disa-
greed, consensus was reached via discussion about the relevance and content of
particular comments. The definitions of PPQ constructs and the PSD model were
used to guide the discussions. Based on the discussion, the constructs primary
task support and perceived effectiveness were merged, since they overlapped
greatly. For example, when users indicated that they think the app supports them
in their information-search tasks, this indicates primary task support (the complex
task of searching for information is made easier by using the app), but also per-
ceived effectiveness (working with the app is beneficial for nurses in quickly and
easily finding relevant information).
3 Results
3.1 Use of the Nurse Antibiotic Information App
To gain insight in actual use of the App, log-data were recorded for eight
months, between pre- and post-intervention measurement. In that period, the app
was visited a 1251 times. It was used an average of 5.11 (SD 3.14) times per day.
Most visitors did not only log-in but explored the App further (10.71% of the
visits consisted of viewing one page only; the entry page). On average, 5.03 pag-
es were seen per visit, and a visit lasted on average 2 minutes and 26 seconds.
3.2 Perceived Persuasiveness
A total of 34 nurses were invited to complete the questionnaire, of these, 30
nurses actually participated (88.24%). The participants’ mean age was 30.8 (SD
9.06), 26 of them were female. On average, they had 8.45 years (min 0.5, max 38,
SD 8.52) of work experience as a nurse. They used the internet for work and
private, for an average of 2.54 hours (SD 1.86) per day. Table 1 shows the accu-
mulated, average scores of the measured constructs.
Table 1. Results of expert-assessment and PPQ questionnaire.
PSD construct Presence PPQ Score*
Primary task support 5 4.25
Dialogue support 1 -
Credibility 4 4.13
Social support 1 -
Unobtrusiveness 4 4.11
Perceived persuasiveness 4 4.11
Second
International
Workshop
on
Behavior
Change
Support
Systems
(BCSS
2014)
11
Perceived effort 4 #
Perceived effectiveness 5 #
Use continuance - #
Items were scored on a 5-point Likert scale, ranging from 1 (totally disagree), 2 (disagree), 3 (don’t
agree, don’t disagree), 4 (agree), to 5 (totally agree); *: Negative items are conversed; #: At the time
of study, this construct was not part of the PPQ yet.
3.3 Presence of PSD Constructs in the App
Participating experts reached high consensus about the presence of the PSD con-
structs in the app. Primary task support, credibility, unobtrusiveness, perceived
persuasiveness, perceived effort and perceived effectiveness were present in the
app. Consensus ratings (reached after discussion) are displayed in Table 1.
3.4 Remarks About PSD Constructs During User-Tests
From the verbatim user test transcripts, remarks on perceived persuasiveness
were identified. The results of the analysis (including exemplary quotes) are
shown in Table 2. Overall, more positive than negative remarks were made. Most
remarks concerned primary task support. Perceived persuasiveness, unobtrusive-
ness, perceived effort and use continuance were also (positively and negatively)
commented on.
Table 2. Overview of user-test analysis results
PPQ N (*) Quote
Primary task support
Pos. 32(14) “[…] that it clearly shows: dose, preparation and administration. That is what I
want to know. That’s why I use the App.”
Neg. 11(7) “I don’t think it always says how long administration of an antibiotic may take.”
Perceived persuasiveness
Pos. 8(5) “[…] And it’s very convenient that it is so easy to search. That’s much like our
good old ‘yellow booklet’ [paper-based antibiotic information, ed.].”
Neg. 4(3) “That’s difficult to read, so it is less interesting, because you’ll soon feel like you
don’t understand and I would then just leave it to the physician.”
Credibility
Pos. 1(1) “[…] Information that you find on the internet is not specifically written for our
hospital. This is.”
Neg. 0(0) n.a.
Social Support
Pos. 6(6) “Or just for your own information. […] Because you want to be as well in-
formed as possible when you call the physician.”
Neg. 0(0) n.a.
12
Second
International
Workshop
on
Behavior
Change
Support
Systems
(BCSS
2014)
Dialogue support
Pos. 3(2) “As soon as you type in ‘am’, that Amoxicilline and Augmentin are already
suggested to you. I personally find that really convenient.”
Neg. 0(0) n.a.
Unobtrusiveness
Pos. 7(4) “This is really easy to find… just type it in and there it is! For the old systems,
we had to go through many steps before you find the information you need.
That’s much easier here.”
Neg. 5(4) “[…] It might be convenient, that if you have a EPS**, you can select the drug
and are automatically brought to the information and don’t have to open the
App separately.”
Perceived effort
Pos. 6(4) “This nicely describes how to prepare the antibiotic, while that [prior infor-
mation source, ed.] requires you to read through the whole story, before you
find the ‘preparation’ heading.”
Neg. 6(3) “I notice that I am using it [the app] increasingly often, but I still have to search
for a little while.”
Use Continuance
Pos. 7(7) “Well, as far as antibiotics are concerned, I check the app. At least I do, and I
think my colleagues do too.”
Neg. 3(2) “Augmentin [an antibiotic, ed.] is something we use very often, so I don’t really
check the app for that.”
*number of unique participants making one or more remarks in this category
** EPS: Electronic Prescribing System
4 Discussion
This study combined user-tests, and expert-assessment to evaluate constructs of
the Perceived Persuasiveness Questionnaire. Log-data show that, over the eight
months between pre- and post-intervention measurement, the Nurse Antibiotic
Information App (NAIA) is structurally being used relatively frequently, repeat-
edly motivating nurses to look-up information. This indicates that the NAIA was
incorporated in daily clinical practice, and fulfils a need for easily accessible and
well-structured information about antimicrobials. This was also found in prior
research [5].
Agreement between experts about the presence of different constructs of PSD
was high. Primary task support, credibility, unobtrusiveness, perceived persua-
siveness, perceived effort and perceived effectiveness were found in the app.
The constructs perceived effort, perceived effectiveness and use continuance
were added to the PPQ after the evaluation study of the NAIA. They are therefore
omitted in the questionnaire, but they all are included in the user-tests and per-
ceived effort and perceived effectiveness are included in the expert-evaluation.
Second
International
Workshop
on
Behavior
Change
Support
Systems
(BCSS
2014)
13
All four constructs, that experts rated as being present in the NAIA, and that were
included in the PPQ at the time, were also positively perceived by the nurses in
the questionnaire study (score >4).
The user-tests analysis showed similar results: positive remarks are made con-
cerning primary task support, perceived persuasiveness and unobtrusiveness.
Credibility did not get as many remarks; it appears to play a relatively smaller
role in the practical use of the app. However, even though nurses did not pro-
actively mention the credibility of the app, when asked (with the questionnaire),
the app is considered to be credible. So, the mere fact that it was not mentioned,
does not necessarily mean it is absent in the app.
A remarkable finding, based on the discussions during the user-tests analysis, was
that primary task support and perceived effectiveness had to be merged. Re-
searchers were unable to structurally distinguish these constructs within the users’
comments (the comments simultaneously fitted-in with both constructs). This
might be due to the nature and purpose of the app (which is directly aimed at
influencing the task performance of nurses), but it might also be an indication of
the importance of having a validated Perceived Persuasiveness Questionnaire
available, to be able to distinguish between constructs. This pilot study only in-
cluded a single system, it is therefore impossible to determine which of the two
(the system or the questionnaire) caused the problem mentioned above. To avoid
such bias (caused by including a single system), we will include multiple apps in
the validation study. The currently studied app mainly focusses on primary task
support. For the validation study, it is necessary to cover the full range of con-
structs of the PSD model. Therefore, other apps are included, that might aim at
different constructs (e.g. Facebook, Twitter, Prevalence app, Ned i Vekt and Vir-
tual Health Check).
These have different aims (e.g. social interaction, weight loss, infection control)
and different target audiences (e.g. professionals, civilians). The PPQ has, in
more or lesser extent, been applied to all of these apps, which allows for compari-
sons to be made.
With this pilot study, we have shown that the PSD model generates consistent
results, when measured using different methods. However, results of this study
should be interpreted with care, due to some limitations. As this was a pilot study,
it had a relatively low number of participants (users and experts). Also, not all
constructs of the PPQ were included in the questionnaire study. Finally, its results
may have been influenced by other questionnaires that were simultaneously used
(concerning e.g. usability and empowerment).
In future research, we will do more in-depth log-file analyses, focussing on which
parts of the NAIA are mainly used and at what moments, as prior research has
shown that log-files may be used to study the effect of persuasive elements in
eHealth technology [13, 14]. Additional user-tests will be analysed, to allow for
conclusions in the field of effectiveness of the NAIA. The study will, as men-
tioned before, additionally be applied to other apps and other settings, and will be
14
Second
International
Workshop
on
Behavior
Change
Support
Systems
(BCSS
2014)
complemented with thorough evaluation of a Dutch PPQ. These are all important
steps to be taken to enable valid PSD evaluations in summative research.
The current pilot study gave us a framework, based on which we will work to-
wards validating the PPQ. We created a protocol for expert-assessment of a be-
haviour change and its support system, we showed how this evaluation enables a
PSD focus, and provided an example of validation via user-tests.
References
1. Fogg, B.J., Persuasive Technology: Using Computers to Change what
We Think and Do. 2003: Morgan Kaufmann Publishers.
2. Oinas-Kukkonen, H. and Harjumaa, M. (2008) Towards Deeper Under-
standing of Persuasion in Software and Information Systems. The First
International Conference on Advances in Human-Computer Interaction
(ACHI ’2008), Sainte Luce, Martinique, February 10-15, 2008, ISBN
978-0-7695-3086-4, pp. 200-205.
3. Oinas-Kukkonen, H. and M. Harjumaa, Persuasive Systems Design: Key
Issues, Process Model, and System Features. Communications of the
Association for Information Systems, 2009. 24: p. 485-500.
4. Lehto, T., Oinas-Kukkonen, H., and Drozd, F. (2012) Factors Affecting
Perceived Persuasiveness of a Behavior Change Support System. Inter-
national Conference on Information Systems (ICIS 2012), Orlando,
Florida, December 16-19.
5. Wentzel, J. and J.E.W.C. van Gemert-Pijnen. Antibiotic Information
App for nurses. in The Sixth International Conference on eHealth,
Telemedicine, and Social Medicine. in press. Barcelona, Spain.
6. Jong de, N, A. Eikelenboom-Boskamp, A. Voss, J.E.W.C. van Gemert-
Pijnen, User-centered and persuasive design of a web-based registration
and monitoring system for healthcare-associated infections in nursing
homes. in The Sixth International Conference on eHealth, Telemedicine,
and Social Medicine. in press. Barcelona, Spain.
7. Lehto, T., Oinas-Kukkonen, H., Pätiälä, T. and Saarelma, O. (2012)
Consumers’ Perceptions of a Virtual Health Check: An Empirical Inves-
tigation. In: 20th European Conference on Information Systems, ECIS
2012 Proceedings, Paper 154. http://aisel.aisnet.org/ecis2012/154.
8. Lehto, T. and Oinas-Kukkonen, H. (2014). Explaining and Predicting
Perceived Effectiveness and Use Continuance Intention of a Behavior
Change Support System. Behaviour and Information Technology, online
first. doi:10.1080/0144929X.2013.866162..
9. Koch, S.H., C. Weir, M. Haar, N. Staggers, J. Agutter, M. Görges and D.
Westenskow, Intensive care unit nurses' information needs and
Second
International
Workshop
on
Behavior
Change
Support
Systems
(BCSS
2014)
15
recommendations for integrated displays to improve nurses' situation
awareness. Journal of American Medical Informatics Association, 2012.
19(4): p. 583–590.
10. Ndosi, M. and R. Newell, Medicine information sources used by nurses
at the point of care. Journal of Clinical Nursing, 2010. 19(17-18): p.
2659-2661.
11. Demo Information Application for nurses. In Dutch. [cited 2014 03-10];
Available from: http://abnurseapp.infectionmanager.com.
12. Wentzel, J., L, van Velsen, A.H.M. van Limburg, N. de Jong, J.
Karreman, R. Hendrix, and J.E.W.C. van Gemert-Pijnen, Participatory
eHealth Development to support Nurses in antimicrobial Stewardship.
BMC Medical Informatics and Decision Making, in review.
13. Gemert-Pijnen van, J.E.W.C., M.S. Kelders, and T.E. Bohlmeijer,
Understanding the Usage of Content in a Mental Health Intervention for
Depression: An Analysis of Log Data. Journal of Medical Internet
Research, 2014. 16(1): p. e27.
14. Kelders, S. and J.E.W.C. Gemert-Pijnen, Using Log-Data as a Starting
Point to Make eHealth More Persuasive, in Persuasive Technology, S.
Berkovsky and J. Freyne, Editors. 2013, Springer Berlin Heidelberg. p.
99-109.