To develop effective behaviour change support systems, persuasive technology can be used. The persuasive systems design model offers a framework to identify and operationalize such elements. In this pilot study, we evaluate the questionnaire developed to measure perceived persuasiveness of information technology. We analyzed verbatim user-test transcripts, and performed expert-assessments of the Nurse Antibiotic Information App (NAIA). These data were compared to questionnaire results on this app. Expert-assessment identified task support, perceived persuasiveness, unobtrusiveness, credibility, perceived effort and perceived effectiveness (as defined in the Persuasive Systems Design model) as being present within the NAIA. These constructs also scored satisfactory in the questionnaire. User-test transcripts are in line with questionnaire results. Given the consistent results in this pilot study, our approach seems promising for evaluating the questionnaire and will be applied to other settings and websites/applications.
Any interactive computing system, designed to change users’ attitudes and/or
behaviour, is called persuasive technology [
For the development and design of such technology, the Persuasive Systems
Design model (PSD) can be used [3]. However, using this model during
develop8
ment and design of a Behaviour Change Support System (BCSS), does not
necessarily mean that users feel more motivated for behaviour change. Therefore,
Lehto et al. developed the Perceived Persuasiveness Questionnaire (PPQ) to
predict the perceived persuasiveness of a BCSS [
In this study, the University of Twente and the University of Oulu cooperate, to
evaluate the PPQ, with the ultimate goal to have a validated tool to measure
perceived persuasiveness available. This is important, since the PPQ offers eHealth
developers an opportunity to measure the perceived persuasiveness of their
technology and to test the assumptions of the PSD-model. For the validation, we
evaluate the results of the PPQ in different settings, aimed at civilians, patients
and professionals [
In this paper, we describe an explorative pilot study in which a questionnaire, usability tests, and expert-evaluations are combined to evaluate PPQ constructs, applied to the Nurse Antibiotic Information App (NAIA) [5]. Research questions are: - Does the users’ perceived persuasiveness relate to expert evaluations of the presence of categories for persuasive system principles? - Does the users’ perceived persuasiveness relate to verbalised userexperiences during user-tests? 1.1
Nurses need easily accessible, centralized information support at the point of
care, especially regarding medication safety [
The NAIA was implemented in two lung wards of a local 1000-bed teaching
hospital. A total of 62 nurses (45 FTE) worked at these wards during the pilot
phase. The app was incorporated within the nurses’ personal hospital start-page,
which allowed for easy access [
The Perceived Persuasiveness Questionnaire (as it was available at the time of study) was aimed at evaluating a weight loss application. The questionnaire was used as part of a larger study for summative evaluation of the NAIA, including other measures for behavior change specifically relevant for antimicrobial stewardship [5]. The PPQ was adapted, to fit the research goals of the evaluation study [5]. This means that some constructs of the PPQ were omitted (i.e. dialogue support and social support). Thus, these are also not included in the current study.
Perceived task support, perceived persuasiveness, unobtrusiveness and credibility were included. Two of the credibility items were merged, since no distinguishing Dutch translations could be formulated. We only incorporated one task support item, addressing the overall aim of the behaviour change (appropriate antimicrobial use), since the behaviour itself (e.g. correct administration, preparation, recognition of side effects) is too diverse to address with one item. The questionnaire was translated into Dutch and back-translated into English. Negative items were conversed and construct scores were calculated as the average score of its items. 2.3
The expert-assessment was executed by two native Dutch speaking
researchers, who were both familiar with the app and its purpose. They also had several
years of experience with working with the PSD model. Experts independently
scored the presence of PSD constructs in a demo-version of the app. Only those
constructs that can (as a persuasive strategy) be built into the technology itself, as
features or characteristics of the system, were scored. This means that use
continuance (the users’ intention to continue working with the system [
Analysis of the scenario-based user-tests of the NAIA is currently work-inprogress, whereas here we report on preliminary results of the summative evaluation via user-tests. It should be emphasized that, in this part of the study, nurses were not specifically asked to comment on persuasiveness elements. Rather the user-tests were aimed at the more general evaluation of the user friendliness of the NAIA. Two independent researchers analyzed the verbatim transcripts of 16 of the 34 user-tests that have been performed. This is done by scanning for any remarks, made by the nurse, about constructs of the PSD model. First, the researchers checked whether they identified the same text fragments for coding (thus, text excerpts that exemplify a persuasiveness construct). Second, they checked whether the same code was applied to the fragment. If researchers disagreed, consensus was reached via discussion about the relevance and content of particular comments. The definitions of PPQ constructs and the PSD model were used to guide the discussions. Based on the discussion, the constructs primary task support and perceived effectiveness were merged, since they overlapped greatly. For example, when users indicated that they think the app supports them in their information-search tasks, this indicates primary task support (the complex task of searching for information is made easier by using the app), but also perceived effectiveness (working with the app is beneficial for nurses in quickly and easily finding relevant information). 3 3.1
To gain insight in actual use of the App, log-data were recorded for eight months, between pre- and post-intervention measurement. In that period, the app was visited a 1251 times. It was used an average of 5.11 (SD 3.14) times per day. Most visitors did not only log-in but explored the App further (10.71% of the visits consisted of viewing one page only; the entry page). On average, 5.03 pages were seen per visit, and a visit lasted on average 2 minutes and 26 seconds. 3.2
A total of 34 nurses were invited to complete the questionnaire, of these, 30 nurses actually participated (88.24%). The participants’ mean age was 30.8 (SD 9.06), 26 of them were female. On average, they had 8.45 years (min 0.5, max 38, SD 8.52) of work experience as a nurse. They used the internet for work and private, for an average of 2.54 hours (SD 1.86) per day. Table 1 shows the accumulated, average scores of the measured constructs. 4 5 Quote Items were scored on a 5-point Likert scale, ranging from 1 (totally disagree), 2 (disagree), 3 (don’t agree, don’t disagree), 4 (agree), to 5 (totally agree); *: Negative items are conversed; #: At the time of study, this construct was not part of the PPQ yet.
Participating experts reached high consensus about the presence of the PSD constructs in the app. Primary task support, credibility, unobtrusiveness, perceived persuasiveness, perceived effort and perceived effectiveness were present in the app. Consensus ratings (reached after discussion) are displayed in Table 1.
From the verbatim user test transcripts, remarks on perceived persuasiveness were identified. The results of the analysis (including exemplary quotes) are shown in Table 2. Overall, more positive than negative remarks were made. Most remarks concerned primary task support. Perceived persuasiveness, unobtrusiveness, perceived effort and use continuance were also (positively and negatively) commented on. “[…] that it clearly shows: dose, preparation and administration. That is what I want to know. That’s why I use the App.” “I don’t think it always says how long administration of an antibiotic may take.” “[…] And it’s very convenient that it is so easy to search. That’s much like our good old ‘yellow booklet’ [paper-based antibiotic information, ed.].” “That’s difficult to read, so it is less interesting, because you’ll soon feel like you don’t understand and I would then just leave it to the physician.” “[…] Information that you find on the internet is not specifically written for our hospital. This is.” n.a. “Or just for your own information. […] Because you want to be as well informed as possible when you call the physician.” n.a. “As soon as you type in ‘am’, that Amoxicilline and Augmentin are already suggested to you. I personally find that really convenient.” n.a. “This is really easy to find… just type it in and there it is! For the old systems, we had to go through many steps before you find the information you need. That’s much easier here.” “[…] It might be convenient, that if you have a EPS**, you can select the drug and are automatically brought to the information and don’t have to open the App separately.” “This nicely describes how to prepare the antibiotic, while that [prior information source, ed.] requires you to read through the whole story, before you find the ‘preparation’ heading.” “I notice that I am using it [the app] increasingly often, but I still have to search for a little while.” “Well, as far as antibiotics are concerned, I check the app. At least I do, and I think my colleagues do too.” “Augmentin [an antibiotic, ed.] is something we use very often, so I don’t really check the app for that.” *number of unique participants making one or more remarks in this category
** EPS: Electronic Prescribing System Dialogue support Unobtrusiveness Neg.
5(4) Perceived effort Pos.
Neg.
Pos.
Pos.
Neg.
Pos.
Neg.
3(2) 0(0) 7(4) 6(4) 6(3) 7(7) 3(2) Use Continuance 4
This study combined user-tests, and expert-assessment to evaluate constructs of the Perceived Persuasiveness Questionnaire. Log-data show that, over the eight months between pre- and post-intervention measurement, the Nurse Antibiotic Information App (NAIA) is structurally being used relatively frequently, repeatedly motivating nurses to look-up information. This indicates that the NAIA was incorporated in daily clinical practice, and fulfils a need for easily accessible and well-structured information about antimicrobials. This was also found in prior research [5].
Agreement between experts about the presence of different constructs of PSD was high. Primary task support, credibility, unobtrusiveness, perceived persuasiveness, perceived effort and perceived effectiveness were found in the app. The constructs perceived effort, perceived effectiveness and use continuance were added to the PPQ after the evaluation study of the NAIA. They are therefore omitted in the questionnaire, but they all are included in the user-tests and perceived effort and perceived effectiveness are included in the expert-evaluation. All four constructs, that experts rated as being present in the NAIA, and that were included in the PPQ at the time, were also positively perceived by the nurses in the questionnaire study (score >4).
The user-tests analysis showed similar results: positive remarks are made concerning primary task support, perceived persuasiveness and unobtrusiveness. Credibility did not get as many remarks; it appears to play a relatively smaller role in the practical use of the app. However, even though nurses did not proactively mention the credibility of the app, when asked (with the questionnaire), the app is considered to be credible. So, the mere fact that it was not mentioned, does not necessarily mean it is absent in the app.
A remarkable finding, based on the discussions during the user-tests analysis, was that primary task support and perceived effectiveness had to be merged. Researchers were unable to structurally distinguish these constructs within the users’ comments (the comments simultaneously fitted-in with both constructs). This might be due to the nature and purpose of the app (which is directly aimed at influencing the task performance of nurses), but it might also be an indication of the importance of having a validated Perceived Persuasiveness Questionnaire available, to be able to distinguish between constructs. This pilot study only included a single system, it is therefore impossible to determine which of the two (the system or the questionnaire) caused the problem mentioned above. To avoid such bias (caused by including a single system), we will include multiple apps in the validation study. The currently studied app mainly focusses on primary task support. For the validation study, it is necessary to cover the full range of constructs of the PSD model. Therefore, other apps are included, that might aim at different constructs (e.g. Facebook, Twitter, Prevalence app, Ned i Vekt and Virtual Health Check).
These have different aims (e.g. social interaction, weight loss, infection control) and different target audiences (e.g. professionals, civilians). The PPQ has, in more or lesser extent, been applied to all of these apps, which allows for comparisons to be made.
With this pilot study, we have shown that the PSD model generates consistent results, when measured using different methods. However, results of this study should be interpreted with care, due to some limitations. As this was a pilot study, it had a relatively low number of participants (users and experts). Also, not all constructs of the PPQ were included in the questionnaire study. Finally, its results may have been influenced by other questionnaires that were simultaneously used (concerning e.g. usability and empowerment).
In future research, we will do more in-depth log-file analyses, focussing on which
parts of the NAIA are mainly used and at what moments, as prior research has
shown that log-files may be used to study the effect of persuasive elements in
eHealth technology [
recommendations for integrated displays to improve nurses' situation awareness. Journal of American Medical Informatics Association, 2012. 19(4): p. 583–590.