=Paper=
{{Paper
|id=None
|storemode=property
|title=Analysis of the Persuasiveness of User Experience Feedback on a Virtual Learning Environment
|pdfUrl=https://ceur-ws.org/Vol-656/paper7.pdf
|volume=Vol-656
|dblpUrl=https://dblp.org/rec/conf/iused/MullerLS10
}}
==Analysis of the Persuasiveness of User Experience Feedback on a Virtual Learning Environment==
https://ceur-ws.org/Vol-656/paper7.pdf
Analysis of the Persuasiveness of User Experience
Feedback on a Virtual Learning Environment
Daniel Müller Effie L.-C. Law Stefan Strohmeier
IMC Dept. of Computer Science Chair of MIS
Altenkesseler Str. 17 D3 University of Leicester Saarland University
66115 Saarbruecken, LE1 7RH 66041 Saarbruecken,
Germany Leicester, UK Germany
+49 681 302 64752 +44 116 252 5341 +49 681 302 64751
daniel.mueller@im-c.de elaw@mcs.le.ac.uk s.strohmeier@mis.uni-
saarland.de
ABSTRACT system design guidelines [23, 32, 34, 36], the ISSM
The main purpose of this paper is to evaluate a set of identifies and provides general qualities which are thought
theory-grounded User Experience (UX)-related measures to enhance user satisfaction, the use of, and the net
which are supposed to persuade course designers of benefit(s) (NB) of using a VLE [2, 3]. However, the main
particular UX-related problem areas based on a specific disadvantage of the ISSM used as a general approach is that
feedback format. Specifically, two online surveys on a specific VLE-related success drivers cannot be directly
university online course were conducted with the former derived from the model itself. Rather, the ISSM offers
focusing on the quantitative ratings and the latter on insights into the process of how general qualities, namely
qualitative comments. The course designers were asked to system- and information quality, influence the final success
assess the persuasiveness of the feedback with respect to [2, 3]. Hence, the ISSM offers a general and “useful
eight dimensions. The results show that UX-related framework for organizing IS success measurements” [27]
problem areas anchored in the DeLone and McLean’s which can and should be adapted to the VLE context [3,
Information Systems (IS) Success Model (ISSM) had a 27]. Though, beside more general recommendations for the
consistently higher level of perceived persuasiveness than selection of success measures [29], there currently lacks a
those anchored in the Technology Acceptance Model widely accepted set of measures relevant to VLE in
(TAM) 3. The discrepancy can be attributed to the nature of particular.
items: process- vs. trait-based). Implications for future
However, some latest research attempts striving for a VLE-
research on fixing UX-related problems are discussed.
specific extension of the ISSM revealed a comprehensive
Keywords and exhaustively validated set of system- and information-
Course Designer, Design Characteristic, Feedback Format, related design characteristics relevant to VLE in particular
IS Success Model, Persuasiveness, TAM3, User [22]. As some of these design characteristics, respectively
Experience. their corresponding items can be adequate measures for UX
as well, these UX-related design characteristics may
INTRODUCTION support designers (here: course designers 1 ) in their attempts
At the present day rigorous, i.e. theory-grounded, and to fix not only usability-related issues [25] but also UX-
relevant, i.e. practice-oriented, approaches for the design related problem areas (e.g. image, see Table 1). Thereby,
and evaluation of Virtual Learning Environments (VLE) UX-related problem areas anchored in the ISSM are
are needed to improve the understanding and compared against the ability of selected UX-related ones
communication of educational needs among all anchored in the TAM3 [32] in order to carve out
stakeholders, including researchers and practitioners [10]. differences in the persuasiveness of the feedback format for
In this paper VLE are understood as systems for the course designers. This construct is assumed to be
administrative and didactical support of learning processes dependent on a) the kind of theory applied (ISSM =
in higher education and vocational training settings by product-oriented; TAM3 = state-/trait-oriented) and b) the
means of formal online courses [22]. Hence, it is of great information richness of the feedback format provided.
importance to investigate the drivers or determinants of
VLE success to assist system and course designers in According to Nørgaard and Hornbæk [25], the underlying
building, operating and sustaining systems and online assumption is as follows: The richer the UX problem area-
courses as integral parts that are useful and accepted by the related contextual information contained in the feedback
end-user (here: students). However, a specific theory of format, the higher persuasiveness of this feedback format is
successful VLE is currently missing as existing approaches for course designers. Thus, in search for a rigorous and
focus on information systems (IS) in general with the
1
DeLone and McLean’s ISSM [2, 3] being one of them. In addition to research efforts solely focusing on system
Contrary to the TAM, which does not propose concrete designers [e.g. 6, 7, 8, 4, 15, 16, 25].
persuasive UX-related feedback format 2 , the following and responses that result from the use and/or anticipated
research questions (RQ) will be addressed in this paper: use of a product, system or service” (clause 2.15), which is
RQ1: To what extent do students as end-users have more relatively simpler than, for instance, the comprehensive
problems in specifying UX-related problem areas based on definition by Hassenzahl and Tractinsky [8], one of the
TAM3-related UX items than those based on their ISSM- many definitions in the literature [15]. In an attempt to
related counterparts (see Table 1, students’ item rating understand the diverse interpretations of UX, the
statements)? researchers, based on the results of a survey [15], have
drawn a conclusion that UX is “dynamic, context-
RQ2: To what extent do course designers perceive UX- dependent and subjective, which stems from a broad range
related problem areas (see Table 1: based on the construct of potential benefits users may derive from a product”.
label, construct definition, item wording, item-UX-match,
students’ item rating and students’ item rating statement) Feedback Formats as a Means to Persuade Designers
originated in the ISSM to be more persuasive than their Feedback can be understood as “information about
TAM3-related counterparts? reactions to a product, a person's performance of a task,
etc. which is used as a basis for improvement” [25, 26].
RQ3: Which of the UX-related problem areas (TAM3- vs. According to Nørgaard and Hornbæk [25], feedback should
ISSM-anchored) do course designers perceive to be more fulfill the requirement of being persuasive: Firstly,
persuasive in case evaluators’ suggestions are provided in feedback should convince developers that the problem
addition to the set of UX problem area-related contextual identified does exist and helps them to understand it.
information illustrated in Table 1? Secondly, the persuasiveness of a feedback format is
Based on these research questions, the main purpose of this determined by the amount of contextual information about
paper is to identify and validate a set of theory-grounded, a problem it conveys. Thirdly, the ease the feedback can be
UX-related measures of which persuasiveness presumably used in the developer’s everyday work is important.
varies with feedback format. In this context, we define Moreover, given that UX is inherently dynamic, context-
persuasiveness in terms of convincing course designers dependent and subjective [15, 16], feedback on UX-related
about the problematicity of particular UX-related issues, problems should essentially be self-reported data to be
which may entail specific resolutions. captured by questionnaire, interview and think-aloud.
In the paper we first explore the concept of UX as well as Nonetheless, data on UX can be gathered with a survey
feedback formats as a means to persuade designers. Then where respondents are first required to rate a set of items
we present the methodological framework regarding with a Likert scale and subsequently interviewed to
students’ specification of UX-related problem areas as well elaborate their comments on the items.
as course designers’ assessment of their persuasiveness of METHOD
the feedback formats generated. Next, we describe the Identifying Students’ UX-related Problem Areas
empirical results with regard to the overall persuasiveness Participants of the current study were eleven students of the
of the feedback format (quantitative evaluation) and online course Organizational Management, which was
particular UX-related problem areas (qualitative delivered during the summer term 2010 by the Chair of
evaluation) as perceived by the course designers. The Management Information Systems (MIS) located at
aforementioned three research questions will be then Saarland University/Germany. Prior to this study, these
discussed. Finally, implications for future research efforts participants had been asked to complete a larger-scale
and conclusion are drawn. online survey with 88 items being originated from ISSM
BACKGROUND and TAM3. The aim of this first survey (N=30) was to
User Experience evaluate students’ acceptance towards the aforementioned
As distinct from usability-centred evaluations which course. Thereby, all items showed high levels of construct
roughly focus on task-related issues such as efficiency and validity, evaluated via a satisfactory convergent (Average
effectiveness [7], “[UX] proposes a more holistic view of Variance Explained, Composite Reliability, significant
the user’s experience when using a product than is usually indicator factor loadings exceeding a threshold of 0.70),
taken in the evaluation of usability” [12]. While discriminant, and nomological validity. Some of the first
pragmatic/do goals are associated with usability, survey items can be mapped to the hedonic attributes of the
hedonic/be goals address cognitive, socio-cognitive and model of UX proposed by Hassenzahl [6]. As the primary
affective aspects of users’ experience in their interaction focus lies on more hedonic attributes, more pragmatic/task-
with artifacts (e.g. users’ enjoyment, aesthetic experience, related ones such as the perceived usefulness as well as the
desire to repeat use, positive decision to use a digital perceived ease of using a VLE are out of scope of this
artifact and enhanced mental models) [1, 16]. However, a paper [8, 12, 32]. The mapping was undertaken by the first
consensual definition of UX does not yet exist, although and second authors of this paper, resulting in 17 items that
ISO 9241-210 [13] provides one: “A person’s perceptions constitute the second online survey. Consequently, the
second survey consists of UX-related items. Specifically,
we define a UX-related item as a problem area if its mean
2
As distinct from research efforts solely focusing on the rating (averaged over all the respondents involved in the
persuasiveness of usability-related feedback formats [e.g. 25]. first survey) falls between 1.00 and 3.99 and as a still-to-
be-improved area if it is between 4.00 and 4.50. In this problem enumerations [14]. In order to survey the
case, a UX-related construct (the column Label in Table 1) persuasiveness of a) the overall feedback format in general
could contain both types of area. The corresponding as well as b) the UX-related problem areas in particular
boundary values are defined by the MIS monitoring team (see Table 1), a questionnaire was administered which was
responsible for the quality control of the MIS’s online mainly based on Norgaard and Hornbaek’s [25]
courses. Moreover, UX-related problem areas originated measurement instrument to evaluate the persuasiveness of a
from either the TAM3 or the ISSM are randomly put in one feedback format as perceived by developers. The
sequence to prevent sequence effects. In addition to the corresponding questions are:
item (Table 1) the participants were provided with the Q1: “How useful is the information provided in Table 1
corresponding average ratings 3 (1 = strongly disagree; 5 = (construct label, construct definition, item wording, item-
strongly agree) as well as a hint to their individual ratings UX-match, students’ item rating and students’ item rating
of the first online survey (students were provided with a statement) to your work on the online course
copy of their individual ratings after having completed the Organizational Management? (1=not useful – 5=very
first online survey) and were asked to comment on them. useful). Furthermore, please comment on the usefulness of
The reasons for providing students with the averages as the information provided by referring to the UX-related
well as a corresponding hint to their individual rating in the problem areas label (e.g. user interface appeal, see Table 1,
first online survey are due to the university’s data policy first column).
prohibiting the chair’s monitoring team to confront students
with their individual ratings of a preliminary survey Q2: How well does the information provided in Table 1
directly. Thus, UX-related problem areas were further help you to understand the UX-related problem area(s)? (1=
specified and thus contextualized based on students’ very poor – 5 = very well).
additional qualitative written input (illustrative example per a) Please comment on the level of understandability of the
item, see Table 1, column 6). information provided in Table 1 by referring to particular
In summary, the tasks the participants of the second survey columns (i.e. vertical evaluation).
had to undertake were: (i) They had to provide their b) Additionally, please differ between the understandability
personal statements to each item rating by referring to the of the information provided in Table 1 by referring to
online course Organizational Management; (ii) based on particular item (i.e. horizontal evaluation).
their statements, they had to explain how they would solve Q3: How well does the information provided in Table 1
the perceived problem areas. This information could serve have an impact on assessing the severity of the UX-related
as a starting point for the evaluator’s suggestions (see Table problem area(s)? (1=very poor – 5= very well). Please
1, last column), which were presented to the course comment on the severity of (a) particular problem area(s).
designers to evaluate their persuasiveness of the feedback
format. Q4: How well does the information provided in Table 1
help you solve the UX-related problem area(s)? (1= very
Evaluating the Persuasiveness of the Feedback Format poorly – 5= very well). Please comment on the ability of
by Course Designers
the information provided in Table 1 to solve a particular
Three course designers (1 professor, 1 research problem area(s).
professional, 1 research assistant) were invited to attend
semi-structured interviews to gather their evaluation of the Q5: Do you intend to solve the UX-related problem areas
persuasiveness of the UX-related problem areas (the first illustrated in Table 1? (1 = not at all – 5 = absolutely). If so,
five columns of Table 1). In addition, the contextual which of the problem area(s) and why?
information gathered from the participants in the previous Q6: Are you convinced that the information provided in
step (the 6th column in Table 1) was further concretized Table 1 depicts real UX-related problem areas? (1 = not at
with evaluators’ suggestions how to solve particular UX- all – 5 = absolutely). If not, which of the problem areas and
related problem areas (the last column of Table 1). The why?
concretization was undertaken by the MIS monitoring team
Q7: Is the information provided in Table 1 easy to use to
based on the participants’ qualitative written input as
solve particular UX-related problem areas? (1 = not at all –
described in the previous step. Hence, the final feedback
5 = absolutely). If not, which of the problem area(s) and
format that the course designers were provided with was a
why?
context-rich blending of a) a problem list and b)
corresponding redesign proposals [25]. Presumably, this Q8: Does the information provided in Table 1 have an
can better illustrate the causes and solutions of the UX- impact on your (re-)design strategy of particular UX-
related problem areas, and thus enhance the persuasiveness related problem areas of the online course Organizational
of such a context-rich feedback format as distinct from pure Management (here: prioritization of particular problem
areas)? (1 = not at all – 5 = absolutely). Please comment on
3
particular problem areas.
The mean values of the corresponding UX-related problem areas
are based on students’ ratings gathered within the realm of the
preliminary large-scaled online survey to evaluate their
acceptance with the online course Organizational Management.
Table 1: UX-related Problem Areas as Perceived by Students’ of the Online Course “Organizational Management”: Overview
A. ISSM-originated
Label** Definition** Item*/** Item-UX-mapping** Average Item Additional UX Problem Specification
(based on the model of Rating from the 1st
UX [6, 16]) survey/ Participants’ qualitative Evaluators’ Suggestion/
UX-related comments given in the UX-related item problem
problem areas second survey/ concretization**
item*/** UX Item-related Problem
Description**
User VLE are appealing if their
The VLE has an attractive beauty,
Interface graphical user interface has 3.36 - -
graphical user interface [5]. visual aesthetics
Appeal a pleasant appearance [11].
„Please clarify with your
students what exactly is
perceived to be incomplete,
i.e. missing sample solutions
and/or missing learning
I trust the learning material's materials? In case the latter
„Your learning materials are
originator (e.g. teacher, does apply, please clarify
arranged very neatly, but
professional trust 4.27 once again the chair’s
they are incomplete in
institution/organization) didactic policy, i.e. the initial
parts.”
[22]. set of slides provided
constitute a “starting point”
which has to be “enriched” by
the students’ themselves
The information provided (dilemma: student vs. teacher
by VLE is credible if they viewpoint).
Information originate from a trustworthy
Credibility source (e.g. teacher, „If applicable, emphasize the
certified and/or reputable chair’s competence in this
The learning material's
organizations, etc.) [21]. subject domain (e.g. insert a
originator (e.g. teacher,
„I am not able to judge if he MIS seal of approval on the
professional
trust 4.18 is a recognized source of learning materials/slides
institution/organization) is
information.“ provided which is associated
an (officially) approved
with the chair’s latest
source of information [22].
publications (papers,
textbooks), awards, etc.).”
„If applicable, insert a MIS
The learning material's
seal of approval on the
originator (e.g. teacher,
„I cannot answer this learning materials/slides
professional trust 4.09
question likewise.“ provided which is associated
institution/organization) has
with the chair’s reputation in
a good reputation [22].
this subject).”
„If applicable, please
Partly. It may be better to
The tasks contained (with)in consider open questions too
work with open questions
the learning materials arouse Stimulation 3.36 which should be discussed
that are discussed with the
my curiosity [19]. mutually in class (i.e.
tutor in class.
“offline”).”
If applicable, please consider
open questions too which
should a) require students’
The tasks contained (with)in
The information provided initiative for further research
the learning materials arouse competence 3,55 [“Partly.”]
by VLE is challenging if the (i.e. stimulate self-directed
Information my ambition [22].
learning materials contain learning processes) and which
Challenge difficult but interesting tasks should b) discussed mutually
which stimulate learners’ in class (i.e. “offline”).”
curiosity to solve them [21].
„The tasks seem to be far
„Please carify if the sample
away from the aspiration
exercises do have the same
level of the final
difficulty level as the ones
The tasks contained (with)in examination. Would be
provided in the final
the learning materials are competence 3,36 better to provide questions
examination. If not, please
appropriately tricky [22]. which do have the same
upload a mock exam which
difficulty level as the ones
should be discussed mutually
asked in the final
in the final tutorial.”
examination.”
B. TAM3-originated
Label** Definition** Item*/** Item-UX-mapping** Average Item Additional UX Problem Specification
(based on the model of Rating from
UX [6, 16]) the 1st survey/ Participants’ qualitative Evaluators’ Suggestion/
UX-related comments given in the second UX-related item problem
problem areas survey/ concretization**
item*/** UX Item-related Problem
Description**
„If applicable, communicate the
benefits of using the VLE
The degree to which an People who influence my (with)in the course to your
individual perceives that behavior (e.g. fellow Those who are not studying at students (e.g. citing empirical
Subjective most people who are students, friends, parents, the university may not deal, studies which proved the VLE
identification 2.00
Norm important to him think he etc) think that I should use and thus may not be to have a positive impact on
should or should not use the the VLE within the scope of interested in the VLE.” students’ training success) so
system [4, 33]. my studies [22, 30, 32]. that most of them may perceive
the use of the VLE/online
course to be “obligatory”.
„If applicable, please point to
the innovative kind of teaching
method which is applied
(with)in the course so that
Fellow students at my
students may get more aware of
university who use the VLE
identification 1.18 „Strange question again!“ the novelty of the approach
have more prestige than
The degree to which an undertaken (as long as this
those who do not [20, 22].
individual perceives that use effect may endure, a
of an innovation will “demarcation” from other
Image fellow students may be
enhance his or her status in
his or her social system possible).”
[20].
Fellow students at my
„One does have a high profile
university who use the VLE identification 1.18 -
when using the VLE?”
have a high profile [20, 22].
Having the VLE is a status
symbol at my university [20, identification 1.27 “What?” -
22].
“In order to avoid any kind of
inhibition in dealing with the
„Sure…the use of the VLE VLE/the online course, a
VLEs do not scare me at all
evocation (negative) 4.00 (inverse) was explained very well by compulsory introduction in the
The degree of “an [31].
the tutor!“ use of the VLE/online course
individual’s apprehension, should be implemented at the
Computer
or even fear, when she/he is beginning of the semester.”
Anxiety
faced with the possibility of
using computers” [31]. Working with a VLE makes
evocation (negative) 1.18 (inverse) “Nervous?” -
me nervous [31].
VLEs make me feel “This question is pretty
evocation (negative) 1.27 (inverse) -
uncomfortable [31]. strange too!”
“In order to avoid any kind of
inhibition in dealing with the
I would characterize myself VLE/the online course, a
„Just do it, don’t think about
very spontaneous when I use stimulation 3.64 compulsory introduction in the
it!“
the VLE [31]. use of the VLE/online course
“….the degree of cognitive should be implemented at the
Computer spontaneity in beginning of the semester.”
Playfulness microcomputer interactions”
I would characterize myself “Strange question likewise.
[35].
very creative when I use the stimulation 2.64 What do you mean by -
VLE [31]. creative?”
I would characterize myself Playful? Do not know how to
very playful when I use the stimulation 2.45 interpret and answer this -
VLE [31]. question?”
RESULTS
Course Designers’ Ratings of the Feedback Format of particular UX-related problem areas. However, in order
Table 2 summarizes the quantitative ratings per question to solve the problem the evaluator’s suggestions are
(Q1-Q8) by providing the overall persuasiveness of the inevitable. Furthermore, the information provided in Table
feedback format (see Table 1) per course designer, the 1 was considered to give valuable hints, which, however,
corresponding mean values, standard deviations and would need more in-depth information to understand what
perceived persuasiveness of UX-related problem areas. actually the problem was (e.g. Is the problem of
Course Designers’ Comments on the Feedback Format understanding the material caused by the fact that questions
Supplement to course designers’ overall ratings of the do not have examination-level? Or is the VLE
feedback format (see Table 2) the following description misunderstood as a mere examination-preparation-tool?).
reveals how course designers perceived the persuasiveness Concerning the horizontal evaluation of the information
of particular UX-related problem areas. provided in Table 1, course designers found that the
Perceived Usefulness of the Information Provided (Q1) following UX-related problem areas were easy to
All in all, the item ratings and evaluator’s suggestions are understand: Computer playfulness, user interface appeal,
considered to be useful to address the following UX-related information credibility and information challenge. On the
problem areas, namely computer playfulness and computer other hand, image and subjective norm were considered to
anxiety. Furthermore, students’ critique concerning be very intangible and not reasonable at all.
information challenge (i.e. provision of sample tests) was Assessing the Severity of UX-related Problem Areas (Q3)
assumed to originate from the lack of publicity for the According to Hertzum [9], the severity of a UX-related
announcement of such tests which were published in the problem area “is an assessment of the amount of trouble
VLE on a regular base. Besides, one of the course designers and inconvenience users will experience as a result of a
raised his concerns regarding image as a useful UX-related specific aspect of a system. Severity assessments are,
problem area as it was considered to be not related to the however, also recommendations about the urgency of
course designers’ work as a lecturer or author of the course. fixing problems”. In the context of the current study, our
Understandability of UX-related Problem Areas (Q2) concern was whether the feedback could facilitate the
With regard to the course designers’ vertical evaluation of prioritization of UX-related problem. In other words, .the
the understandability of individual variables in the evaluation feedback was intended to persuade the course
feedback format presented in Table 1, the following results designers to fix problem areas of different levels of severity
were obtained: The item as well as the combination of item (here: ranging from 1.00 – 4.50) (cf. [14]).
and students’ comments per UX-related problem area was Regarding the severity assessments of the given UX-related
found to a) give the first idea that there exist particular UX- problem areas (Q3 in Table 2), course designers did not
related problem areas and to improve the understanding evaluate the construct computer anxiety. In addition to that,
Table 2: Perceived Persuasiveness of UX-related subjective norm and image were considered not to
Problem Areas Feedback Format constitute severe UX-related problem areas. On the other
Question Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
hand, the remaining set of UX-related problem areas (user
Designer #1 4 3 2 2,5 5 5 2 4
interface appeal, information credibility, information
challenge and computer playfulness) was considered to
Overall
Rating
Designer #2 4 4 4 5 4 5 5 5
constitute severe UX-related problem areas which should
Designer #3 4 3 3 4 2 4 4 4
be addressed to improve the course for the forthcoming
Mean Values 4.00 3.33 3.00 3.83 3.67 4.67 3.67 4.33
semester. However, in order to improve course designers’
Standard severity assessments, they required the feedback format to
0.00 0.58 1.00 1.26 1.53 0.58 1.53 0.58
Deviation
contain even more explicit students’ severity rankings in
Perceived Persuasiveness of Particular UX-related Problem Areas
order to understand how severe a problem was actually
User Interface
+ + 0 + perceived by them.
Appeal
Capability of the Feedback to Solve UX Problems (Q4)
Information
+ + 0 + + Concerning the capability of the information provided in
Credibility
Information Table 1 for solving the UX-related problem areas, the
- + + 0 + + ++
Challenge course designers appreciated the way the information was
Subjective Norm - 0 + - presented (“the table does provide the causes and solutions
Image - - 0 ++ - of the problems”), and here especially evaluators’
Computer Anxiety + 0 ++ + suggestions even though they were considered to be “not
Computer operative enough”. Thus, the item was considered to give
+ + + 0 ++ +
Playfulness “an idea that there exist particular UX-related problem
- = UX problem areas considered not to be persuasive areas, in order to solve them the evaluator’s suggestions are
inevitable.”
0 = neutral
+ = UX problem areas considered to be persuasive
Course Designers’ Intention to Solve UX Problems (Q5) related problem areas listed in Table 1 as “real” problems
Concerning the course designers’ intention to solve the (Q5), and they were found to be not operative enough (Q7).
UX-related problem areas illustrated in Table 1, the
In particular, the findings revealed that computer anxiety
variation was relatively large (i.e. the column Q5 in Table
and computer playfulness were perceived as very useful
2, Mean = 3.67, SD = 1.53). Specifically, we computed the
(Q1). Furthermore, the feedback regarding user interface
so-called impact ratios [28] per course designers:
appeal, information credibility, information challenge and
Number of problems committed to be fixed *100 computer playfulness helped course designers to understand
Total number of problems found the corresponding UX-related problem areas (Q2), had an
The results range from one course designer showing an impact on the course designers’ severity assessments of the
impact ratio of 14% (“The UX-related problem areas corresponding problem areas (Q3), their intention to solve
presented do not constitute real problems […] So why the UX-related problem areas revealed (Q5, + subjective
should I solve them?”) to the other two course designers norm, image and computer anxiety) as well as the
showing an impact ratio of 100% (“I’ll try to tackle all persuasiveness of the UX-related problem areas (Q6)
problems so as to improve the course and contributing to illustrated in Table 1. Moreover, information challenge and
good learner relations”; “Depending on the resources computer anxiety were considered to be the two most
available one could tackle each of the UX-related problem critical UX-related problem areas to be addressed within
areas illustrated, especially to foster students’ computer the forthcoming re-design of the online course (Q8).
playfulness and their perceived image of using the online Finally, in order to solve particular UX-related problem
course”). However, given that the re-design of the online areas as revealed by use of students’ item ratings (see Table
course takes place within the upcoming semester break, the 1, column 5), course designers required the feedback
completed-to-date impact ratio is out of scope of this paper format to contain evaluator’s suggestions (Q4).
[28]:
Regarding the research questions addressed in Introduction,
Number of problems committed receiving a fix *100 we revisit them here with reference to the empirical
Total number of problems found findings gathered:
Persuasiveness of the UX Problems (Q6) RQ1: Students had more problems in specifying UX-
Two of the three course designers considered the following related problem areas based on TAM3-related UX items
UX-related problem areas 1 to constitute no “fake than those based on their ISSM-related counterparts (see
problems” (see Table 1): Information challenge, the column Participants’ qualitative comments in Table 1).
information credibility and computer playfulness. On the In particular, students struggled in specifying TAM3-
contrary, image and subjective norm were considered not to related UX problem areas such as image (“One does have a
constitute real UX-related problem areas. This was mainly high profile when using the VLE?”), computer anxiety
due to the fact that course designer could not relate them to (“This question is pretty strange too!”) and computer
the course. playfulness (“Playful? Do not know how to interpret and
Ease of Use of the Feedback to Solve UX Problems (Q7) answer this question?”), whereas this was only the case for
In total, the feedback was considered to be not operative information credibility („I am not able to judge if he is a
enough. One of the course designers remarked that ”for recognized source of information“) regarding ISSM-
instance, knowing that students perceive materials as anchored UX problem areas.
incomplete does not help me which concrete information
RQ2: As depicted in Table 2, course designers considered
lacks, in which part and why?”
UX-related problem areas originated in the ISSM to be
Impact on the Prioritization of UX Problems (Q8) more persuasive than their TAM3-related counterparts.
In a nutshell, the feedback format helped course designers This may be mainly due to the fact that course designers
to classify the subsequent UX-related problem areas as did not perceive the “intangible” TAM3-anchored UX-
critical, namely computer anxiety and information related problem areas as relevant to their particular course
challenge. The corresponding fixing plan was to formulate (e.g. subjective norm and image).
exercises more precise and understandable.
RQ3: Course designers perceived UX-related problem
DISCUSSION areas anchored in TAM3 or the ISSM to be most persuasive
The present study provides a systematic evaluation of how in case evaluators’ suggestions (see Table 1, column 7)
course designers perceived the persuasiveness of a were provided in addition to the other UX problem area-
theoretically-grounded feedback format concerning related contextual information provided in Table 1 (see
students’ UX with a university’s online course. Q1/4: “in order to solve the problem the evaluator’s
In total, course designers’ mean ratings across the eight suggestions are inevitable”). No significant differences
questions in the questionnaire were all above 3.00 with between TAM3- and ISSM-anchored UX-related problem
standard deviations ranging between 0.00 (usefulness of the areas were reported by the course designers.
feedback) and 1.53 (intention to solve UX-related problem IMPLICATIONS
areas; ease of use of the feedback to solve particular UX- The above mentioned results should generally provide a
related problem areas). The large variations are due to the starting point for future research. In particular, future
fact that one course designer did not perceive the UX- research efforts should focus on elaborating ways how to
formulate items related to TAM3 constructs such as Conference on Web-Age Information Management
subjective norm and image so that they may better relate to (WAIM 2007), LNCS 4505, 670-677.
(course) designers’ concrete work. Furthermore, course [6] Hassenzahl, M. The interplay of beauty, goodness, and
designers’ severity assessments as well as their evaluation usability in interactive products. Human-Computer
of the persuasiveness of the realness of such problem areas Interaction, 19 (2004), 319-349.
may be improved and facilitated by the use of more explicit
students’ severity ratings (i.e. underline the meaning of [7] Hassenzahl, M., and Sandweg, N. From mental effort
students’ item ratings). The main benefit of further refining to perceived usability: Transforming experiences into
UX-related items anchored in theory-grounded constructs summary assessments, in Proceedings of CHI ’04
may be the improvement of the transparency and (New York, NY, 2004), ACM Press, 1238-1286.
comparability of the corresponding research outcomes. [8] Hassenzahl, M., and Tractinsky, N. User Experience:
In addition, as the capability of the feedback to solve UX- A research agenda. Behavior & Information
related problem areas was considered to be limited due to Technology 25, 2 (2006), 91-97.
its lack of information richness, future research work [9] Hertzum, M. Problem prioritization in usability
should investigate which potentially persuasive elements evaluation: from severity assessments toward impact
need to be included in a re-design proposal for fixing UX- on design. International Journal of Human-Computer
related problems. Interaction 21, 2 (2006), 125-146.
CONCLUSION [10] Hevner, A.R., March, S.T., Park, J., and Ram, S.
In this paper a systematic evaluation of how course Design science in information systems research, MIS
designers’ perceived persuasiveness of a theoretically Quarterly 28, 1 (2004), 75-105.
grounded feedback format was carried out. Specifically, a [11] Hong, W., Thong, J.Y.L., Wong, W.-M., and TAM,
problem list with corresponding redesign proposals of K.-Y. Determinants of user acceptance of digital
TAM3- and ISSM-anchored UX-related problem areas was libraries: An empirical examination of individual
identified and evaluated. Outcomes of this study will differences and system characteristics. Journal of
presumably stimulate future research on resolving UX Management Information Systems, 18, 3 (2001-2002),
problems. In particular, the combination of quantitative and 97-124.
qualitative data can not only gain better insights into issues
but also support future (course) design and evaluation [12] Isleifsdottir, J., Larusdottir, M. Measuring the user
efforts that may contribute to students’ positive UX while experience of a task oriented software, in Effie L-C.
interacting with a VLE and online courses. Law, Nigel Bevan, Georgios Christou, Mark Springett,
and Marta Lárusdóttir (eds.), Proceedings of the
ACKNOWLEDGMENTS International Workshop on Meaningful Measures:
We thank Anke Diederichsen for the review and pretest of Valid Useful User Experience Measurement ’08
the course designers’ questionnaire, Christian Gasper and (Reykjavik Iceland, June 2008), 97-101.
his research assistant for providing their persuasiveness
[13] ISO 9241-110. Ergonomics of human-system
with the feedback format.
interaction - Part 110: Dialogue principles (2006), ISO
REFERENCES Copyright Office, Geneva, Switzerland.
[1] Bevan, N. What is the difference between the purpose
[14] Law, E. L-C. Evaluating the downstream utility of user
of usability and user experience evaluation methods, in
tests and examining the developer effect: a case study.
Proceedings of the Workshop UXEM’09 (Uppsala
International Journal of Human-Computer Interaction
Sweden, 2009).
21, 2 (2006), 147-172.
[2] DeLone, W.H., and McLean, E.R. Information systems
[15] Law, E. L-C., Roto, V., Hassenzahl, M., Vermeeren,
success: The quest for the dependent variable.
A., and Kort, J. Understanding, scoping and defining
Information Systems Research 3, 1 (1992), 60-95.
user experience, in Proceedings of CHI ’09 (Boston,
[3] DeLone, W.H., McLean, E.R. The DeLone and USA, April 2009), 1-10.
McLean model of information systems success: A ten-
[16] Law, E. L-C, and van Schaik, P. Modelling user
year update. Journal of Management Information
experience: an agenda for research and practice.
Systems 19, 4 (2003), 9-30.
Interacting with Computers Interaction in press
[4] Fishbein, M., and Ajzen, I. Belief, Attitude, Intention (2010), 1-10.
and Behavior: An Introduction to Theory and
[17] Lee, G.T., Dahlan, N., Ramayah, T., Karia, N., and
Research. Addison-Wesley, Reading, MA, 1975.
Hasmi Abu Hassan Asaari, M. Impact of interface
[5] Fu, F.-L., Chou, H.-G., Yu, S.-C. Activate interaction characteristics on digital libraries usage. Malaysian
relationships between students’ acceptance behavior Online Journal of Instructional Technology 2, 1
and e-learning, in Dong, G., Lin, X. Wang, W., Yang, (2005), 1-9.
Y., Xu Yu, J. (eds.), Joint 9th Asia-Pacific Web
[18] Lindgaard, G. Usability Testing and System
Conference (APWeb 2007) and 8th International
Evaluation: A Guide for Designing Useful Computer
Systems. Chapman & Hall, London and New York, [27] Petter, S., DeLone W.H., and McLean, E.R. Measuring
1994. information systems success: Models, dimensions,
[19] Martínez-Torres, M.R., Toral Marín, S.L., Barrero measures, and interrelationships. European Journal of
Garciá, F., Gallardo Váquez, S., Arias Oliva, M, Information Systems 17 (2008), 236-263.
Torres, T. A technological acceptance of e-learning [28] Sawyer, P., Flanders, A., and Wixon, D. Making a
tools used in practical and laboratory teaching, difference: The impact of inspections, in Proceedings
according to the European Higher Education Area. of CHI '96 (Vancouver BC, April 1996), ACM Press,
Behaviour & Information Technology 27, 6 (2008), 376-382.
495-505. [29] Seddon, P.B., Staples, S., Patnayakuni, R., Bowtell, M.
[20] Moore, G.C., and Benbasat, I. Development of an Dimensions of information success. Communication of
instrument to measure the perceptions of adopting an the Association for Information Success 2 (1999), 2-39.
information technology innovation. Information [30] Taylor, S., and Todd, P.A. Understanding information
Systems Research 2 (1991), 192-222. technology usage: A test of competing models.
[21] Mueller, D., and Strohmeier. Design characteristics of Information Systems Research 6 (1995), 144-176.
virtual learning environments: An expert study, [31] Venkatesh, V. Determinants of perceived ease of use:
International Journal of Training and Development Integrating perceived behavioral control, computer
(2010), in press, 1-14. anxiety and enjoyment into the technology acceptance
[22] Mueller, D., and Strohmeier, S. Developing and model. Information Systems Research 11 (2000), 342-
validating a rigorous and relevant model of VLE 365.
success: A learner perspective, Proceedings of ECTEL [32] Venkatesh, V., and Bala, H. Technology acceptance
’10 (Barcelona Spain, September/October 2010), in model 3 and a research agenda on interventions.
press, 1-16. Decision Sciences 39, 2 (2008), 273-315.
[23] Mueller, D., and Zimmermann, V. A learner-centred [33] Venkatesh, V., and Davis, F.D. A theoretical extension
design, implementation, and evaluation approach of of the technology acceptance model: Four longitudinal
learning environments to foster acceptance. field studies. Management Science 46 (2000), 186–
International Journal of Advanced Corporate Learning 204.
2 (2009), 50-57.
[34] Venkatesh, V., Morris, M., Davis, G., and Davis, F.
[24] Nielsen, J. Heuristic evaluation, in J. Nielsen, and R.L. User acceptance of information technology: Toward a
Mack (eds.), Usability Inspection Methods. Wiley, unified view, MIS Quarterly 24, 3 (2003), 425-478.
New York, 1994, 25-62.
[35] Webster, J., and Martocchio, J.J. Microcomputer
[25] Norgaard, M., and Hornbaek, K. Exploring the value playfulness: Development of a measure with
of usability feedback formats. International Journal of workplace implications. MIS Quarterly 16 (1992),
Human-Computer Interaction 25, 1 (2009), 49-74. 201-226.
[26] Oxford Dictionaries. Available at: [36] Wixom, B.H, and Todd, P.A. A theoretical integration
http://oxforddictionaries.com/view/entry/m_en_gb029 of user satisfaction and technology acceptance.
0410#m_en_gb0290410. Information Systems Research 16, 1 (2005), 85-102.