=Paper=
{{Paper
|id=None
|storemode=property
|title=The Relevance of UX Models and Measures
|pdfUrl=https://ceur-ws.org/Vol-656/paper1.pdf
|volume=Vol-656
|dblpUrl=https://dblp.org/rec/conf/iused/Folstad10
}}
==The Relevance of UX Models and Measures==
https://ceur-ws.org/Vol-656/paper1.pdf
The Relevance of UX Models and Measures
Asbjørn Følstad
SINTEF
Po. Box 124, Blindern
0314 Oslo
+47 22067515
asf@sintef.no
ABSTRACT The second approach represent an intermediary position to
Two approaches to research on UX models and measures what has been referred to as phenomenological/pragmatist
are discussed on basis of experiences from the field of vs. inspired by experimental psychology [2]. This approach
usability research and an ongoing case of user involvement is exemplified by a case from an ongoing research project.
in software development (SD) by way of social media. It is
The intended contribution of the paper is to serve as a
suggested that simple measures and ad-hoc models, rather
starting point for discussions on the relevance of UX
than complex models and measures, may be beneficial to
research.
the relevance of UX research for SD practice.
APPROACH 1: COMPLEX MODELS AND MEASURES
Categories and Subject Descriptors
A recurring theme of UX research discussions is the
H5.m. Information interfaces and presentation (e.g., HCI):
components of UX [2]. Which model components are
Miscellaneous.
needed for measurement and systematic UX improvement?
Given the comprehensiveness of the UX concept [3] the set
Keywords
of model components is likely to be voluminous, indicating
User experience, software development, model, measure
that future UX models well may be complex. Suggested
components include, for instance, motivation, trust,
INTRODUCTION
aversion, hedonics, and fun [2].
A key objective of research on user experience (UX)
models and measures should be to influence the practice of This trend towards complexity mirrors parts of the usability
software development (SD). However, to reach this research during the 80’es and 90’es. A large number of
objective, UX research need to address research relevance usability components were suggested, including Nielsen’s
as well as rigor. efficiency, satisfaction, learnability, memorability, and error
rate [4]. A similar complexity is found in measures such as
A useful discussion of rigor and relevance in research was
SUMI [5]. ISO 9241-11 [6] decomposes usability in
made by Lee [1] within the context of information systems
effectiveness, efficiency and satisfaction.
(IS). He argued that knowledge produced by IS research
emulating the rigorous natural sciences does not by The complexity of usability models and measures is
necessity imply relevance. Rather, for the research to be beneficial for the rigor of usability research. However,
relevant it needs to generate “knowledge about how to complexity may be detrimental to the relevance for
intervene in the world and change it in order to satisfy real usability models in SD. For SD practitioners, complex
world needs” (p. 29). generic models may be impractical. The generic model may
include several components that seem irrelevant to the
In this paper, I will discuss two distinct approaches to
development project at hand. Also, seemingly important
research on UX models and measures, and their
aspects may not be handled by the model. One may
implications for research rigor and relevance. The first
speculate that the prevalence of homegrown usability
approach, complex models and measures, is what I see as a
measures at the expense of standardized measures [7] may
likely trend given that relevance is not prioritized in UX
be a consequence of the complexity of the underlying
research. The second approach, simple measures and ad-
usability models and their mismatch with the SD
hoc models, is suggested as an alternative.
practitioner’s understanding of which aspects of usability
The first approach is paralleled in previous work of the that are relevant.
related field of usability research. In order to learn from the
Sauro and Kindlund [8], a key critic of the relevance of
experiences made in this more mature field, my argument
usability research, argued that to increase the practical
will be supported by reference to research on usability
impact of usability data “usability metrics need to be easier
models and methods.
to use” (p. 401). To this end they suggested a single
summative usability metric.
The position of Sauro and Kindlund is controversial [9] comments prior to making their own and half of them were
though it recently has been underpinned by substantial not. All participants were allowed to comment on each
empirical evidence from industry projects [10]. other’s comments. None were allowed to see the others’
Controversial or not, an important lesson may be learnt ratings prior to making their own.
here for research on UX models and metrics: Complex The user feedback provided the following key information:
models, underpinning complex measures, are likely to be
valuable to the advancement of UX theory. However, in • Differentiation between the concepts: Three of the
order to advance the interplay between UX and SD, concepts were rated low, three rated high. The ratings
simplified models and measures may be required. corresponded closely to the frequency of positive vs.
negative comments for each concept.
APPROACH 2: SIMPLE MEASURES AND AD-HOC • Establishment of concept specific issues that may
MODELS
serve as basis for an ad-hoc UX model: For one of the
An alternative approach to UX models and measures may
concepts, detailing functionality for ‘Postponed sending
be pursued through simple measures and ad-hoc models.
of messages’, relevant issues were mainly targeting
A simple UX measure is a single rating scale, common in utility. For another, ‘Reading aid for long e-mails on
social software. Consider for instance the book ratings of small screens’, relevant issues were lack of comfort and
Amazon (1-5 stars) or video ratings at YouTube (thumbs utility. For a third, ‘Context-dependent e-mail receipt’,
up / down). Such scales are typically shunned for the relevant issues were privacy and utility.
scientific measurement of experiences or attitudes, due to
The case thus illustrates that simple measures and ad-hoc
reliability issues. Similarly, Sauro and Lewis argue for
models may serve as basis for choosing which design
composite rather than single item measures [9]. Even so,
alternative to pursue, and enable the establishment of ad-
such measures seem to serve their purpose as practical
hoc models that may be used to control the development
social navigation tools. Also, research within the field of
process. In the present case, a UX model for the first
marketing indicates that single-item measures may hold
concept would need to include only utility in addition to
similar predictive validity as multi-item measures for
usability, whereas the UX models for the second and third
concrete constructs such as ad liking and brand attitude
concepts would also need to include comfort and
[11].
privacy/trust respectively.
By ad-hoc UX models I mean models developed in
response to a given concept, prototype or running system. CONCLUSION
Instead of utilizing a general UX model as basis for a By this paper, I hope to contribute to a discussion on how
generic UX measure, relevant UX components may be research on UX models and measurements should be
established on basis of users’ responses. approached in order to obtain relevance. I hold that
complex models and methods will indeed be beneficial for
As an exemplification of simple measures and ad-hoc
the establishment of UX theory. However, research
models, I will briefly present an ongoing development case
relevance may require a different approach. Possibly, such
addressing new functionality for mobile phone e-mail
an approach may be simple measures and ad-hoc models.
clients. Design typically involves the “simultaneous
investigation of multiple alternatives by the same designer ACKNOWLEDGMENTS
or team” [12, p. 1243]. In the present case 22 ideas were This paper was written as part of the research project
generated and then refined as six early concepts; all across RECORD (www.recordproject.org), supported by the
a working period of 60 hours. Following this, the concepts VERDIKT programme of the Norwegian Research
were made available for user feedback. Council.
User feedback was collected through a social software
application for sharing audio-visual content, modified for REFERENCES
the purposes of design feedback. The participants were 212 [1] Lee, A. S. 1999. Rigor and relevance in MIS research:
regular users of e-mail-clients on mobile phones (use Beyond the approach of positivism alone. MIS Q. 23, 1,
several times a week or more). They were presented for six 29-33.
concepts in sequence. For each concept they were asked to [2] Law, E. L.-C., Vermeeren, A., Hassenzahl, M. and
make a rating (1-5 stars) and one or more comments. The Blythe, M. 2007. Towards a UX Manifesto.
comments were made in response to open questions: How Proceedings of Towards a UX Manifesto. COST294-
would you use the suggested function? How may the MAUSE affiliated workshop.
suggested function be improved?
[3] ISO. 2010. ISO 9241-210: Human-centred design of
As part of a research design not to be detailed here, half of interactive systems. Geneva: International standards
the participants were allowed to see the other participants’ organization.
[4] Nielsen, J. 1993. Usability engineering. Academic [9] Hornbæk , K., & Law, E. (2007). Meta-analysis of
Press, San Diego, CA. correlations among usability measures. Proceedings of
[5] Kirakowski, J. SUMI Background Reading: The Use of CHI’07. ACM, New York. NY, 617-626.
Questionnaire Methods for Usability Assessment. [10] Sauro, J., Lewis, J. R. 2009. Correlations among
Available at: http://sumi.ucc.ie/sumipapp.html Prototypical Usability Metrics: Evidence for the
(downloaded July 6, 2010) Construct of Usability. Proceedings of CHI’09. ACM,
[6] ISO. 1997. ISO 9241-11: Ergonomic requirements for New York, NY, 1609-1618
office work with visual display terminals (VDTs). Part [11] Bergkvist, L., and Rossiter, J.R. 2007. The predictive
11 – guidelines for specifying and measuring usability. validity of multi-item versus single-item measurements
Geneva: International standards organization. of the same constructs. Journal of marketing research,
[7] Hornbæk, K. 2006. Current Practice in Measuring 44, 175-184.
Usability: Challenges to Usability Studies and [12] Tohidi, M., Buxton, W., Baecker, R., and Sellen, A.
Research. International Journal of Human-Computer 2006. Getting the right design and the design right.
Studies, 64, 79-102. Proceedings of CHI’06. ACM, New York, NY, 1243-
[8] Sauro, J., Kindlund, E. 2005. A method to standardize 1252.
usability metrics into a single score. Proceedings of
CHI ’05. ACM, New York, NY, 401-409.