=Paper=
{{Paper
|id=Vol-1910/paper0207
|storemode=property
|title=Comparison of UX Evaluation Methods that Measures the UX Over Time
|pdfUrl=https://ceur-ws.org/Vol-1910/paper0207.pdf
|volume=Vol-1910
|authors=Ayako Hashizume,Masaaki Kurosu,Yuuki Ueno
|dblpUrl=https://dblp.org/rec/conf/chitaly/HashizumeKU17
}}
==Comparison of UX Evaluation Methods that Measures the UX Over Time==
https://ceur-ws.org/Vol-1910/paper0207.pdf
Comparison of UX Evaluation Methods
That Measures the UX over Time
Ayako Hashizume Abstract
Tokyo Metropolitan University Evaluation of UX is deeply related to the concept of UX.
Tokyo, Japan UX cannot be evaluated directly from the quality in
hashiaya@tmu.ac.jp design but should be evaluated from the quality in use.
As a result, the evaluation of UX should use different
Masaaki Kurosu tools from what usability professionals have been using
The Open University of Japan for evaluating the usability. There are two types of UX
Chiba, Japan evaluation methods, one is the real-time method and
masaakikurosu@spa.nifty.com another is the retrospective method based on memory.
Because of the difficulty of conducting the real-time
Yuuki Ueno method for a long time, authors adopted the
Otsuka Business Service retrospective approach. We first focused on the UX
Tokyo, Japan Curve and revised it to become the UX Graph. Authors
uenoyuuki@gmail.com analyzed the accumulated data obtained so far by
applying the UX Graph and concluded that the time
dimension (horizontal axis of the graph) and the graph
itself was not of much importance. Instead, they
proposed the Experience Recollection Method (ERM)
which accepts ambiguity in the time of use. In this
paper, we compared UX evaluation methods, and
Copyright is held by the author/owner(s). clarified the advantages of the ERM.
CHItaly ’17, September 18-20, 2017, Cagliari, Italy.
Author Keywords
user experience (UX); evaluation; Experience
Recollection Method (ERM); rating scale, satisfaction
ACM Classification Keywords
H.5.2 User Interfaces Evaluation/methodology.
Introduction A New Method for UX Evaluation the “ERM”
UX is the perception and reaction of users in the real The UX Curve is a method to let users draw the curve
environment for real products and services. UX from the start of use till the time of measurement in
evaluation should adopt tools and methods that are terms of the attractiveness, usability, utility and the
different from what have been used for evaluating the frequency of use by assigning those values to the
usability. Although many evaluation methods have vertical axe and time to the horizontal axe. While it is
been proposed as can be found on the “AllAboutUX” adequate to ask real users their experiences in the real
web site [1], there are many usability evaluation context, to provide the curve that is visual and
methods intermingled with the UX evaluation methods impressive and to facilitate the grasp of the general
with little consideration of the difference between the tendency by a glance, it has some issues that should be
concept of usability and UX. Based on our model of reconsidered as follows: (1) the focus is rather on the
usability and UX, we defined that the usability is a part curve itself and the content of each episode is not much
of the “quality in design” and the UX is related to the focused, (2) drawing similar three curves in addition to
“quality in use” [7]. the frequency is time-taking and boring. Thus authors
developed the UX Graph with the idea of improving the
The purpose of UX evaluation is to grasp changes in UX Curve [5].
actual experiences; hence short-term UX evaluation is
not sufficient. Among methods used for evaluating UX, In the UX Graph, only the satisfaction that is regarded
there are real-time methods and retrospective methods. as the same with ‘utility’ in the context of economics is
Real-time methods include the use of the Experience used as the sole measure. In the study on the concept
Sampling Method (ESM) [4] using mobile phone, but of satisfaction [6], it is described that the satisfaction is
it’s difficult to conduct for a long time. On the other an integrative concept of hedonic aspects and all the
hands, retrospective methods including UX Curve and quality characteristics as well. Besides, the coordinate
UX Graph fit well to the goal of UX evaluations. Because of each episode is thought to be important and the
we thought that the ultimate evaluation measure is the graph is drawn connecting those points on the sheet. In
satisfaction [6], we adopted it as the scale for other words, the graph as the visual expression is just
evaluating the UX [8]. Thus we proposed the UX Graph in a supplementary position. A WEB tool was developed
method [2, 8] as a revised version of UX Curve [5], and to draw the UX Graph. Based on the collection of data,
then, the ERM (Experience Recollection Method) [8, 7] authors summarized the evaluation of the UX Graph as
as an advanced method rooted in the UX Graph. follows: (1) the visual expression is attractive but is not
much substantially important, and (2) the memory of
In this paper, we picked up typical timeline-based [9] informants in terms of the time is ambiguous and the
UX evaluation methods and made a comparison among value on the horizontal axes changes the shape of the
them. graph drastically.
Thus we decided to waste the
visual expression and not to ask
the exact time to informants.
This is the process how we
developed the Experience
Recollection Method (ERM) [8, 7].
In the ERM, informants are not
asked to draw the curve nor the
graph. Time line is not the exact
year but the roughly sectioned
periods as can be seen in Figure
1. Informants are requested to
write down the episodes with the
satisfaction rating from +10 to -
10.
Comparison of UX
Evaluation Methods
Table 1 summarizes UX
evaluation methods that measure
the UX over time. Following Figure 1: An example of ERM regarding the using smartphone
methods are included in the table:
a. b. c.
Nature of Method Type of Method Phase
Method Quantitative PC- Purchase / Middle
Qualitative Verbal Hand-written Expectation First Phase Last Phase Current Future
(Timeline) based Obtain Phase
CORPUS
[11] ○
(interview) ○ ○ ○ ○ ○ ○ ○
iScale
[3] ○ ○ ○ ○ ○ ○ ○
UX Curve
[5] ○ ○ ○ ○ ○ ○ ○
UX Graph
[2, 9] ○ ○ (both) ○ ○ ○ ○ ○ ○ ○ ○
ERM
[9, 8] ○ ○ (both) ○ ○ ○ ○ ○ ○ ○ ○
Table 1: Comparison of six timeline-based UX evaluation methods
CORPUS [10], iScale [3], UX Curve [5], UX Graph [2, References
8], and ERM [8, 7]. 1. AllAboutUX http://www.allaboutux.org/all-methods
2. Ayako Hashizume and Masaaki Kurosu. 2016. "UX
The column of Table 1 (a.-e.) are following aspects for Graph Tool" for Evaluating the User Satisfaction,
clarifying the characteristics of each method. International Journal of Computer Science Issues,
13(5): 86-93.
a. Nature of Method - Qualitative (i.e. interview) / 3. Evangelos Karapanos, John Zimmerman, Jodi
Quantitative (i.e. numerical measures be obtained) Forlizzi and Jean-Bernard Martens. 2010.
Measuring the Dynamics of Remembered
b. Type of Method – Verbal / Hand-written / PC-based
Experience Over Time, Interacting with Computers,
c. Period – Expectation / Purchase or Obtain / First doi: 10.1016/j.intcom.2010.04.003
phase / Middle phase / Last phase / Current /
4. Reed Larson and Mihaly Csikszentmihalyi, 1983,
Future (i.e. regarding which time phase, the The experience sampling method, Jossey-Bass, 41-
method can get the data) 56.
5. Sari Kujala, Virpi Roto, Kaisa Vaananen-Vainio-
Regarding “b. Type of Method”, PC-based method Mattila, Evangelos Karapanos and Arto Sinnela.
facilitates the acquisition of data from informants living 2011. UX Curve: A Method for Evaluating Long-
in the remote area if it is conducted online by using the term User Experience, Interacting with Computers,
internet. In this respect, the iScale (if it can be used on doi:10.1016/j.intcom.2011.06.005
the internet), UX Graph and ERM are better. Regarding 6. Masaaki Kurosu, 2015, Satisfaction as a Measure of
“h. Phase”, it is necessary to include the expectation Quality in Use for Kansei Experience, International
because it should be regarded as a part of UX as in the Symposium on Affective Science and Engineering
2015 Proceedings
UX Graph and ERM.
7. Masaaki Kurosu, 2016, Theory of User Engineering,
Comments for ERM CRC Press
Comments obtained from 53 university students 8. Masaaki Kurosu, Ayako Hashizume, Yuuki Ueno,
included positive comments such as “the table (not the Tuyoshi Tomida and Hirotoshi Suzuki, 2016, UX
Graph and ERM as Tools for Measuring Kansei
graph or curve) is suitable for reminding the past
Experience, HCI International 2016 Proceedings
events”, “it is easy to use because episodes are not
9. Tonya J. Tidline, 2005, Dervin’s Sense-making, in
always be remembered in the chronological order”, etc.
Karen E. Fisher, Sanda Erdelez, and Lynne
McKechnie (eds.), Theories of Information Behavior
Conclusion (Asist Monograph), Information Today Inc.
Considering the characteristics of each UX evaluation
10. Margeritta von Wilamowitz-Moellendorff, Marc
method, each method has its own uniqueness. ERM, of Hassenzahl, and Axel Platz, 2006, Dynamics of
course, has its own uniqueness and merits, thus should User Experience: How the Perceived Quality of
be used by many practitioners in the real life UX Mobile Phones Changes Over Time”. pp, 74–78.
situation.