=Paper=
{{Paper
|id=None
|storemode=property
|title=To Measure or not to Measure UX: An Interview Study
|pdfUrl=https://ceur-ws.org/Vol-922/paper12.pdf
|volume=Vol-922
|dblpUrl=https://dblp.org/rec/conf/iused/LawS12
}}
==To Measure or not to Measure UX: An Interview Study==
https://ceur-ws.org/Vol-922/paper12.pdf
To Measure or Not to Measure UX: An Interview Study
Effie Lai-Chong Law Paul van Schaik
University of Leicester Teesside University
Dept. of Computer Science School of Psychology
LE1 7RH Leicester, UK TS1 3BA Middlebrough, UK
elaw@mcs.le.ac.uk P.Van-Schaik@tees.ac.uk
ABSTRACT solved in tandem. However, the definitional issue on UX
The fundamental problem of defining what UX is (or is not) remains unresolved, UX researchers and practitioners tend
has a significant influence on another challenging question: to select and adapt one of the many definitions out there to
to measure or not to measure UX constructs. The answer of serve their particular goals and needs. The recent efforts of
most, if not all, UX researchers and practitioners, would deepening the understanding of the theoretical roots of UX
probably be “It depends!” As we were motivated to find [10] can complement the earlier work on UX evaluation
out “depending on what”, we conducted semi-structured methods on the one hand [13] and the current
interviews with eleven UX professionals where a set of operatonalisation work for UX measurement on the other
questions in relation to UX measurement were explored. hand (e.g. [11]). As UX research studies have hitherto
Participants expressed scepticism as well as ambivalence relied heavily on qualitative methods [1], the progress on
towards UX measures and shared anecdotes related to such UX measures has thus been slow. A plausible reason is the
measures in different contexts. To improve the interplay scepticism about the measurability of UX.
between UX evaluation and system development, a clear
definition of UX, combining various data types, and robust The field of HCI in which UX is rooted has inherited
education in UX concepts are deemed essential. theoretical concepts, epistemological assumptions, values,
and methodologies from a diversity of disciplines, ranging
Author Keywords from engineering where measures are strongly embraced
User experience; Measurement; Interview; Feedback loop (cf. William Thomson’s [14] dictum ‘to measure is to
ACM Classification Keywords know’) to humanities where measures can be regarded as
H.5.m. Information interfaces and presentation (e.g., HCI): naïve or over-simplistic, especially when the concepts to be
Miscellaneous. measured are ill-defined, leaving (too) much for
interpretation [2]. As UX subsumes a range of fuzzy
General Terms experiential qualities such as happiness, disgust, surprise
Design; Evaluation; Measurement and love, controversies and doubts about the measurability
INTRODUCTION & BACKGROUND of UX are inevitable. A main divergence between two
To measure or not to measure concepts of interest? A clear major camps of UX researchers is the legitimacy of
cut “Yes!” if this question is raised in the context of breaking down experiential qualities into components,
physical sciences whereas an ambiguous “It depends!” rendering them to be measured; it is rooted in the age-old
when it is addressed in the context of social sciences in philosophical debate on reductionism versus holism.
general and the emerging research area of User Experience INTERVIEW ON UX MEASUREMENT
(UX) in particular. We aimed to explore such stipulations Instrument
(i.e. ‘depending on what’) for UX measures and their The interviews were semi-structured with 12 questions
implications to design and evaluation of interactive grouped into three main parts. Part A comprises four
systems. To meet this purpose, we conducted an empirical background questions (Table 1).
study in which eleven UX researchers and practitioners
were interviewed. In this paper we report some main Q1. Gender: Female, Male
findings of the study that are particularly relevant to Q2. Age: <=20, 21-30, 31-40, 41-50, >50
understanding the interplay between UX measurement and Q3: I am a: Practitioner, Researcher, Student, Other
Q4. How long have you worked in the area of UX? (Never, <1
iterative system redesign. Specifically, we adopt Hand’s
year, 1-3 year, 3-5 year, >5 year). Please describe the topic and
([4], p.3) definition of measurement “quantification: the related work.
assignment of numbers to represent the magnitude of Table 1. Background questions
attributes of a system we are studying or which we wish to Part B comprises five questions on the measurability of UX
describe.’ qualities (Table 2). The inclusion of Q5 is to know if the
The exploration of the issue of UX measurement was respondent’s understanding aligns with any of the existing
embarked on (e.g. [6]) after another, if not more, thorny definitions of measurement. For Q6, the rationale
issue of UX - its multiple definitions - had been examined underpinning each statement varies. The first one was
[7]. In principle these two foundational issues should be derived from the classic justification for measurement
advocated by Thomson [14]. The second and third ones Seven of them were female, five aged between 31 and 40,
were two rather extreme views against UX measures another five between 41 and 50 and one above 50. All were
expressed in some informal contexts (e.g. group discussion researchers except P5, who was a practitioner. The job of
in a workshop). They were aimed to stimulate thoughts and eight of the participants was predominantly design-oriented,
should not be treated as scientific claims. In contrast, the be it practical or theoretical, such as empathic design for
fourth and fifth statements represent views on the potential house renovation, co-design for persuasive games, and
uses of UX measures. They were deliberately broad in design theories. The other three focused more on UX
scope to stimulate discussions. evaluation of interactive products such as mobile phone.
Two of them have worked in UX for less than 1 year, three
Q5. What is a ‘measure’?
1-3 years, five 3-5 years and one for than 5 years. All the
Q6. (a) Please rate your agreement with each of the following
statements (5-point Likert scale); (b) Explain your ratings interviews were conducted on an individual basis in
UX measures lead to increase of knowledge English, audio-taped and transcribed subsequently.
UX measures are insane RESULTS AND DISCUSSIONS
UX measures are a pain For analysing the data, we developed coding schemes for
UX measures are important for design
UX measures are important for evaluation
individual interview questions by applying thematic
Q7. (a) Name a specific experiential quality (e.g., fun, surprise) analysis [3] and the CUE model [15]. Due to limited space,
that is most relevant to your work; (b) Explain the relevance; (c) here we do not report results of Q5 (What is a ‘measure’?).
Do you think the named quality can be measured: If ‘yes’, Statements on UX Measures
describe how; If ‘no’, describe why. Given the small sample size, no inferential statistics of the
Q8. (a) Name an experiential quality that you are (almost)
ratings are computed. Justifications for the ratings are of
certain is measurable; (b) How can it be measured and when
(before/during/after interaction)? (c) Why are you so (almost) higher relevance and the analyses are presented below.
certain about its measurability? What is your reservation, if any? UX measures lead to increase of knowledge (mean = 4.0,
Q9. (a) Name an experiential quality that you think (almost) range: 2-5). When prompted to specify which kinds of
impossible to measure; (b) Why do you think so? What is your
knowledge would be increased, several were mentioned,
reservation, if any?
references against which products can be compared;
Table 2. Five main questions on UX measures the extent to which the development goals is achieved;
The notion of “experiential qualities” is central for Q7, Q8 values to be delivered by certain design methods;
and Q9. In the simplest sense, they are referred to as information helpful for future projects;
feelings. In the broadest sense, they are related to the experience per se;
concept of emotional responses, as defined in the Ambivalence was observed, for instance: “There are ways
Components of User Experience (CUE) model [15], which to get knowledge about UX in a more meaningful way
are influenced by instrumental (i.e. usability) and non- rather than using measures, but I still think that they are
instrumental qualities (i.e. aesthetic, symbolic and important.” (P6). Besides, the need for including qualitative
motivational). While CUE focuses more on evaluation, in data as complementary knowledge was emphasized: “We
the context of the design the notion of experiential qualities should have both… qualitative is to know what the reason
is defined as articulations of key qualities in the use of a is for user experience and for the related design issue.”
certain type of digital artefact intended for designers to (P8). Furthermore, conditions for benefiting from UX
appropriate in order to develop their own work [8]. Note measures were specified: “It requires people using the
that in order to enable open discussion no definition was measure, understand the measure and what it actually
provided to the interviewees unless requests for means… There might be people who are not trained to use
clarification were solicited. Part C comprises three UX measures, no matter how well we define the measures.”
questions aimed to simulate in-depth discussion (Table 3). (P5). This observation highlights the need for enhancing
education and training in UX.
Q10. Which theoretical arguments (e.g. reductionism) are for or
against UX measurement?
UX measures are insane (mean = 2.0, range: 1-4). A
Q11. Which methodological arguments (e.g. validity) are for common view was that the insanity lies not in UX measures
or against UX measurement? but rather in what claims to be made about them, especially
Q12. Which practical arguments (e.g. cost) are for or against UX when people do not understand such measure, intentionally
measurement? misuse them, are unaware of their inherent limitations (e.g.
Table 3. Questions for in-depth discussions incompleteness) or over-formalize them. There were also
concerns whether UX measures can explain why people
Participant and Procedure
experience something or have any use for design, as
An invitation to the interview was circulated in the intranet
remarked by P11 (a designer):
of a university. Eight participants volunteered to take part in
it. The other three participants were recruited by the first “… for the purpose of design, measuring variables up to a very
high degree and intricate level of measurement might not be that
author via personal invitation. Their participations were
also voluntary. They were designated as P1, P2 and so on.
purposeful because you have to translate the numbers back to In summary, the interplay between UX measures, which are
design requirements, and I am not sure whether that works.” common evaluation outcomes, and (re)design is ambiguous.
UX measures are a pain (mean = 3.27, range: 1 – 5). Pain UX measures are important for evaluation (mean = 4.6,
inflicted was psychological rather than physical. Reasons range: 2-5). On this claim the participants were somewhat
for such pain varied with the phase of UX measurement. In less ambivalent. Supporting arguments such as justifying
the preparation phase, defining valid and meaningful decisions, validating design goal, and giving reliability (cf.
metrics, which entailed deep and wide knowledge of P2’s remark: “If you only use the designer intuition, only
various matters, was cognitively taxing and thus painful. use empathic interpretation, it is not very reliable for the
For data collection, participant recruitment and time rest of the world”) were given. Some participants pointed
constraint were a pain for researchers, as illustrated by P4’s out the time issue: in which development phase UX
remark: “We would not use half-an-hour to measure measures are taken and how much time the process of
something but rather get some qualitative data out of measuring is allowed, for instance:
participants.” On the other hand, the intrusiveness and “… in industry-led cases they are more keen on fast
lengthiness of the procedure could be pain for users. For phenomenon … the industrial people want to improve the design
data analysis, statistical analysis was deemed challenging but not really want to provide input for the academic world in
by four participants. This again is a clear implication for the general” (P4)
training of UX. Interpretation of UX measures was another There are also reservations about the role of UX measures
common concern: it could be an issue of lack of knowledge, in evaluation, for instance:
confirmation bias, and attempts to draw implications from
“it's not been proven yet that [UX measures] can make any
exact measures for design. difference to outcomes…. I mean, they could be; certainly if you
UX measures are important for design (mean = 4.0, range: include traditional usability measures, then persistent task failure
2-5). Participants’ stance on this claim was ambivalent. for many designs is going to be something you want to know
They recognized that UX measures could help identify about. But I don't think they're automatically important; they're
design constraints and justify design decisions by all hinges around design objects” (P11)
convincing developers and management, given that Measurable and Non-measurable Experiential Qualities
numbers could convey a sense of reliability. However, they In response to Q7, Q8 and Q9 (Table 2), participants
stipulated the importance of UX measures in design with identified different experiential qualities (EQ), which we
the need of combining with qualitative data, for instance: categorized by the adapted CUE model [15]:
“I mean they are important, but I’d not base my design solely Instrumental qualities (NQ) – “the experienced amount of
on UX measures... there are lot of things that I don’t think that support the system provides and the ease of use” (e.g.
we can measure properly enough yet… it would cause too much
controllability, learnability, effectiveness);
work to get really really good measurement that would be our
main basis for design… [UX measurement] would only be Non-instrumental qualities (NIQ) – “the look and feel of
second; the first being an overall understanding of qualitative the system”, including aesthetic, symbolic and
views we have found out from users.” (P4) motivational qualities ([15], p. 916; [9]);
“If UX measures are clusters that are described through numbers Affective responses (AR) – subjective feelings, motor
or questionnaires, then they are not important for design, expressions, and physiological reactions [12] arising from
whereas if UX measures are, for instance, clusters of qualitative interacting with the system (NB: It broadens the scope
data and users’ accounts, then they are important for design” implied by original notion of ‘emotional reactions’ to
(P11) accommodate mildly affective responses with an artefact).
Some participants explicitly expressed their doubt about the Evaluation (cf. system appraisal) – long-term effects of
role of UX measures in design, for instance: interacting with the system on user affect, attitude and
“I can see relatively little value of applying UX measures, cognition;
because they don’t really link to the product’s attributes in most Several interesting observations are noted:
cases… they link it at an abstract level… it is hard to trace what i) All three EQs considered as non-measurable fall into the
the underlying causes for certain response. It is almost category of Evaluation; it seems implying that long-term
impossible if we just use UX measures without combining them effects of interaction are considered not amenable to
with qualitative data” (P1)
measurement;
Furthermore, one participant pointed out the differences ii) No non-measurable instrumental and non-instrumental
between usability and UX measures: qualities were identified by the participants; this is not
“… sometimes it is difficult to explain why we design like this surprising as instrumental qualities are closely related to
even when we provide evidence. From usability point of view traditional software attributes that have explicitly been
we can more easily give this measurement that it is better, but operationalised and operationlising non-instrumental
designing for UX is problematic. People with technical qualities such as aesthetic and symbolic has been
backgrounds have problems making the difference between UI endeavoured in recent UX research efforts (e.g. [5]);
and UX. They think they are the same thing.” (P3)
iii) Fun is the EQ that was dually considered as measurable
as well as non-measurable. This is somewhat surprising
because game experiences of which fun is an integral reporting (e.g. diary, interview, scale); and psycho-
part have been one of the hot topics in UX research physiological (e.g. eye-tracking, heart rate). Obstacles for
where different attempts to measure fun have been implementing measurement were also mentioned, including
undertaken (see the review in [1]). This observation various forms of validity, individual differences, cultural
underpinned P11’s argument for the measurability of factors, confidence in interpreting non-verbal behaviour,
fun as it is a well-defined concept. In contrast, P1’s translating abstract concepts into concrete design property,
counterargument referred to the complexity and and consistency of observed behaviour
multidimensionality of fun; reporting on overall fun
Anecdotal Descriptions on the Interplay between
after interaction seemed more plausible than on Evaluation and Development
individual sub-constructs; In responding to the interview questions, some participants
iv) Several high-level concepts were mentioned: ‘hedonic described intriguing cases that can well illustrate the
quality’ for measurability and ‘long-term experience’ challenges of enhancing the interplay between UX
and ‘deep [sub]-conscious experience’; they do not fit evaluation and system development. Subsequently we
into any of the categories. highlight the challenges and related anecdotes, which are
grouped as theoretical (Q10), methodological (Q11) and
practical issues (Q12).
Theoretical issues
Problem of measuring UX in a holistic way and breaking
down into components seems not an ideal solution.
P3: When we go through the issues with uses, we observe the
whole expression, their comments on certain issues. If we
have a lot of things to study, it is more difficult to run this kind
of a holistic study; in a lab test where we only study some
specific items. In an evaluation session when we study several
issues, we can show users some of them and then the whole
one. Holistic approach is the way to go, but measures about
some specific details help as well.
P4: I'd say UX is holistic in nature, it is difficult to break it
down into very small pieces. From the traditional scientific
perspective, the way to measure something, to break it down
and separate different factors … The value of the
Furthermore, the main argument for measurability is that measurement gets lower if you break it down to small pieces...
the EQs of interest are well defined and documented in the My colleague studied 3D video. She was able to measure
literature. Two participants, however, could not name any objectively some aspects in lab by breaking things down, but
certainly measurable EQ because they considered that when she went to realistic context for certain kinds of
qualitative data were better for understanding feelings and arrangement, the results are really different…. Your
that experiential concepts were in general fairly vague. In experience may change dramatically.
contrast, the key arguments for non-measurability are the Memorized experiences prone to fading and fabrication
epistemological assumption about the nature of certain P5: the actual intensity of the moment fades very fast… So it
experiences and lack of a unified agreement on what UX is. is interesting to see how to recall and how we change the
The five participants could not name any certainly non- memory of the experience. When we ask people whether they
like something or not it depends on the moment you are
measurable EQ. They, while assuming that everything can
asking. iPhone, there is so much positive information of that
be measured, had the reservations for the validity, impact product out there that even if you did not like it, your
and completeness of UX measures. Specifically, P9 pointed environment is so positive about it that you are positive as
out the issue of conflating meaningfulness with relevance: well. It is the same as with reconstructing the memories. …
“I think anything can be measured in a meaningful way; it Most people as well as I myself are sure I have memories
depends who the audience is… the issues with measurement … where I cannot make a difference between the reconstructed
are well understood in the psychometric system whether you are and actual memory.
really measuring what you think you are measuring. So, and,
again you need to distinguish between meaningfulness and UX measures are highly sensitive to timing and nature of
relevance… there are things that are irrelevant … but I don’t tasks
think it’s possible for things in this world to have no meaning… P2: When to measure depends the duration and complexity of
people are natural interpreters. the task. For a small task, we can let people complete it and take
measures at the end. For the longer one may need to be
With regard to the question on how to measure EQ, the interrupted…. I am thinking a lot how much I am manipulating
participants identified a range of known HCI methods, everything when I am organizing a workshop with some tasks
which can be categorized into three major types: overt how everything would be different if the tasks would be
behaviour (e.g., time-on-task, number of trials to goal); self-
different…. Resource-demanding evaluation with a large number of
heterogeneous users
P4: Our perspective is very design-oriented. My experience in
measuring UX in design process is not so much. It is so easy
and fast to make the participants fill out AttrakDiff, it really
would not make sense not to do it. How we analyse the results
and get out of it, that's still to be seen. We don’t have so many
P8: Different measures in different phases of the usethey
participants that we could see what the different ways of using
complement each other if we need long-term evaluation.
those results are. Like a backup, we get a general understanding
Sometimes you can get details out of there supporting design.
of the situation to compare for making the second prototype,
They are more for prioritising the essential issues.… You don’t
what things to change. When we have the second prototype and
have exact measures for evaluating emotions at the moment.
we use the same measurement, we can see where the design is
Very momentary info can be useful, but you also need other
going. As measurement depending so heavily on individual
measures. Even though you can capture all the momentary
participants, it is difficult to make conclusion about the
emotional measures, you don’t know how the user interprets the
measurements… it is hard to say why there is a difference in the
emotion. The interpretation of the person is very important
results because of different social groups.
a negative experience can be interpreted as a positive experience
later on.
Need of sophisticated prototypes for eliciting authentic
Methodological Issues user experiences
P7: Difficult, especially housing business … we cannot build
Different preferences for qualitative and quantitative data only one prototype and then ask people experience it, get
by design- and engineering-oriented stakeholders feedback and then do it… we need good examples, media we
P7: … we are not fond of measures … we have smart design can use to produce our tools, social media, TV, etc to show what
work, something we have emphasized more on qualitative and kind of solution we might have.. the storytelling method like
inspirational aspect of UX. We have something to do with movie; I’d like to see sophisticated level like what would be
design perspective; kind of measurement only gives basic done with professional actors, directors, writers, like real life,
constraints and do not give directions. It depends where you feeling like real life with different natural mistakes.
apply the methods; how they should be interpreted and position
the methods. Measures are good background knowledge but we Practical Issues
have more unpredictable, qualitative data. Lack of knowledge in exploiting feedback on UX for
P8: Qualitative data could cover everything, but then how to future system development
convince the engineers, that's why we need numbers. Also for P5: Most people in industry, whether they have backgrounds in
research purpose, it could be interesting to find the relationships economics, engineers or marketing, for them handling
between factors. I have to measure somehow to find out which qualitative information is very difficult and they even don’t
is more influential, hedonic or pragmatic quality, on customer know how to use that or they would need that…. We've been
loyalty… quantitative data are more convincing, but developers criticising the UX evaluation, not about how we measure UX,
need qualitative data as well because they want to understand but how we use the information it in industry. … But there is so
the reason for frustration… the developers like videos because much information that people don't bother to read or follow
they can describe very lively the situation. They can also believe them. We need to make things simple and easy so that people
textual descriptions. … It is important to measure both don't have backgrounds they can understand. In fact, the
immediate experience and memorable experience. Practitioners majority of usability people, at least in Finland, have
are very thrilled by the idea that you can do it afterwards engineering or computer science background but have little
because it is so easy. So the companies are very interested in about psychology. There are a lot of things natural for
long-term UX or this kind of retrospective evaluation, they don't psychologists or sociologists during the study handling control
mind that, because they are convinced that memories are very vs. experiment. They don't necessarily come to think of; there
important because they are telling stories to other customers; are experts in company talking about human beings, but they
they are loyal to the companies based on the memories. Only the have certain views. It is challenging. This area of UX has the
reviewers are criticising the validity of retrospective methods. good side of interdisciplinary as well as the negative ones.
Practitioners are very interested in it and like the idea. P4: Quite often field experiments lead to straightforward results
P10: You have to interpret psycho-physiological data and map that can be exploited in their design work right away. One
these data to one of these experiential concepts and it is very project quite a while ago… We had purely lab experiments. We
hard to know whether you get it right. You can have a high heart were doing lab test applying Fitt's law with different input
rate because you really love it or you hate it. So may be it also devices, we were creating some constants that could be used for
depends on how many categories you have; the more categories evaluating early stages of design to see if input device Design A
you have, the more difficult to find a good mapping. is better than Design B. The partners were really excited about
the results. They were well done, theoretically and practically
P11: To see the impact of the goal of the system, how people validated and applicable… Industrial people were quite lost
perceive it. I think that's fine. For the purpose of design, when we were not there. They needed our guidance.
quantitative measures do not make sense. It is a wrong method Unfortunately we had no choice. We had good results, but no
for the purpose of design. real exploitation of the results since the customer did not know
what to do with the results.
Lack of standard UX metrics renders redesign decisions than it can answer. As the number of participants was
prone to personal biases relatively low with most of them originating from one
P5: People make decisions based on their personal beliefs. They country, namely, Finland, the views expressed might not be
just pick from the UX measures the ones that support their representative. Given this drawback, we have been
existing belief, and ignore the other results that don't support. … motivated to expand the investigation on UX measurement
They don't even realize it themselves that they are manipulating with a larger scale survey of which results are documented
the results. … People don't know how to use information on
human beings. … we had noticed that the same icon did not
elsewhere (under review). With a better understanding of
work for various kinds of notification… We got feedback the the issues about UX measures, especially how they can be
people were annoyed… there was a very strong personality in translated into new design requirements, insights into the
the design lead who said that he did not want the design changes interplay between UX evaluation and design can be gained.
because they look ugly… It is problematic that UX have no
ACKNOWLEDGEMENT
commonly agreed definition or no commonly agreed metrics. It
allows people to use this kind of argumentation that “I believe
Many thanks should be given to Dr. Virpi Roto, Aalto
that it is better UX”. You don't need to justify, it can be a University, Finland for her generous support in arranging
personal opinion even though there are tons of user feedback. the interviews when Effie Law had her short-term scientific
mission in Helsinki funded by COST IC0904 TwinTide.
Packaging UX measures for decision makers and
speaking their language REFERENCES
P4: … social TV case we did Attrakdiff questionnaire and 1. Bargas-Avila, J.A., & Hornbæk, K. (2011). Old wine in new
industry partner was very interested in that. They saw the bottles or novel challenges? A critical analysis of empirical
potential in that when we had enough data, more convincing, studies of user experience. In Proc. CHI’11
more easily convince their superior of the organization to 2. Bartholomew, D. J. (2006) (Ed). Measurement (Sage
finance their projects, show the need for working on some Benchmarks in Social Research Methods). Volume 1. Sage.
aspects further; objective foundations. 3. Boyatzis, R. E. (1998). Transforming qualitative information:
Thematic analysis and code development. Sage.
P5: It is not meaningless to measure moment-to-moment 4. Hand, D.J. (2004). Measurement theory and practice. Wiley-
experience, but the question is how you use this information… Blackwell.
But how to pack the thing and sell the thing to people making 5. Hassenzahl, M., & Monk, A. (2010). The influence of
product or legislation decisions. In this area we should talk perceived usability from beauty. Human-Computer
about how we use the information in this domain for the Interaction, 25(3), 235-260.
legislation and guiding the decision makers of different 6. Law, E.L-C. (2011). The measurability and predictability of
countries… Even when I think about from the industry user experience. In Proc. of the 3rd ACM SIGCHI Symposium
perspective. Strategy management what they are most interested on Engineering Interactive Computing System (EICS 2011),
in is that what are the elements that make users buy next devices Pisa, Italy, June 2011.
from the same company as well and what can reduce the number 7. Law, E. L-C, Roto, V., Hassenzahl, M., Vermeeren, A., &
of helpdesk contacts. The first one is related to the future Kort, J. (2009). Understanding, scoping and defining user
revenue of the company and the second one is related to the cost experience: a survey approach. In Proc. CHI ’09, 719–728.
saving. It is mostly transfer it to money. It is the language that ACM, 2009.
the management understands. 8. Löwgren, J. (2007). Fluency as an experiential quality in
augmented spaces. International Journal of Design, 1, 1-10.
CONCLUDING REMARKS 9. Mahlke, S., Lemke, I., & Thüring, M.(2007). The diversity of
UX, as an immature research area, is still haunted by the non-instrumental qualities in human-technology interaction.
challenges of defining the scope of UX in general and MMI-Interaktiv, Nr. 13, Aug 2007, ISSN 1439-7854.
operationalising experiential qualities in particular. Apart 10. Obrist, M., Law, E.L-C., Väänänen-Vainio-Mattila, K., Roto,
V., Vermeeren, A., & Kuutti, K. (2011). UX research- which
from addressing these basic issues, it is necessary for UX
theoretical roots do we build on – if any. In Extended Abstract
professionals to identify plausible means for compromising CHI’11.
the difficulties of evaluating UX in a holistic manner with 11. Schaik, van P., Hassenzahl, M., & Ling, J. (2012). User
the limitations of adopting the reductionist approaches. experience from an inference perspective. ACM Transaction
Deeper understandings about the relationship between on Human-Computer Interaction.
experience and memory and about the temporality of UX 12. Scherer, K. (2005). What are emotions? And how can they be
are also required. While the utility and necessity of measured? Social Science Information, 44(4), 695-729.
employing both quantitative and qualitative methods is 13. Vermeeren, A. P.O.S. Law, E. L-C., Roto, V., Obrist, M.,
commonly recognized, the concomitant issue of providing Hoonhout, J., Väänänen-Vainio-Mattila, K. (2010). User
experience evaluation methods: current state and development
appropriate education and training in UX needs to be
needs. In Proc NordiCHI 2010 (pp. 521-530).
explored. Specifically, UX researchers and practitioners 14. Thomson, W. (1891). Popular Lectures and Addresses, Vol. I.
should be equipped with knowledge and skills to know why (p.80). London: MacMillan.
certain UX measures are taken and how to use and interpret 15. Thüring, M., & Mahlke, S. (2007). Usability, aesthetics and
them in order to inform design and development decisions. emotions in human-technology interaction. International
Journal of Psychology, 42(4), 253-264.
Insights into the issues of UX measures have been gained
from the interviews. The study has raised more questions