=Paper=
{{Paper
|id=None
|storemode=property
|title=Interplay between User Experience Evaluation and Software Development: State of the Art
|pdfUrl=https://ceur-ws.org/Vol-922/paper1.pdf
|volume=Vol-922
|dblpUrl=https://dblp.org/rec/conf/iused/LawAVH12
}}
==Interplay between User Experience Evaluation and Software Development: State of the Art==
https://ceur-ws.org/Vol-922/paper1.pdf
Interplay between User Experience Evaluation and System
Development: State of the Art
COST IC0904 TwinTide Open Workshop
Effie Lai-Chong Law Silvia Abrahão
Department of Computer Science Department of Computer Science
University of Leicester Universitat Politècnica de València
UK Spain
elaw@mcs.le.ac.uk sabrahao@dsic.upv.es
Arnold P.O.S. Vermeeren Ebba Thora Hvannberg
Faculty of Industrial Design Engineering School of Engineering and Natural Sciences
Delft University of Technology University of Iceland
The Netherlands Iceland
A.P.O.S.Vermeeren@tudelft.nl ebba@hi.is
ABSTRACT The shift of emphasis in the field of HCI from usability
User Experience (UX) is an emerging research area engineering to a much richer scope of user experience
pertaining to as well as extending beyond the traditional where users’ emotions, affects, motivations, and values are
usability. Issues in the realm of usability may be amplified given as much, if not more, attention than ease of use, ease
in UX because of its larger scope. Four key non-orthogonal of learning and basic subjective satisfaction [7]. Among
issues are: definition, modeling, method selection, and others, four challenges engendered by the new focus of UX
interplay between evaluation and development. Leveraging are particularly relevant to software development: (i)
the legacy of a series of earlier workshops, I-UxSED 2012 definition of UX; (ii) modelling of UX; (iii) selection of UX
aims to develop a deeper understanding of how evaluation evaluation methods; (iv) interplay between UX evaluation
feedback shapes software development, especially when feedback and software development.
experiential qualities such as fun, trust, aesthetic values are
concerned. Is feedback on these fuzzy qualities less useful The concept of UX is commonly understood as subjective,
for problem prioritization or less persuasive for problem context-dependent and dynamic [7]. A “formal” definition
fixing? This and other challenging questions will be of UX issued by ISO 9241-210: 2010 - A person’s
explored in I-UxSED 2012 that brings together researchers perceptions and responses that result from the use and/or
and practitioners from two communities - HCI and anticipated use of a product, system or service – is
Software Engineering. ambiguous and needs to be refined.
Author Keywords In contrast to usability, UX metrics are yet to be defined.
User experience; Usability; Software development; The task is related to ongoing debates on the measurability
Interaction design; Downstream utility; Interplay of experiential qualities [8]. Both usability and UX
measures should enable professionals to benchmark
ACM Classification Keywords competitive design artefacts and to select right design
H.5.m. Information interfaces and presentation (e.g., HCI): options. The intriguing question is whether the respective
Miscellaneous. measures have different persuasive power and impact on
General Terms (re)design and development.
Human Factors; Design; Evaluation; Measurement Modelling users’ experiences is especially important for
BACKGROUND AND MOTIVATION understanding, predicting and reasoning about processes of
Leveraging the legacy of a series of successful workshops UX with consequences for software design. However, a
([1] [2] [3]) that brought together people from Human- number of issues pertaining to UX modelling remain to be
Computer Interaction (HCI) and Software Engineering (SE) resolved [9].
communities to discuss the interplay between software
Recently, research efforts have been undertaken to collect,
evaluation and development, the proposed I-UxSED 2012 is
consolidate and categorize UX evaluation methods (e.g.
further inspired by more recent insights into the issues
[10]). It is envisaged that taxonomies of UX qualities,
pertaining to traditional usability (e.g. [4]) as well as the
which can facilitate the selection of UX methods and
emerging User Experience (UX) (e.g. [5], [6]).
measures, will come to fruition from these ongoing How to translate observational or inspectional data into
endeavours. prioritised usability problems or redesign proposals is
thinly documented in the literature [4]. Analysis
The first three issues have significant impacts on their
approaches developed by researchers are applied to a
fourth one – the theme of I-UxSED 2012 - is only explored
limited extent by practitioners [4]. Such divorce
to a limited extent.
between research and practice could be bitterer in UX
WORKSHOP GOALS AND THEMES analysis approaches, which are essentially lacking.
We understand the relationship between UX and usability
as the latter is subsumed by the former. Usability While the gap between HCI and SE with regard to usability
evaluation methods (UEMs) and metrics are relatively more has somewhat been narrowed (e.g. [1]. [2]), it may be
mature [11]. In contrast, UX evaluation methods (UXEMs) widened again due to the emergence of UX.
which draw largely on UEMs [12] are still taking shape. It The main goal of I-UxSED 2012 is to bring together people
is conceivable that feeding outcomes of UX evaluation back from HCI and SE to identify challenges and plausible
to the software development cycle to instigate the required resolutions to optimize the impact of UX evaluation
changes can even be more challenging than doing so for feedback on software development.
usability evaluation (UE). It leads to several key issues:
RELEVANCE TO THE FIELD
UX attributes are (much) more fuzzy and malleable, The main contribution of I-UxSED 2012 to the field of HCI
what kinds of diagnostic information and improvement and SE is the understanding of state-of-the-art about the
suggestion can be drawn from evaluation data. For interplay between UX evaluation feedback and system
instance, a game can be perceived by the same person development. Specifically, there are limited studies
as a great fun on one day and a terrible boredom the investigating how different UX evaluation feedback formats
following day, depending on the player’s prevailing such as textual (e.g. diary), audio (e.g. interview), visual
mood. The waning of novelty effect (cf. learnability (e.g. pictorial scale) and physiological (e.g. eye-tracking)
differs over time in case of usability) can account for determine their usefulness as well as persuasiveness.
the difference as well. How does the evaluation Besides, visual and physiological data are more commonly
feedback enable designers/developers to fix this used in UX than in usability, based on the observations that
experiential problem (cf. usability problem) and how experiences are more difficult to verbalize and more
can they know that their fix works (i.e. downstream subjective. The role of such evaluation data in system
utility)? redesign entails further exploration. Besides, there are very
few methodological and practical guidelines on integrating
Emphasis is put on conducting UE in the early phases UX evaluation and system design in a software
of a development lifecycle with the use of low fidelity development process. The workshop will heighten the
prototypes, thereby enabling feedback to be awareness of the need for more research studies on the
incorporated before it becomes too late or costly to above-mentioned issues.
make changes [13]. However, is this principle
applicable to UX evaluation? Is it feasible to capture CONTRIBUTIONS
authentic experiential responses with a low-fidelity Eleven quality contributions have been accepted. They are
prototype? If yes, how can we draw insights from categorized into four groups:
these responses? Domain-specific design and evaluation case study
The persuasiveness of empirical feedback determines (Winckler et al. on e-citizen, Panayiotis et al on e-
its worth. Earlier research (e.g. [14]) indicates that the learning, Nilsson & Følstad on emergency services)
development team needs to be convinced about the Models on usability and UX evaluation (Oliveria et al
urgency and necessity of fixing usability problems. Is on customer satisfaction, Sikorski on customer
UX evaluation feedback less persuasive than usability relationship, and Srđević et al on decision-making )
feedback? If yes, will the impact of UX evaluation be Agile and UX practice (Lárusdóttir et al on UX role in
weaker than UE? scrum, Lindell on design-driven organization, and
Jokela on the role of evaluation in UX)
The Software Engineering (SE) community has
recognized the importance of usability. Efforts are Attitudes towards and awareness of UX (Ardito on UX
focused on explaining the implications of usability for practice in companies; Law and Schaik on attitudes
requirements gathering, software architecture design, towards UX measurement)
and the selection of software components [15]. Can In-depth discussions in the workshop can shed light on
such recognition and implications be taken for granted these aspects with regard to the interplay between UX
for UX, as UX evaluation methodologies and measures evaluation and software development. Future research
could be very different (e.g. artistic performance)? challenges along this inquiry will be identified.
REFERENCES user experience: a survey approach. In Proc. CHI 2009
[1] Abrahão, S., Law, E. L-C., Stage, S., Hornbæk, K., (pp. 719-728).
Juristo, N. (2008). Proceedings of the First Workshop [8] Law, E. L-C. (2011). The measurability and
on the Interplay between Usability Evaluation and predictability of user experience. In Proc. EICS 2011
Software Development, Pisa, Italy, September 24, 2008, (pp. 1-10).
CEUR-WS.org 2008 [9] Law, E. L-C., & van Schaik. P. (2010). Modelling user
[2] Abrahão, S., Hornbæk, K., Law, E. L-C., Stage, J. experience – an agenda for research and practice.
(2009). Interplay between Usability Evaluation and Interacting with Computers, 22(5):313–322.
Software Development (I-USED 2009). In Proceedings [10] Vermeeren, A. P.O.S. Law, E. L-C., Roto, V., Obrist,
of INTERACT (2) 2009: 969-970 M., Hoonhout, J., Väänänen-Vainio-Mattila, K. (2010).
[3] Law, E. L-C., Abrahão, S., Stage, J. (2010). User experience evaluation methods: current state and
Proceedings of the First International Workshop on the development needs. In Proc NordiCHI 2010 (pp. 521-
Interplay between User Experience and Software 530).
Development (I-UxSED 2010) (CEUR-WS.org 2010), [11] Law, E. L-C., Hvannberg, E., & Cockton, G. (eds.)
NordiCHI 2010, Iceland. (2008). Maturing usability: Quality in software,
[4] Følstad, A., Law, L-C., & Hornbæk, K. (2012): interaction and value. Springer.
Analysis in practical usability evaluation: a survey [12] Tullis, T., & Albert, W. (2008). Measuring the User
study. In Proc. CHI 2012 (pp. 2127-2136). Experience: Collecting, Analyzing, and Presenting
[5] Bargas-Avila, J. A., & Hornbæk, K. (2011). Old wine Usability Metrics (Interactive Technologies). Morgan
in new bottles or novel challenges: a critical analysis of Kaufman.
empirical studies of user experience. In Proc. CHI 2011 [13] Hertzum, M. (2006). Problem Prioritization in
(pp. 2689-2698). Usability Evaluation: From Severity Assessments
[6] Hassenzahl, M. (2011): User Experience and Toward Impact on Design. International Journal
Experience Design. In: M. Soegaard, & R. F. Dam, Human Computer Interaction 21(2): 125-146.
(eds.), Encyclopedia of Human-Computer Interaction". [14] Nørgaard, M., & Hornbæk, K (2009). Exploring the
http://www.interaction- Value of Usability Feedback Formats. International
design.org/encyclopedia/user_experience_and_experie Journal Human Computer Interaction 25(1), 49-74.
nce_design.html [15] Juristo, N., Moreno, A. M., Sanchez-Segura, M.
[7] Law, E. L-C., Roto, V., Hassenzahl, M., Vermeeren, A., (2007). Guidelines for Eliciting Usability
& Kort, J. (2009). Understanding, scoping and defining Functionalities. IEEE Transactions on Software
Engineering, 33 (11), 744-758.