Interplay between User Experience Evaluation and System Development: State of the Art

Abstract

User Experience (UX) is an emerging research area pertaining to as well as extending beyond the traditional usability. Issues in the realm of usability may be amplified in UX because of its larger scope. Four key non-orthogonal issues are: definition, modeling, method selection, and interplay between evaluation and development. Leveraging the legacy of a series of earlier workshops, I-UxSED 2012 aims to develop a deeper understanding of how evaluation feedback shapes software development, especially when experiential qualities such as fun, trust, aesthetic values are concerned. Is feedback on these fuzzy qualities less useful for problem prioritization or less persuasive for problem fixing? This and other challenging questions will be explored in I-UxSED 2012 that brings together researchers and practitioners from two communities - HCI and Software Engineering.

-

BACKGROUND AND MOTIVATION Leveraging the legacy of a series of successful workshops ([ 1 ] [2] [3]) that brought together people from HumanComputer Interaction (HCI) and Software Engineering (SE) communities to discuss the interplay between software evaluation and development, the proposed I-UxSED 2012 is further inspired by more recent insights into the issues pertaining to traditional usability (e.g. [4]) as well as the emerging User Experience (UX) (e.g. [5], [6]).

Recently, research efforts have been undertaken to collect, consolidate and categorize UX evaluation methods (e.g. [10]). It is envisaged that taxonomies of UX qualities, which can facilitate the selection of UX methods and measures, will come to fruition from these ongoing endeavours.

The first three issues have significant impacts on their fourth one – the theme of I-UxSED 2012 - is only explored to a limited extent.

WORKSHOP GOALS AND THEMES We understand the relationship between UX and usability as the latter is subsumed by the former. Usability evaluation methods (UEMs) and metrics are relatively more mature [11]. In contrast, UX evaluation methods (UXEMs) which draw largely on UEMs [12] are still taking shape. It is conceivable that feeding outcomes of UX evaluation back to the software development cycle to instigate the required changes can even be more challenging than doing so for usability evaluation (UE). It leads to several key issues: UX attributes are (much) more fuzzy and malleable, what kinds of diagnostic information and improvement suggestion can be drawn from evaluation data. For instance, a game can be perceived by the same person as a great fun on one day and a terrible boredom the following day, depending on the player’s prevailing mood. The waning of novelty effect (cf. learnability differs over time in case of usability) can account for the difference as well. How does the evaluation feedback enable designers/developers to fix this experiential problem (cf. usability problem) and how can they know that their fix works (i.e. downstream utility)? Emphasis is put on conducting UE in the early phases of a development lifecycle with the use of low fidelity prototypes, thereby enabling feedback to be incorporated before it becomes too late or costly to make changes [ 13 ]. However, is this principle applicable to UX evaluation? Is it feasible to capture authentic experiential responses with a low-fidelity prototype? If yes, how can we draw insights from these responses? The persuasiveness of empirical feedback determines its worth. Earlier research (e.g. [14]) indicates that the development team needs to be convinced about the urgency and necessity of fixing usability problems. Is UX evaluation feedback less persuasive than usability feedback? If yes, will the impact of UX evaluation be weaker than UE? The Software Engineering (SE) community has recognized the importance of usability. Efforts are focused on explaining the implications of usability for requirements gathering, software architecture design, and the selection of software components [15]. Can such recognition and implications be taken for granted for UX, as UX evaluation methodologies and measures could be very different (e.g. artistic performance)? How to translate observational or inspectional data into prioritised usability problems or redesign proposals is thinly documented in the literature [4]. Analysis approaches developed by researchers are applied to a limited extent by practitioners [4]. Such divorce between research and practice could be bitterer in UX analysis approaches, which are essentially lacking. While the gap between HCI and SE with regard to usability has somewhat been narrowed (e.g. [ 1 ]. [2]), it may be widened again due to the emergence of UX.

The main goal of I-UxSED 2012 is to bring together people from HCI and SE to identify challenges and plausible resolutions to optimize the impact of UX evaluation feedback on software development.

RELEVANCE TO THE FIELD The main contribution of I-UxSED 2012 to the field of HCI and SE is the understanding of state-of-the-art about the interplay between UX evaluation feedback and system development. Specifically, there are limited studies investigating how different UX evaluation feedback formats such as textual (e.g. diary), audio (e.g. interview), visual (e.g. pictorial scale) and physiological (e.g. eye-tracking) determine their usefulness as well as persuasiveness. Besides, visual and physiological data are more commonly used in UX than in usability, based on the observations that experiences are more difficult to verbalize and more subjective. The role of such evaluation data in system redesign entails further exploration. Besides, there are very few methodological and practical guidelines on integrating UX evaluation and system design in a software development process. The workshop will heighten the awareness of the need for more research studies on the above-mentioned issues.

CONTRIBUTIONS Eleven quality contributions have been accepted. They are categorized into four groups:  Domain-specific design and evaluation case study (Winckler et al. on e-citizen, Panayiotis et al on elearning, Nilsson & Følstad on emergency services)  Models on usability and UX evaluation (Oliveria et al on customer satisfaction, Sikorski on customer relationship, and Srđević et al on decision-making )  Agile and UX practice (Lárusdóttir et al on UX role in scrum, Lindell on design-driven organization, and Jokela on the role of evaluation in UX)  Attitudes towards and awareness of UX (Ardito on UX practice in companies; Law and Schaik on attitudes towards UX measurement) In-depth discussions in the workshop can shed light on these aspects with regard to the interplay between UX evaluation and software development. Future research challenges along this inquiry will be identified.

REFERENCES [ 1 ] Abrahão, S., Law, E. L-C., Stage, S., Hornbaek, K., Juristo, N. (2008). Proceedings of the First Workshop on the Interplay between Usability Evaluation and Software Development, Pisa, Italy, September 24, 2008, CEUR-WS.org 2008 [2] Abrahão, S., Hornbaek, K., Law, E. L-C., Stage, J. (2009). Interplay between Usability Evaluation and Software Development (I-USED 2009). In Proceedings of INTERACT (2) 2009: 969-970 [3] Law, E. L-C., Abrahão, S., Stage, J. (2010).

Proceedings of the First International Workshop on the Interplay between User Experience and Software Development (I-UxSED 2010) (CEUR-WS.org 2010), NordiCHI 2010, Iceland. [4] Følstad, A., Law, L-C., & Hornbaek, K. (2012): Analysis in practical usability evaluation: a survey study. In Proc. CHI 2012 (pp. 2127-2136). [5] Bargas-Avila, J. A., & Hornbaek, K. (2011). Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In Proc. CHI 2011 (pp. 2689-2698). [6] Hassenzahl, M. (2011): User Experience and Experience Design. In: M. Soegaard, & R. F. Dam, (eds.), Encyclopedia of Human-Computer Interaction". http://www.interactiondesign.org/encyclopedia/user_experience_and_experie nce_design.html [7] Law, E. L-C., Roto, V., Hassenzahl, M., Vermeeren, A., & Kort, J. (2009). Understanding, scoping and defining

References

user experience: a survey approach . In Proc. CHI 2009

(pp. 719 - 728 ). [8] Law , E. L-C. ( 2011 ). The measurability and

predictability of user experience . In Proc. EICS 2011

(pp. 1 - 10 ). [9] Law , E. L-C. , & van Schaik. P. ( 2010 ). Modelling user

Interacting with Computers , 22 ( 5 ): 313 - 322 . [10] Vermeeren , A. P.O.S. Law , E. L-C. , Roto , V. , Obrist ,

M. , Hoonhout , J. , Väänänen-Vainio-Mattila , K. ( 2010 ).

development needs . In Proc NordiCHI 2010 (pp. 521 -

530). [11] Law , E. L-C. , Hvannberg , E. , & Cockton , G . (eds.)

( 2008 ). Maturing usability: Quality in software,

interaction and value . Springer. [12] Tullis , T. , & Albert , W. ( 2008 ). Measuring the User

Kaufman. [13] Hertzum , M. ( 2006 ). Problem Prioritization in

Human Computer Interaction 21 ( 2 ): 125 - 146 . [14] Nørgaard , M. , & Hornbaek , K ( 2009 ). Exploring the

Journal Human Computer Interaction 25 ( 1 ), 49 - 74 . [15] Juristo , N. , Moreno , A. M. , Sanchez-Segura , M.

( 2007 ). Guidelines for Eliciting Usability

Engineering , 33 ( 11 ), 744 - 758 .