Interplay between User Experience Evaluation and Software Development: Challenge and Outlook

Abstract

Theme and Goals Following up the tradition established in the I-USED (International Workshop on the Interplay between Usability Evaluation and Software Development) series of workshops1, this workshop is aimed at bringing together Human-Computer Interaction (HCI) and Software Engineering (SE) professionals and researchers interested in discussing recent trends and perspectives of the role of usability in software development. With respect to I-USED, I-UxSED (http://users.dsic.upv.es/workshops/i-uxsed10/) extends its scope to target the broader concept of User Experience (UX) in software development. This change of focus was mainly motivated due to recent advances in mobile, ubiquitous, social, and tangible computing technologies that has moved HCI into practically all areas of human activity. This has led to a shift away from usability engineering to a much richer scope of user experience where user's feelings, motivations, and values are given as much, if not more, attention than ease of use, ease of learning and basic subjective satisfaction (i.e., the three traditional usability metrics). To accommodate the shift, evaluation approaches need to respond in a way that is sensitive to increasingly diverse use contexts, user goals and roles, and new interaction styles [8].

-

Modelling users’ experience - as a basis for producing design guidance - is especially important. First, measurement models are required to provide a sound basis for UX measures with desirable properties (e.g. reliability, validity, sensitivity). Second, structural models are needed for the purpose of understanding, predicting and reasoning about processes of UX with consequences for software design. Despite some visible progress (e.g. [ 12 ]), a number of issues pertaining to UX modelling remain to be resolved [ 19 ]. Furthermore, it is very important to develop practical guidelines for selecting evaluation methods and an associated set of measures to meet requirements specific to the context of interest. Currently, research efforts have been invested in collecting, consolidating and categorizing UX evaluation methods (e.g. [ 23 ]). It is envisaged that taxonomies of UX qualities, which can facilitate the selection of UX methods and measures, will come to fruition from these ongoing endeavours.

Presumably, the aforementioned work pertinent to the three challenges (i.e. defining UX, modelling UX, and selecting UX methods) can contribute to the resolution of the fourth one (i.e. interplay between UX evaluation and system development), which, as far as we know, is only explored to a limited extent.

We understand the relationship between UX and usability as the latter is subsumed by the former. While usability evaluation methods (UEMs) and metrics are relatively more mature, UX evaluation methods (UXEMs), which draw largely on UEMs [ 26 ], are still taking shape. It is conceivable that feeding outcomes of UX evaluation back to the software development cycle and instigating the required changes can even be more challenging than doing so for Usability Evaluation (UE). Several concerns are as follows: ƒ UX attributes are (much) more fuzzy and malleable, what kinds of diagnostic information and improvement suggestion can be drawn from evaluation feedback. For instance, user-based evaluation of fun - one of the critical qualities that have triggered the shift of attention from usability to UX ([ 5 ], [ 9 ]) – can involve subjective data with interviews or scales and objective data with psychophysiological measures [ 20 ]. A game can be perceived by the same person as a great fun on one day and a terrible boredom the following day, seemingly depending on the player’s prevailing mood. The waning of novelty effect (cf. learnability differs over time in case of usability) can account for the difference as well. How does the evaluation feedback enable designers to fix this experiential problem (cf. usability problem) and how can they know that their fix works? ƒ Emphasis is put on conducting UE in the early phases of a development lifecycle with the use of low fidelity prototypes, thereby enabling feedback to be incorporated before it becomes too late or costly to make changes [ 13 ]. However, is this principle applicable to UX evaluation? Is it feasible to capture authentic experiential responses with a low-fidelity prototype? If yes, how can we draw insights from these responses? ƒ Irrespective of whether formal or informal evaluation approaches are applied to traditional HCI phenomena like usability or emerging ones like UX, it is the persuasiveness of empirical evidence that is ultimately the test of its worth. Indeed, earlier research (e.g. [ 22 ]) indicates that the development team needs to be convinced about the urgency and necessity of fixing usability problems. Is UX evaluation feedback less persuasive than usability feedback? If yes, will the impact of UX evaluation be weaker than UE? ƒ Software Engineering (SE) community has recognized that usability does not only affect the design of user interfaces but the software system development as a whole. In particular, efforts are focused on explaining the implications of usability for requirements gathering [ 16 ], software architecture design ([ 3 ], [ 4 ]), and the selection of software components [ 21 ]. Can such recognition and implications be taken for granted for UX, given that the evaluation methodologies and measures of UX could be very different (e.g. artistic performance)? While the gap between HCI and SE with regard to usability has somewhat been narrowed, thanks to years of endeavour of the researchers in both communities (e.g. [ 2 ]), it may be widened again due to the emergence of UX.

The goal of this workshop is to bring together researchers and practitioners from the HCI and SE fields to identify challenges and plausible resolutions to optimize the impact of UX evaluation feedback on software development. Presentations of new ideas on how to improve the interplay between HCI & SE to the design of usable, pleasurable and desirable software systems should be based on empirical studies. Within this focus, topics of discussion include, but are not limited to: • • • • • • • •

Which artifacts of software development are useful as the basis for UX evaluations?

How do the specific artifacts obtained during

software development influence the techniques that are relevant for the UX evaluation?

In which forms are the results of UX evaluations supplied back into software development (including the UI design)?

What are the characteristics of UX evaluation

results that are needed in software development?

Do existing UX evaluation methods deliver the

results that are needed in user interface design?

How can UX evaluation feedback be integrated

more directly in user interface design?

How can UX evaluation methods be applied in

emerging techniques for user interface design?

How can UX evaluation methods be integrated to novel approaches for software development (e.g., model-driven development, agile development)?

Relevance to the Field The main contribution of the workshop is the understanding of state-of-the-art about the interplay between UX evaluation feedback and system development and the identification of areas for improvement and further research. The HCI field includes a rich variety of techniques for UX evaluation and user interface design. However, there are very few methodological guidelines for the interplay between these key activities; and more important, there are few guidelines on how to properly integrate these two activities in a software development process.

Accepted Submissions Based on the results of the systematic peer review process, nine submissions have been accepted for the workshop, representing a spectrum of views on the theme on interplay between user experience evaluation and software development. Here below we highlight the main arguments of each submission and our reflections on them.

Følstad succinctly puts forward a stimulating proposition: “Complex models, underpinning complex measures, are likely to be valuable to the advancement of UX theory.

However, in order to advance the interplay between UX and SD, simplified models and measures may be required.”

A concomitant query can be: While simplicity tends to yield receptivity, could simplification have a similar property? Simple is different from simplified: If something is inherently simple, it is likely to be accepted. However, if something is inherently complex, simplifying it may lead to misrepresentation and thus confusion.

Jääskeläinen and Heikkinen have conducted a national survey to identify the differences between professionals and end-users in their understanding of UX definitions and attributes. A host of intertwined factors can contribute to the differences observed. Methodologically it is challenging to isolate the respective impacts of these factors. Tim and Huang present inspiring proposals for formalizing trust requirements with notations such as extended UML with which software developers are likely familiar. The use of such a boundary object [ 24 ] to facilitate communications between designers (or evaluators) and developers seems promising. The authors’ proposal of using physiological measures to triangulate cognitive metrics sounds exciting as well.

Jokela provocatively argues that there are two gaps instead of one: between UX and interaction design and between interaction design and software development. The argument can be boiled down to the basic issue of the very nature of UX. Indeed, some researchers and practitioners tend to synonymise UX with interaction design. Besides, Jokela’s JFunnel user experience life-cycle model seems built upon usability. Debates on how to demarcate UX from usability and the other related concepts are ongoing [ 24 ] Karahasanović and Obrist extend the issue of downstream utility of usability evaluation methods (UEMs) to user experience evaluation methods (UXEMs). The applications of interest are social media. The authors scope seven UXEMs against eight UX factors. Six recommendations are derived from the scoping exercises. Empirical validations of these recommendations are called for.

Wolkerstofer and his colleagues aim to bridge the cultural gap between the two communities: HCI and eXtreme Programming (XP). The observation that XP developers are resistant to persona coincides with similar findings in some other non-XP projects. It is intriguing to explore which factors contribute to such resistance.

Müller, Law and Strohmeier address the issue of persuasiveness – a significant notion of downstream utility, which is related to the work of Karahasanović and Obrist in this volume. Müller et al. map the constructs of the two traditional models in the domain of Information Systems to UX attributes and then compare whether usability-oriented ones are more persuasive than UX-based one in enhancing developers’ problem-fixing tendency. The study serves as a precursor to a more ambitious investigation of the actual fixing behaviours of developers.

Alsos studies the notion of indirect user experience in the context of hospital with physician being primary users and patients indirect ones. User experience of the former can have influence that of the latter, or vice versa. The work may shed light onto the issue of co-experience or vicarious experience, which entail further conceptual and practical analysis.

Vold and Wasson investigate the ever prevailing phenomenon of participatory culture of learning, thanks to the advent of social software applications. The authors put emphasis on the role of fun in terms of playful feedback in enhancing learners’ user experience in an online community. Their work, like Alsos’, can contribute to the deeper understanding of social experience.

In summary, the nine workshop papers address some basic as well as applied research questions in the domain of User Experience, which is still being defined and scoped. With the notion of UX being somewhat fluid, it is deemed especially challenging to analyse and engineer the effect of UX evaluation feedback on software development. We would like to say a few words about the picture on the cover of the workshop proceedings. It has been generated by feeding the main bodies of the nine papers (i.e. without abstract, the other front matters or references) into a software application TagCrowd (http://tagcrowd.com/). It visualizes individual words extracted from the submitted text with different shapes and shades of blue according to their relative frequencies. The top 50 words thus identified have further been fed into another similar application Wordle (http://www.wordle.net/), which beautifies the word cloud. Not surprisingly, the words UX, Design, Experience,

Users, Evaluation, and Feedback are salient ones.

References

[1] Abrahão , S. , Hornbaek , K. , Law , E. L-C. , Stage , J. ( 2009 ). Interplay between Usability Evaluation and Software Development (I-USED 2009) . In Proceedings of INTERACT (2) 2009 : 969 - 970

[2] Abrahão , S. , Law , E. L-C. , Stage , S. , Hornbaek , K. , Juristo , N. ( 2008 ). Proceedings of the First Workshop on the Interplay between Usability Evaluation and Software Development , Pisa, Italy, September 24 , 2008 , CEUR-WS. org 2008

[3] Bass , L. , & John , B. ( 2003 ). Linking Usability to Software Architecture Patterns through General Scenarios . Journal of Systems and Software , 66 ( 3 ), 187 - 197

[4] Bass , L. , John , B. , & J. Kates, ( 2001 ), “Achieving Usability through Software Architecture,” Technical Report CMU/SEI-2001 - TR- 005 , Software Eng . Inst., Carnegie Mellon Univ.

[5] Blythe , M. A. , Overbeeke , K. , Monk , A.F.. , & Wright , P. C.(Eds.) ( 2004 ). Funology: from usability to enjoyment . Kluwer Academic Press.

[6] Blythe , M. , Wright , P. , McCarthy , M. , Bertelsen , O.W. ( 2006 ). Theory and method for experience centered design . In Extended Abstracts Proceedings of CHI'06 , Montréal , Québec, Canada, ACM Press.

[7] Bødker , S. , Kensing , F. , & Simonsen , J. ( 2004 ). Participatory IT Design. Designing for Business and Workplace RealitiesI

[8] Bødker , S. , ( 2006 ). When second wave HCI meets third wave challenges . In Proc. NordiCHI 2006 , (pp. 1 - 8 ), 14 - 18 , October 2006 , Oslo, Norway.

[9] Carroll , J.M. , & Thomas , J.M. ( 1998 ). Fun. ACM SIGCHI Bulletin , 19 ( 3 ), 21 - 24 .

[10] Cockton , G. , 2008 . Designing Worth: Connecting Preferred Means with Probable Ends . Interaction, 15 ( 4 ), July+August issue , 54 - 57 .

[11] Dourish , P. , 2006 . Implication for design . In Proc. of the Conference on Human Factors in Computing Systems (CHI 2006 ), Montréal, Québec, Canada, April 22-27 , 2006 , pp. 541 - 550 .

[12] Hassenzahl , M. , 2004 . The interplay of beauty, goodness, and usability in interactive products . Human-Computer Interaction , 19 , 319 - 349 .

[13] Hertzum , M. ( 2006 ). Problem Prioritization in Usability Evaluation: From Severity Assessments Toward Impact on Design . International Journal Human Computer Interaction 21 ( 2 ): 125 - 146 .

[14] ISO 9241-110 , 2006 . Ergonomics of human-system interaction - Part 110: Dialogue principles , ISO Copyright Office, Geneva, Switzerland.

[15] Juristo , N. , Moreno , A. M. , Sanchez-Segura , M. ( 2007 ), “Guidelines for Eliciting Usability Functionalities” , IEEE Transactions on Software Engineering , Vol. 33 , No. 11 , November 2007 , pp. 744 - 758 .

[16] Juristo , N. , Windl , H. , & Constantine , L. ( 2001 ), "Introducing usability" , IEEE Software , 20 - 21

[17] Law , E. L-C. , Hvannberg , E. , & Cockton , G . (eds.) ( 2008 ). Maturing usability: Quality in software, interaction and value , London, Springer Verlag.

[18] Law , E. L-C. , Roto , V. , Hassenzahl , M. , Vermeeren , A. , & Kort , J. ( 2009 ). Understanding, scoping and defining user experience . In Proc. CHI 2009 , 4 -9 April 2009 , Boston, USA

[19] Law , E.L-C. , & Schaik van, P. ( 2010 ). Modelling UX - an agenda for research and practice . Interacting with Computers , 22 ( 5 ), 312 - 323 .

[20] Nacke , L. , Lindley , C.A. , Stellmach , S. ( 2008 ). Log Who's Playing: Psychophysiological Game Analysis Made Easy through Event Logging . In Proceedings of Fun and Games 2008 (LNCS 5294) , (pp. 150 - 157 ), Eindhoven, the Netherlands.

[21] Perry , D. , & Wolf , A. ( 1992 ) “Foundations for the Study of Software Architecture,” ACM Software Eng. Notes , vol. 17 , no. 4 , pp. 40 - 52 , Oct. 1992 .

[22] Nørgaard , M. , & Hornbaek , K ( 2009 ). Exploring the Value of Usability Feedback Formats . International Journal Human Computer Interaction 25 ( 1 ): 49 - 74 .

[23] Roto V. , Obrist M. , Väänänen-Vainio-Mattila K. , 2009 . User experience evaluation methods in academic and industrial contexts . In Proceedings of Workshop UXEM'09 , CHI' 09 , Boston, USA.

[24] Roto , V. , Law , E. L-C. , Vermeeren , A. , & Hoonholt , J. ( 2010 ). Demarcating User Experience . Dagstuhl Seminar . Accessible at: http://www.dagstuhl.de/en/program/calendar/semhp/?s emnr= 10373

[25] Star S. L. , & Griesemer J.R. ( 1989 ). "Institutional Ecology, 'Translations' and Boundary Objects: Amateurs and Professionals in Berkeley's Museum of Vertebrate Zoology, 1907 - 39 ". Social Studies of Science 19 ( 4 ): 387 - 420 .

[26] Tullis , T. & Albert , W. ( 2008 ). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics . Morgan Kaufmann.