The Big Picture of UX is Missing in Scrum Projects Marta Kristín Lárusdóttir Åsa Cajander Jan Gulliksen Reykjavik University Uppsala University Royal Institute of Technology Menntavegur 1, 101 Reykjavik Box 337, 751 05 Uppsala Lindstedsv. 3, 10044 Stockholm marta@ru.is asa.cajander@it.uu.se jangul@kth.se ABSTRACT experiences are taken into considerations automatically. The Scrum development process has gained increasing Hence the need to study how Scrum development projects popularity during the last decade. At the same time user address and manage user experience and the development experience (UX) has emerged as an important quality of effective and agile ways of addressing usability and UX feature. However, the integration of UX related activities is much sought after. into Scrum projects has not been without problems, and this area needs to be further examined. This paper describes the User Experience Measures and Evaluation results from two in depth interviews with knowledgeable Researchers agree that UX is a complex concept, including UX specialists working in Scrum projects in the product aspects like fun, pleasure, beauty and personal growth. UX development industry. It describes their ways of working focuses on the more emotional aspects of user interactions, generally with UX, their experiences from UX evaluations shifting the focus on how the users feel while and after and the challenges encountered from their UX work. The using the software, the sensation, and the meaning as well main concern when working with UX in Scrum projects is as the value of such interactions in everyday life. that the big picture of UX is often lacking. Finally, the Evaluating the UX has been a challenge for IT paper discusses the differences and similarities between the professionals. experiences from the UX specialists. UX is described by Hassenzahl as having pragmatic and Author Keywords hedonic attributes [6]. The pragmatic attributes are task User experience, user experience evaluation, usability, agile related. The main two pragmatic attributes are ease-of-use, software development, Scrum. described by effectiveness and efficiency and usefulness, described by words like clear, supporting and controllable. ACM Classification Keywords The difference of usability and UX measures has been H.5.2. User Interfaces-User-centered design. General discussed extensively, especially the difference of user Terms: UCSD, Design, Human Factors. satisfaction and UX [11]. One of the main discussion topics is that user satisfaction is more quantitative and UX is more INTRODUCTION qualitative. Moreover, it has been pointed out that usability The international standard ISO 9241-210 defines user measures may hint to a particular problem and sometimes experience as: "a person's perceptions and responses that to a solution of it, whereas UX measures are more general. result from the use or anticipated use of a product, system According to Law, this makes the usability measures more or service" [8]. The standard extends the concept of useful and persuasive for the IT professionals [11]. usability from the ISO 9241-11 standard in several ways [7]. User experience (UX) deals with much more than the Many methods have been suggested to evaluate the effectiveness and efficiency that is the main focus of different aspects of UX. The usage of 96 UX evaluation usability measurements. UX addresses satisfaction in its methods in various software development activities was widest possible application, from the hedonic feelings about studied in a recent study [20]. The methods were analyzed a product before it has even been unpacked to the feelings according to at what stage in the development process the raised that goes far beyond the very task-oriented nature of method could be used. Most of the methods could be used the usability focus. in the implementation and testing stages, and around one- third could be used either in requirements analysis or design In software development the need to focus on UX keeps stages. increasing as products and services become more competitive and need to function in a much broader context UX in Scrum Projects than previously. Recently development processes of a more The agile process Scrum has gained popularity in the agile nature have emerged that put emphasis on team work software industry in the Nordic countries in recent years. and production rather than on structure and documentation. One third of IT professionals in Iceland used this process in One of the most popular agile software development 2009 [9]. In Scrum, self-organizing and well compounded processes is Scrum [14]. Many people associate Scrum with teams are emphasized, typically with six to eight UX, but there is nothing in the process saying that user interdisciplinary team members [14]. In Scrum, the projects are split up in two to four week long iterations called The male UX specialist is a 46 years old man who has 13 sprints. At the end of each sprint, a potential shippable years of experience from working in different consultant product is delivered to the customer, meaning that it should companies. In his present employment he works for one of be functioning for the users. the largest IT companies in the Nordic countries with about 10 000 employees and 14 000 customers. His job title is The Scrum development process has been criticized for not usability designer, and he holds a PhD in Human Computer involving real users in the software development and for Interaction with the focus of adding a usability and user not adequately addressing their usability needs [15]. One of experience perspective in software development. He has the main conclusions in an extensive literature survey on worked with the integration of Scrum and UX in several the integration of the end user needs into agile processes is different projects in industry. These projects range from that these have not yet been sufficiently included in the "public interfaces to internal systems" but his main focus agile development processes [17]. Because of the short during the last years has been on public applications. sprints and the emphasis on completing a particular part of the software during each sprint, the IT professionals do not The female UX specialist is a 35 years old woman with a have much time in their development for involving users Master-degree in media technology science with the and for conducting UX evaluation [10]. specialization towards human computer interaction and sound. She has worked in industry for four years. Her Some researchers have suggested that some human-centred present employer is a large Swedish product development activities are conducted before the actual implementation in company founded in 1994. The company has offices in the project starts in order to address usability from a more eight countries and clients from all over the world. During holistic perspective. This is also the method used in the her four years as a UX specialist she has made much organization described by Sy [18] where a strategic phase progress in her company, and she has managed to establish before the project begins, contains specified human-centred UX as a core activity. Her formal role is a user experience activities to understand the context of use for example. manager, and she is in the middle of a process to hire ten Additionally, other researchers have recommended that members for a UX team where she will be the manager. activities related to UI design should be performed before The products she is working on are adaptable custom the actual implementation starts [4, 21]. products related to social media and the web. METHOD RESULTS This workshop paper presents the experiences that two In this section the experiences made by the UX specialists much knowledgeable UX specialists had from integrating are presented. Each person’s experience is categorized UX evaluation into Scrum projects. These two UX according to the three predefined categories: Ways of specialists were interviewed in a large interview study working, experience from UX evaluations in Scrum and focusing on the integration of a wider concept, namely User challenges encountered. The experiences that each person Centred Design in Scrum practice made in 2010. For the has had are presented separately with the help of quotations purpose of this workshop paper these interviews from the interviews. haveconseque been re-examined for data regarding UX evaluation and Scrum and these two interviews were found. The Male UX Specialist The two interviews were semi-structured and an interview Way of Working – Importance of UX vision template was used. The interviews were carried out on site The male UX specialist describes that the strategic vision of and lasted for about an hour. Two researchers conducted the the product is very important for him and that he uses that interviews. One researcher was taking notes and the other as a starting point in his work with prototypes in the asked the questions. The interviews have been transcribed development: “What I usually do when I work with verbatim. The quotations provided in the text are however products like this is to look at the vision. The strategic not always verbatim, but sometimes slightly rephrased in vision for the product is stated and then I describe that in order to be more readable and representative. terms of prototypes and develop it from that. So usually I In the data analysis three predefined categories were used, work both on a strategic level and in the actual production as described for example by Silverman [16]. The categories (in the development of the product). “ are: 1) The UX specialists way of working, 2) their remarks The UX specialist explains that his way of working is an on UX evaluation and 3) the challenges they have adaptation of his UX work made for Scrum. The strategic encountered when working with UX and Scrum. The vision and the UX goals are necessary to define before the interviews were read through and coded by two researchers actual project starts, ie before the sprints, but also to have in according to the predefined categories. The writing of this mind during the whole project when defining what to do in workshop paper was also a part of the analysis. Data was the different sprints. He stresses that he and the team work discussed and interpreted as a part of the writing like in with the UX vision before the project as well as during the Wolcott [22]. project and that the vision and the development work needs to run in parallel.“We work in the strategic level usually have a good vision of the UX experience during the both before the sprints, before the project starts, but also project:“I think it’s more important to do user testing during the project you need to both develop a vision to get before production (development) starts and then every now the big picture, basically about the whole user experience and then on the actual products to make sure.” and then from here we can decide that okay here’s a chunk The UX specialist stresses the importance of doing UX of work that needs to go into production. Then it goes into evaluations on big chunks of functionality when working in the Scrum project. It’s not like first we do a lot of work Scrum projects. He maintains that doing tests on small beforehand and then suddenly the project starts and we do pieces of functionality is unimportant, as it is the big picture nothing more. Because I think it needs to be developed in that adds to the UX. The timing of the UX evaluations parallel.” depends on the progress in the project, and in his When asked about if he is a member of the team or outside experience user tests should be done as soon as there is a team he answers: “A bit of both. I was a member of the enough to evaluate. He describes the timing of user teams, but at the same time I become more, almost like, evaluation by saying:“That could be anytime when you since I worked more on the requirements part of the have a decent chunk of functionality to test. Again this is development I was a bit of both, you need to be both on the because I think it’s more important for us as usability requirement side and also part of the actual production to people to test the big picture, to get the full, it’s not like make it work.” okay I know we are able to log in, but it’s not so interesting; the interesting part is when you have the big The UX specialist explains this double role of the picture in place. And then you can test maybe a number of development of the vision before the project, and the use of things at the same time.” the vision in the development: “You need to get the big picture, but you also need to be involved in the actual Challenges Encountered –UX vision Difficult to Maintain production to be sure of that what’s actually produced is The UX Specialist describes that despite his work and what was decided on in the first place.“ experience with the UX vision it is especially difficult to get an overview of the UX in Scrum projects:“The UX Evaluation – Common Understanding is Vital drawback in Scrum is that it’s so feature oriented and the When asked about how the UX is evaluated the UX problem is that you don’t have a big picture of the whole specialist explains that a common understanding of the UX user experience.” experience is crucial in Scrum projects since he is often not directly involved in the work during the sprints. Hence, he The UX specialist describes the challenge further, and often has meetings before the sprints with developers and maintains that the Scrum process that focuses on delivering testers to set the requirements together, and to decide what small pieces of functionality suits most programmers to include in the different sprints. The goal of these perfect. The programmers have a responsibility to deliver a meetings is to have a common understanding in the team small piece of the software, but they often do not feel regarding the product and the UX. Note that the UX responsible for the UX or the whole system:“I guess for a specialist uses the word testing while he is actually is programmer it’s perfect to get a small piece of work that talking about evaluation of the user interface.“When you you can work on and deliver. But the problem is that there have the vision clear, you can make sure that this user is now no one that actually is responsible for putting this requirement is going to be implemented in this sprint and piece of functionality into the big picture. So there is no one before the sprint starts you make sure that all the detailed responsible for the actual full user experience. That’s the requirements are set. And then we walk through it. The problem.” detailed requirements are something that we do together, The UX specialist describes that it can be difficult to some members of the team, and make sure that they are in maintain the UX vision in Scrum projects: “After a while place before the actual sprint starts. And we usually have you have added so many features that you don’t know meetings together with some developers and testers. The where to put them anymore. And if you don’t have the testers can make sure that they have the test cases in place vision clear in your head or on paper it’s starting to get based on this. Developers can make sure that they know quite difficult to know what to do with this piece of what to do before the sprint starts. And then during the functionality and then you do something, just to squeeze it sprint you can be there for ad hoc discussions when a case in. And that’s the reason for that I think it’s so important to needs to be straightened out. But most of the work is done do a thorough pre-study before prototyping and testing. before the sprint starts and some of the work is done during Because if you have that it’s so much easier to prioritize the sprint. Then of course when they (the team) have and say okay say that from this vision we have decided to something to show, you can actually test it by walking do this piece now and that piece then. At least we know through it yourselves.” where it all fits in. And then of course this vision will The UX specialist explains the importance of doing UX change all the time, because the market changes or evaluation before the actual development starts in order to whatever. But still you can work on the vision then and was done in pre-studies. Note that the UX specialist uses know where to put the pieces.” the word testing while she actually is talking about evaluation of the user interface: “We just did a huge task The Female UX Specialist and we worked for three weeks doing the prototypes and Way of Working – UX is a Part of the Whole Project testing the prototypes and those kind of things, before the The second UX specialist explains her way of working and development actually started. I started small defining it and it is noticeable that she works together with developers as then we tested out and then we came up with a concept and well as managers, in other words she is both working in and we presented it for developers and the product managers. outside the development team. She does different kinds of This is what we think it should look like, feel like, these UX activities throughout the project:“Right now I’m a user features are what we need in this system to be able to experience manager. Basically it’s the role of an support it and all the motivation around the concept. Then interaction designer, but I don’t do the visual design at all. we changed it a little bit then we got that input.” We have designers doing that. My main part is to come up The UX specialist describes that often she gives feedback to with low-fi prototypes and wireframes and stories and those the developers. Here it is noticeable that this UX specialist kind of things. I’m looking at the information architecture, manages to give severe critique to the developers without the interaction architecture and then I handle it over to them becoming really irritated. Sometimes they react when people who design them and implement them. In the end I she gives comments, but she manages to solve the situation also do testing as well and I take care of the focus group.” by joking and smiling. In the following the UX specialist The specialist also explains how she prepares her work. She describes how she managed to keep good attitude in the works in a very strategic way and influences the people that team:“My bosses are saying you’re too diplomatic. You have informal power in the project. She sees to it that her should be more strict. You should point with your hand and solutions are presented to the developers by someone they say this is wrong. But I don’t believe in that. I’ve gained listen to, and she does not do the presentation respect from not doing that, so that’s what I’m telling them. herself:“When I come up with a solution I don’t do it If I had come in and starting doing that in the beginning, I myself. I always consult their leaders. You pick the mean I don’t think it would work but right now you know developer that they are listening to. You kind of work they (the developers) don’t feel threatened. It’s been more around with them (the development leaders), then they are of collaboration and I’ve told them that this is how I am as the ones telling the developers that this is technically a person as well. I could have pointed and said do it like possible and it suits our platform and it’s definitely best this. It’s a give and take.” way to go.” Furthermore, the UX specialist describes a very informal The UX specialist explains further that in her experience approach to UX evaluation where paper prototypes are the UX activities need to be a part of the whole project, used. She usually tests her paper prototypes on developers from idea to testing and the UX people need to work in working in the company, and invites users over lunch to parallel to the developers:“Right now the company is really make them evaluate the prototype:“We tested these focusing on user experience and have that as a mission to prototypes in-house and with two contacts outside the enhance it and provide, well as they say in the business company that I can test quickly with because they know who goal, the business strategy to have an exceptional user I am. Then it’s a little bit more simple to say like: ‘Hey let’s experience so we’re going from a team of one (me) to ten take a lunch and you can come here and test the product, people I think… we’re going to have a team, user rather than making such a big deal out of it. Because it’s experience team that I’m going to be managing with I think just a paper-prototype so it’s quite hard to get people three interaction designers, two web designers and four motivated to come here and assign an hour and leave their developers as well. Because we want the whole chain. We job.” “I think that’s the biggest struggle that we have don’t want interaction focusing on one thing and then getting people motivated.” handing it over to development, and then implementing and then testing. We want it to be within the same team, all the Challenges Encountered –Timing of UX Evaluation is Hard expertise.” The UX specialist explains that it is hard to find a good timing for the UX evaluation in Scrum. She explains that The UX specialist explains the motivation for this change is evaluation too early in the project is difficult since the that the company is selling UX rather than features:“The different features are too small to be relevant to evaluate the company has noticed that it makes money. I think that’s the full user experience with users. If sufficient amount of main force. I think they were selling features, now they are features have been developed to evaluate then it is difficult selling experience rather than features.” to make large changes on the product because some parts of it have already been delivered to the customers and there is UX Evaluation – The Value of Social Skills little time to evaluate the remaining part before the delivery. The UX specialist describes the UX evaluation of one She explains: “When one back-log item (user requirement) particular feature, and explains that most of the evaluation is done you can’t really test that separately. Because it’s just a component within the whole feature you know. So prototypes are used by the UX specialist as a means to that’s really a little bit hard because then you have to wait evaluate UX early on in the project. for all the components to be ready and that can take, it took two months I think to get it ready and then I could test it The Collaboration - UX Specialists and the Team again. But then you can just tweak it a little bit, you can’t Both our informants believe that UX specialists need to do big changes there.” work closely with the developers. It has been suggested that the UX design should happen in a parallel track to the DISCUSSION development track [12,13,18]. Still, the UX specialists should view themselves as a part of the team because the The Big Picture of UX is Missing in Scrum Projects Both our UX specialists mention that Scrum is feature team needs to include everyone necessary to go from idea oriented. One of them stresses that a consequence of this to implementation [3]. According to our informants the real characteristic is that the big picture of the user experience is life situation is not necessarily as black and white as often missing in Scrum projects. One of their challenges is described in the literature. They describe their roles as to keep their vision of the user experience of the whole being both in and outside the development teams. Our software, while small pieces of the software are developed informants use informal ways of collaborating with the in each sprint. Salah et al. [13] argue that the HCI developers, like what is practiced in general in agile community and agile community do not share the same development. Both our UX specialists stress that understanding of how much design and how detailed it maintaining good co-operation with the developers is vital. needs to be before the actual implementation starts. The It is also noticeable that the UX specialists do only mention agile developers argue that UX designers want “big design a few documents in their way of explaining their work. It up front”, meaning that the design needs to be complete and seems that most collaboration is informal and oral. documented, but UX design iterative in nature [1]. The Responsibility for UX in Scrum is Complex developers’ concern is that the requirements will change so One of our UX specialists explains the big picture of UX is much, so designing big parts of the software up-front will missing also because of the lack of responsibility for UX in be a waste of time because some parts of the design will Scrum projects. One interesting aspect in software never be used. Successful projects that have significant user development is defining the responsibility for particular interaction have found that some level of design is activities. Responsibility here may refer to either the state necessary before the implementation [19]. It has also been of having a duty to deal with something, or the state of argued that the fundamental requirements from the users do being accountable or to blame for something. This can be not change that substantially, so designing the fundamental seen as either a rule based view of responsibility, or a user interaction up-front will not be a waste of time [1]. A consequence based view, as in Gotterbarn [5]. This problem vision of the user experience needs to be made before the can also be found in other system development processes, implementation starts, but it needs to be iterated during the as is reported in Boivie et al. [2]. The notion of whole Scrum project, like one of our UX specialists responsibility for UX is closely related to discussions of stresses. It seems like the HCI and the agile communities do responsibility generally in social science in relation to not agree on how much and how detailed design is needed groups. Here phenomena such as “the diffusion of before the actual implementation starts. responsibility” and the notion of “somebody else’s problem” are interesting to investigate. Diffusion of Designing One Sprint Ahead of Implementation Both our informants describe the need of designing the user responsibility is a social phenomenon, which might occur, experience some days before the implementation of one in larger groups, where no one in the group takes particular feature starts. Some researchers have suggested responsibility for phenomena. When a task is placed before this [12,13,18] but there is a conflict in their guidelines on a group of people, there is a tendency for each individual to when the design and evaluation of the user interface should assume someone else will take responsibility for it—so no take place. Sy et al. suggests that design happens one sprint one does. This is a negative outcome that might occur in ahead the implementation and the evaluation one sprint groups where responsibility is not clearly assigned. after the implementation [18], but Salah et al. suggest that Previous research in the area have indicated that the the particular UI is designed two sprints before diffusion of responsibility might have negative effects in implementation and evaluated one sprint ahead [13]. Both systems development [5]. our informants seem to design and evaluate before the CONCLUSION implementation of particular feature. One our UX specialist It is hard to make any general conclusions from our study in our study stresses that getting a clear vision of the user because we only analysed the interviews with two UX experience at the very beginning of a project is vital. When specialists. Still, it can be concluded that working on designing during the project the UX specialist uses the projects using the software development process Scrum vision as a reference point. If requirements change the affects the UX specialists’ way of working and their vision is changed too. It is also noticeable that low fidelity possibilities to conduct UX evaluation. Furthermore, the challenges that these two UX specialists are facing while 10. Larusdottir, M. K., Bjarnadottir, E., Gulliksen, J. The planning, conducting and describing the results of UX Focus on Usability in Testing Practices in Industry. In: evaluation are considerably affected by the overall values of Proc. HCI Symposium at WCC 2010, (2010), 98-109. Scrum especially that Scrum is feature oriented, and 11. Law, E.L.-C. The measurability and predictability of informal co-operation in the team is emphasised. user experience. In: Proc. 3rd ACM SIGCHI symposium on engineering interactive computing systems, ACM ACKNOWLEDGMENTS We would like to thank COST Action IC0904, Twintide, Press (2011). for the financial support that they have provided. Also, we 12. Miller, L. Case study of customer input for a successful would like to thank the two informants in the study that product. In: Proc. Agile 2005, IEEE Computer Society took their time to participate. (2005), 225-234. REFERENCES 13. Salah, D. and Petrie, H. Towards a framework for 1. Beyer, H. User-Centered Agile Methods. In: Carrol, J. integrating user centered design and agile software M. ed. Synthesis lectures on human-centered development processes. In: Proc. Irish CHI 2009, informatics, Morgan & Claypool Publishers (2010). (2009). 2. Boivie, I., Gulliksen, J. and Göransson, B. The 14. Schwaber, K., 1995. Scrum development process. In: lonesome cowboy: A study of the usability designer role SIGPLAN Notices, 30(10). in system development. Interacting with Computers, 15. Singh, M. U-SCRUM: An agile methodology for 18(4), (2006) 601 – 634. promoting usability. In: Proc. AGILE '08, IEEE 3. Cohn, M. Succeeding with agile, Addison-Wesley, Computer Society, (2008). USA, (2010). 16. Silverman, 2011. Qualitative Research, 3rd edition, 4. Detweiler, M. Managing UCD within agile projects. Sage Publications Ltd., UK. Interactions, 14(3), (2007), 40-42. 17. Sohaib, O. and Khan, K. Integrating usability 5. Gotterbarn, D. Informatics and professional engineering and agile software development: A responsibility. Science and Engineering Ethics 7(2), literature review. In: Proc. ICCDA 2010, (2010). (2001), 221-230. 18. Sy, D. Adapting usability investigations for agile user- 6. Hassenzahl, M. The thing and I: Understanding the centered design. Journal of usability Studies, 2(3), relationship between user and product. In: K. Blyth, (2007), 112-132. P.C. Overbeeke, A.F. Monk, P.C. Wright eds., 19. Takats A. and Brewer N. Improving Communication Funology: From usability to enjoyment. Kluwer between Customers and Developers. In: Proc. Agile Academic Publishers, (2003) 1-12. 2005, IEEE Computer Society, (2005). 7. ISO IS 9241-11. Ergonomics of human system 20. Vermeeren, A.P.O.S., Law, E.L.-C., Roto, V., Obrist, interaction - Part 11: Guidance of usability. ISO, M., Hoonhout, J. and Väänänen-Vainio-Mattila, K. User Switzerland, (1998). experience evaluation methods: Current state and 8. ISO IS 9241-210. Ergonomics of human system development needs. In: Proc. NordiCHI 2010, ACM interaction - Part 210: Human-centred design for Press (2010). interactive systems, ISO, Switzerland, (2010). 21. Williams, H. and Ferguson, A.The UCD perspective: 9. Larusdottir, M.K., Haraldsdottir, O. and Mikkelsen, B. Before and after agile. In: Proc. Agile 2007. IEEE User involvement in Icelandic software industry. In: Computer Society, (2007) 285-290. Proc. I-Used 2009 workshop at INTERACT 2009, 22. Wolcott, H. 2009. Writing up Qualitative Research. (2009), 51 – 52. California, Sage Publications Inc.