The Scrum development process has gained increasing popularity during the last decade. At the same time user experience (UX) has emerged as an important quality feature. However, the integration of UX related activities into Scrum projects has not been without problems, and this area needs to be further examined. This paper describes the results from two in depth interviews with knowledgeable UX specialists working in Scrum projects in the product development industry. It describes their ways of working generally with UX, their experiences from UX evaluations and the challenges encountered from their UX work. The main concern when working with UX in Scrum projects is that the big picture of UX is often lacking. Finally, the paper discusses the differences and similarities between the experiences from the UX specialists.
INTRODUCTION
The international standard ISO 9241-210 defines user
experience as: "a person's perceptions and responses that
result from the use or anticipated use of a product, system
or service" [
In software development the need to focus on UX keeps
increasing as products and services become more
competitive and need to function in a much broader context
than previously. Recently development processes of a more
agile nature have emerged that put emphasis on team work
and production rather than on structure and documentation.
One of the most popular agile software development
processes is Scrum [
User Experience Measures and Evaluation Researchers agree that UX is a complex concept, including aspects like fun, pleasure, beauty and personal growth. UX focuses on the more emotional aspects of user interactions, shifting the focus on how the users feel while and after using the software, the sensation, and the meaning as well as the value of such interactions in everyday life. Evaluating the UX has been a challenge for IT professionals.
UX is described by Hassenzahl as having pragmatic and
hedonic attributes [
UX in Scrum Projects
The agile process Scrum has gained popularity in the
software industry in the Nordic countries in recent years.
One third of IT professionals in Iceland used this process in
2009 [
The Scrum development process has been criticized for not
involving real users in the software development and for
not adequately addressing their usability needs [
Some researchers have suggested that some human-centred
activities are conducted before the actual implementation in
the project starts in order to address usability from a more
holistic perspective. This is also the method used in the
organization described by Sy [
METHOD This workshop paper presents the experiences that two much knowledgeable UX specialists had from integrating UX evaluation into Scrum projects. These two UX specialists were interviewed in a large interview study focusing on the integration of a wider concept, namely User Centred Design in Scrum practice made in 2010. For the purpose of this workshop paper these interviews haveconseque been re-examined for data regarding UX evaluation and Scrum and these two interviews were found. The two interviews were semi-structured and an interview template was used. The interviews were carried out on site and lasted for about an hour. Two researchers conducted the interviews. One researcher was taking notes and the other asked the questions. The interviews have been transcribed verbatim. The quotations provided in the text are however not always verbatim, but sometimes slightly rephrased in order to be more readable and representative.
In the data analysis three predefined categories were used,
as described for example by Silverman [
The male UX specialist is a 46 years old man who has 13 years of experience from working in different consultant companies. In his present employment he works for one of the largest IT companies in the Nordic countries with about 10 000 employees and 14 000 customers. His job title is usability designer, and he holds a PhD in Human Computer Interaction with the focus of adding a usability and user experience perspective in software development. He has worked with the integration of Scrum and UX in several different projects in industry. These projects range from "public interfaces to internal systems" but his main focus during the last years has been on public applications. The female UX specialist is a 35 years old woman with a Master-degree in media technology science with the specialization towards human computer interaction and sound. She has worked in industry for four years. Her present employer is a large Swedish product development company founded in 1994. The company has offices in eight countries and clients from all over the world. During her four years as a UX specialist she has made much progress in her company, and she has managed to establish UX as a core activity. Her formal role is a user experience manager, and she is in the middle of a process to hire ten members for a UX team where she will be the manager. The products she is working on are adaptable custom products related to social media and the web.
RESULTS In this section the experiences made by the UX specialists are presented. Each person’s experience is categorized according to the three predefined categories: Ways of working, experience from UX evaluations in Scrum and challenges encountered. The experiences that each person has had are presented separately with the help of quotations from the interviews.
The Male UX Specialist
Way of Working – Importance of UX vision
The male UX specialist describes that the strategic vision of
the product is very important for him and that he uses that
as a starting point in his work with prototypes in the
development: “What I usually do when I work with
products like this is to look at the vision. The strategic
vision for the product is stated and then I describe that in
terms of prototypes and develop it from that. So usually I
work both on a strategic level and in the actual production
(in the development of the product). “
The UX specialist explains that his way of working is an
adaptation of his UX work made for Scrum. The strategic
vision and the UX goals are necessary to define before the
actual project starts, ie before the sprints, but also to have in
mind during the whole project when defining what to do in
the different sprints. He stresses that he and the team work
with the UX vision before the project as well as during the
project and that the vision and the development work needs
to run in parallel.“We work in the strategic level usually
both before the sprints, before the project starts, but also
during the project you need to both develop a vision to get
the big picture, basically about the whole user experience
and then from here we can decide that okay here’s a chunk
of work that needs to go into production. Then it goes into
the Scrum project. It’s not like first we do a lot of work
beforehand and then suddenly the project starts and we do
nothing more. Because I think it needs to be developed in
parallel.”
When asked about if he is a member of the team or outside
a team he answers: “A bit of both. I was a member of the
teams, but at the same time I become more, almost like,
since I worked more on the requirements part of the
development I was a bit of both, you need to be both on the
requirement side and also part of the actual production to
make it work.”
The UX specialist explains this double role of the
development of the vision before the project, and the use of
the vision in the development: “You need to get the big
picture, but you also need to be involved in the actual
production to be sure of that what’s actually produced is
what was decided on in the first place.“
UX Evaluation – Common Understanding is Vital
When asked about how the UX is evaluated the UX
specialist explains that a common understanding of the UX
experience is crucial in Scrum projects since he is often not
directly involved in the work during the sprints. Hence, he
often has meetings before the sprints with developers and
testers to set the requirements together, and to decide what
to include in the different sprints. The goal of these
meetings is to have a common understanding in the team
regarding the product and the UX. Note that the UX
specialist uses the word testing while he is actually is
talking about evaluation of the user interface.“When you
have the vision clear, you can make sure that this user
requirement is going to be implemented in this sprint and
before the sprint starts you make sure that all the detailed
requirements are set. And then we walk through it. The
detailed requirements are something that we do together,
some members of the team, and make sure that they are in
place before the actual sprint starts. And we usually have
meetings together with some developers and testers. The
testers can make sure that they have the test cases in place
based on this. Developers can make sure that they know
what to do before the sprint starts. And then during the
sprint you can be there for ad hoc discussions when a case
needs to be straightened out. But most of the work is done
before the sprint starts and some of the work is done during
the sprint. Then of course when they (the team) have
something to show, you can actually test it by walking
through it yourselves.”
The UX specialist explains the importance of doing UX
evaluation before the actual development starts in order to
have a good vision of the UX experience during the
project:“I think it’s more important to do user testing
before production (development) starts and then every now
and then on the actual products to make sure.”
The UX specialist stresses the importance of doing UX
evaluations on big chunks of functionality when working in
Scrum projects. He maintains that doing tests on small
pieces of functionality is unimportant, as it is the big picture
that adds to the UX. The timing of the UX evaluations
depends on the progress in the project, and in his
experience user tests should be done as soon as there is
enough to evaluate. He describes the timing of user
evaluation by saying:“That could be anytime when you
have a decent chunk of functionality to test. Again this is
because I think it’s more important for us as usability
people to test the big picture, to get the full, it’s not like
okay I know we are able to log in, but it’s not so
interesting; the interesting part is when you have the big
picture in place. And then you can test maybe a number of
things at the same time.”
Challenges Encountered –UX vision Difficult to Maintain
The UX Specialist describes that despite his work and
experience with the UX vision it is especially difficult to
get an overview of the UX in Scrum projects:“The
drawback in Scrum is that it’s so feature oriented and the
problem is that you don’t have a big picture of the whole
user experience.”
The UX specialist describes the challenge further, and
maintains that the Scrum process that focuses on delivering
small pieces of functionality suits most programmers
perfect. The programmers have a responsibility to deliver a
small piece of the software, but they often do not feel
responsible for the UX or the whole system:“I guess for a
programmer it’s perfect to get a small piece of work that
you can work on and deliver. But the problem is that there
is now no one that actually is responsible for putting this
piece of functionality into the big picture. So there is no one
responsible for the actual full user experience. That’s the
problem.”
The UX specialist describes that it can be difficult to
maintain the UX vision in Scrum projects: “After a while
you have added so many features that you don’t know
where to put them anymore. And if you don’t have the
vision clear in your head or on paper it’s starting to get
quite difficult to know what to do with this piece of
functionality and then you do something, just to squeeze it
in. And that’s the reason for that I think it’s so important to
do a thorough pre-study before prototyping and testing.
Because if you have that it’s so much easier to prioritize
and say okay say that from this vision we have decided to
do this piece now and that piece then. At least we know
where it all fits in. And then of course this vision will
change all the time, because the market changes or
whatever. But still you can work on the vision then and
know where to put the pieces.”
The Female UX Specialist
Way of Working – UX is a Part of the Whole Project
The second UX specialist explains her way of working and
it is noticeable that she works together with developers as
well as managers, in other words she is both working in and
outside the development team. She does different kinds of
UX activities throughout the project:“Right now I’m a user
experience manager. Basically it’s the role of an
interaction designer, but I don’t do the visual design at all.
We have designers doing that. My main part is to come up
with low-fi prototypes and wireframes and stories and those
kind of things. I’m looking at the information architecture,
the interaction architecture and then I handle it over to
people who design them and implement them. In the end I
also do testing as well and I take care of the focus group.”
The specialist also explains how she prepares her work. She
works in a very strategic way and influences the people that
have informal power in the project. She sees to it that her
solutions are presented to the developers by someone they
listen to, and she does not do the presentation
herself:“When I come up with a solution I don’t do it
myself. I always consult their leaders. You pick the
developer that they are listening to. You kind of work
around with them (the development leaders), then they are
the ones telling the developers that this is technically
possible and it suits our platform and it’s definitely best
way to go.”
The UX specialist explains further that in her experience
the UX activities need to be a part of the whole project,
from idea to testing and the UX people need to work in
parallel to the developers:“Right now the company is really
focusing on user experience and have that as a mission to
enhance it and provide, well as they say in the business
goal, the business strategy to have an exceptional user
experience so we’re going from a team of one (me) to ten
people I think… we’re going to have a team, user
experience team that I’m going to be managing with I think
three interaction designers, two web designers and four
developers as well. Because we want the whole chain. We
don’t want interaction focusing on one thing and then
handing it over to development, and then implementing and
then testing. We want it to be within the same team, all the
expertise.”
The UX specialist explains the motivation for this change is
that the company is selling UX rather than features:“The
company has noticed that it makes money. I think that’s the
main force. I think they were selling features, now they are
selling experience rather than features.”
UX Evaluation – The Value of Social Skills
The UX specialist describes the UX evaluation of one
particular feature, and explains that most of the evaluation
was done in pre-studies. Note that the UX specialist uses
the word testing while she actually is talking about
evaluation of the user interface: “We just did a huge task
and we worked for three weeks doing the prototypes and
testing the prototypes and those kind of things, before the
development actually started. I started small defining it and
then we tested out and then we came up with a concept and
we presented it for developers and the product managers.
This is what we think it should look like, feel like, these
features are what we need in this system to be able to
support it and all the motivation around the concept. Then
we changed it a little bit then we got that input.”
The UX specialist describes that often she gives feedback to
the developers. Here it is noticeable that this UX specialist
manages to give severe critique to the developers without
them becoming really irritated. Sometimes they react when
she gives comments, but she manages to solve the situation
by joking and smiling. In the following the UX specialist
describes how she managed to keep good attitude in the
team:“My bosses are saying you’re too diplomatic. You
should be more strict. You should point with your hand and
say this is wrong. But I don’t believe in that. I’ve gained
respect from not doing that, so that’s what I’m telling them.
If I had come in and starting doing that in the beginning, I
mean I don’t think it would work but right now you know
they (the developers) don’t feel threatened. It’s been more
of collaboration and I’ve told them that this is how I am as
a person as well. I could have pointed and said do it like
this. It’s a give and take.”
Furthermore, the UX specialist describes a very informal
approach to UX evaluation where paper prototypes are
used. She usually tests her paper prototypes on developers
working in the company, and invites users over lunch to
make them evaluate the prototype:“We tested these
prototypes in-house and with two contacts outside the
company that I can test quickly with because they know who
I am. Then it’s a little bit more simple to say like: ‘Hey let’s
take a lunch and you can come here and test the product,
rather than making such a big deal out of it. Because it’s
just a paper-prototype so it’s quite hard to get people
motivated to come here and assign an hour and leave their
job.” “I think that’s the biggest struggle that we have
getting people motivated.”
Challenges Encountered –Timing of UX Evaluation is Hard
The UX specialist explains that it is hard to find a good
timing for the UX evaluation in Scrum. She explains that
evaluation too early in the project is difficult since the
different features are too small to be relevant to evaluate the
full user experience with users. If sufficient amount of
features have been developed to evaluate then it is difficult
to make large changes on the product because some parts of
it have already been delivered to the customers and there is
little time to evaluate the remaining part before the delivery.
She explains: “When one back-log item (user requirement)
is done you can’t really test that separately. Because it’s
just a component within the whole feature you know. So
that’s really a little bit hard because then you have to wait
for all the components to be ready and that can take, it took
two months I think to get it ready and then I could test it
again. But then you can just tweak it a little bit, you can’t
do big changes there.”
DISCUSSION
The Big Picture of UX is Missing in Scrum Projects
Both our UX specialists mention that Scrum is feature
oriented. One of them stresses that a consequence of this
characteristic is that the big picture of the user experience is
often missing in Scrum projects. One of their challenges is
to keep their vision of the user experience of the whole
software, while small pieces of the software are developed
in each sprint. Salah et al. [
Designing One Sprint Ahead of Implementation
Both our informants describe the need of designing the user
experience some days before the implementation of one
particular feature starts. Some researchers have suggested
this [
The Collaboration - UX Specialists and the Team
Both our informants believe that UX specialists need to
work closely with the developers. It has been suggested that
the UX design should happen in a parallel track to the
development track [
CONCLUSION It is hard to make any general conclusions from our study because we only analysed the interviews with two UX specialists. Still, it can be concluded that working on projects using the software development process Scrum affects the UX specialists’ way of working and their possibilities to conduct UX evaluation. Furthermore, the challenges that these two UX specialists are facing while planning, conducting and describing the results of UX evaluation are considerably affected by the overall values of Scrum especially that Scrum is feature oriented, and informal co-operation in the team is emphasised. ACKNOWLEDGMENTS We would like to thank COST Action IC0904, Twintide, for the financial support that they have provided. Also, we would like to thank the two informants in the study that took their time to participate.