Involvement of end user in Scrum-driven organizations: Four anti- patterns

Abstract

Agile software development processes, especially Scrum, have become more common in the software industry due to the fundamental principles that focus on speed and communication. However, it is noteworthy that even though the Scrum framework emphasizes frequent stakeholder involvement, the involvement of end users is often not a primary focus within organizations using Scrum, nor is usability a high-priority task. In the worst cases, the result is a low-quality product that is unable to meet user needs. To better understand the role of end users and how usability issues are handled in Scrum processes in practice, we performed interviews with professionals practicing Scrum, focusing on two areas: 1) the role of end users and their involvement in the Scrum process, 2) the importance of usability and usability evaluation techniques used in practice. Our results identify four novel antipatterns in organizations using Scrum: 1) Involvement of end user is not mandatory, and a customer maybe used to represent an end user; 2) Product demonstration occurs with no end user involvement; 3) Low focus on usability and usability is often ignored; 4) No formal usability techniques used in Scrum process. Based on our interviews we discuss the reasoning of why certain practices are performed by a Scrum practitioner. Finally, the study is subject to several limitations that have also been addressed.

Keywords

• eol>Scrum Challenges
• End User Involvement
• Agile Software Development Process
• Usability 1

1. Introduction

Agile is a popular approach within software development processes [ 31 ]. However, requirements engineering (RE) challenges in agile have been identified by many studies. Agile software development is a lightweight method, that focuses on delivering working software, organizing, and making coding more efficient. As such, issues related to usability are often neglected, and end user involvement – while advocated for is not mandatory in the process [ 3, 5 ]. A Delphi study was conducted by Schön et al. [ 20 ] with 26 experts in agile to identify the key challenges in agile requirement engineering. The result showed six challenges for industry practicing agile processes and two of these challenges are related to stakeholders and users. A summary of these challenges is i) lack of involvement of direct end users ii) difficulty in involving stakeholders in the development process. A systematic literature review on “Obstacles Faced by Requirements Engineering Professionals in an Agile Context by Applying User Experience” also showed that “lack of user participation” is the common problem mentioned in most of the studies [ 1 ]. More specifically, to identify agile requirements engineering practices and challenges a systematic literature review was conducted by Inayat et al. [ 10 ]. The study identified eight challenges posed by agile requirements engineering and one of them is neglecting nonfunctional requirements such as testability and external quality usability. According to Helmy et al. [ 9 ] the issue related to nonfunctional requirements should be taken care of to improve the product quality. The aforementioned factors demand the necessity for engaging end users and focus more on usability when enacting agile methodologies.

In this paper, we focus on inspecting the most popular agile approach, Scrum [ 22 ], and how Scrum is utilized in industry. Scrum is a lightweight agile project management framework mostly used for software development. Scrum focuses on frequent stakeholder involvement and short releases, which is why Scrum is broadly adopted by industries. Since Scrum is an agile approach, it also faces some of these same requirements engineering challenges when put into practice [ 23 ]. Specific challenges include lack of time to focus on the end user’s needs [2], lack of techniques for the design process [ 21 ], and usability requirements being ignored [ 29 ]. However, proper utilization of Scrum depends on professionals. The identified challenges are not a Scrum framework challenge but a Scrum application challenge. According to the Scrum guide the Scrum framework is purposely incomplete. Different types of techniques and methods can be used within the framework in different situations and environments [ 8 ]. So, our goal is to focus on how practitioners are involving end users in Scrum processes and how they are ensuring usability of the product. The paper aims to answer the following research questions: In practice what is the role of end users? How are they involved in the Scrum process? How are usability issues handled? Our results identify four novel anti patterns in organizations using Scrum. Here we refer to anti patterns as the common Scrum practices that appear convenient in the beginning but harmful in the long run and therefore must be avoided [ 28 ]. The results also provide holistic understanding of the challenges faced by software companies related to end user involvement when employing Scrum.

The structure of the paper organized as follows. Section 2 provides related work. Section 3 discusses the research methodology and data analysis. The results of the study are presented on Section 4 and discussed on Section 5. Section 6 and Section 7 respectively address the limitations and conclusions of the study.

2. Related work

2.1. Involvement of end user in Scrum

Normally in Scrum, most frequent review activities such as end of sprint review and testing involve the product owner and the development team. End users are not commonly involved in these activities, but instead may be involved in feedback sessions when they are using the product. By this stage, change is still possible, but requires costly rework [ 15 ]. Even though agile processes such as Scrum are iterative approaches that conduct user testing in the process, when put into practice Scrum often does not focus on creating user experience based on user needs [2]. Most of the time, the Scrum team is having communication only with the customer. It is possible that the customer who is representing the end user is inexperienced, not efficient in providing feedback and not having enough knowledge. In that case the product may not work well with the end user [ 27 ]. There are exceptions where a sample of real users are also involved, but this happens only occasionally. Usually, it is the customer who represents user needs, requirements and gives feedback [ 24 ].

Even the role of end users versus customers is still unclear in many software development companies [ 18 ]. Whilst doing an analysis on the fitness of Scrum and Kanban towards user experience, researchers [ 25 ] noticed that interviewees were using the terms “customer” and “user” as synonyms which motivated them to do additional research on the issue. The result showed that whilst many respondents did understand the difference between “user” and “customer” there were still a significant number who were unclear about the differences and couldn’t answer properly or answered incorrectly.

One of the core values of agile development is that working software should be demonstrated to the end users throughout short development iterations. The reason behind this value is that working software elicits more detailed information toward understanding end user needs than written words. However, the situation is different in practical life. In one organization, team members were interviewed after a transformation to Scrum from a more traditional waterfall process. In their interview, they specified that they never did a full product demonstration. In other words, there wasn’t any demonstration after each sprint where they could gather feedback. Instead, they delivered the software when all the required functions were working. So essentially, they ended up doing a waterfall approach every three weeks [ 4 ].

2.2. Usability techniques used in Scrum

In many Scrum projects the Scrum team (product owners & developers) concentrate more on the goal of implementing a working function rather than the usability or usefulness of that function. A common argument here is that the usability can be improved later [ 17 ]. In practice, the overall result is a product that may obtain all the functional requirements but still fails in the market due to poor user satisfaction [ 14 ]. In [ 11 ] authors performed a survey study to find what usability techniques are used in Scrum processes and how often they are used. They listed 13 usability techniques used by the practitioners. Later, they performed another survey to find the usefulness and usage of these 13 techniques. Results showed that according to 75% of the respondents, formal usability evaluation with users is a very good technique but only 30% of participants used formal usability in their project. The author mentioned one possible explanation and that is that Scrum is all about speed. Scrum sprints are short (2-4 weeks). It takes a longer time to use a specific usability technique which may slow the whole Scrum process down. Also, the output of such sprints is minor, and evaluating them in a quantitative way is not useful as changes are small. Therefore, Scrum professionals tend to rely on informal usability techniques such as lo-fi prototyping, and meetings with users instead of formal techniques such as heuristic evaluation, questionnaires, field studies, scenarios, or digital prototyping. One advantage of informal usability techniques is they can be performed quickly without prior preparation. In some cases, informal usability techniques are helpful, but the main drawback of this approach is the number of users involved. Only a few users are taking part in informal usability evaluation, so evaluators are not sure if they can rely on the result or not [ 12 ]. In agile, especially within Scrum, usability requirements and their implementations are ignored most of the time. Usability or usefulness is often ignored or a less prioritized task in a Scrum process. Usability features are often absent in product backlog [ 29 ]. Another study result, on the user perspective in Scrum practice, showed that the involvement of a user in the process is informal, meaning that user involvement is done based on an individual’s own initiative, preferences and knowledge. There’s no systematic plan or approach within the development process and often there’s no specific person responsible for usability issues, nor any clear usability goals. As such, the user perspective is only considered if the product owner or team has an interest in usability [ 6 ]. One survey study was conducted on user involvement methods in Icelandic software industry [ 13 ]. 37% of the respondents were using Scrum as the software development process and rest of the respondents were using other methods including organization’s own process models, waterfall, and XP. The result shows that developers who were using Scrum are the most skeptical compared to others when asked about whether usability is important or not. The definition of usability within that study described it as “Usability is a qualitative attribute that assesses how easy user interfaces are to use. Usability is mainly made up of three factors: Effectiveness – Can the users solve their tasks with the software? Efficiency – Can the user solve their tasks without major problems? Satisfaction – How satisfied are the users?

Most of the literature on Agile or Scrum mainly focused on integrating UCD, UX [ 2, 17, 5, 3, 1, 15, 25 ] or requirement engineering challenges [ 4, 31, 7, 20, 10, 9 ]. Usability issues are considered in several studies [ 11, 12, 29, 13, 6 ] but with no proper discussion on the reasons for ignoring usability. While several researchers have been interested in end user’s perspectives in Scrum, the research does not reveal what is the role of end users in practice, or how practitioners are (or are not) involving them. Therefore, our research will focus on understanding not just what gaps exist in end user involvement in practice but why the involvement of end users and usability is often ignored.

3. Method

To familiarize ourselves with the issues related to end user involvement and usability within companies’ agile processes, we conducted semi-structured interviews with professionals having experience working with Scrum or working as a product owner (PO), scrum master (SM) or developers in a Scrum driven software organizations. We continued conducting interviews until a saturation point was reached in which no new information was emerging, which happened after 11 interviews had been conducted. We observed consistency among the interviewees as they articulated recurring themes and issues. All interviews were conducted online (MS Teams) and recorded by one researcher. The interviews lasted from 45 minutes to an hour.

3.1. Participants

The 11 individuals were selected randomly through personal networks, recommendations from people who had already been interviewed and via a post on social media (Facebook, LinkedIn). All individuals who participated in the research had practical experience working in Scrum or Agile as SM, PO and developer. There were in total 9 males and 2 females. More information can be found in Table 1. When participants were asked about what type of software development process they were using, most of them didn’t answer directly Scrum. Only two participants replied they were using Scrum. The Scrum process that others were using had been modified to adapt to the needs, requirements, and context of the organization. Similar to the outcome presented in [ 25 ], most of the organizations were using features of Scrum and combined Scrum with another agile development process, Kanban. A mix of Scrum and Kanban. Participants answered the questions based on the project they were working on. They were also asked to answer from one previous project where they had to involve end users, or the involvement of end users was mandatory. The interview questions were designed to be adaptable to the level of experience of the individuals and in accordance with the situation. The interview questions focused on two areas: 1) end user and 2) usability in the Scrum process. To collect demographic, background information (role, education, experience) and sign the ethical form (voluntary participation & data privacy), we performed a survey, using the GDPR-compliant survey tool Webropol, prior to the interview.

3.2. Data analysis

Interview recordings were transcribed and analyzed with inductive thematic analysis [ 30 ]. The first phase was to get familiarized with the data by reading transcripts, listening to the audio recordings, and making notes. In our case we highlighted all potential transcribed conversations related to our research question, putting notes and rephrased when necessary. The second phase was to generate initial codes which provided us with a summary of the highlighted conversations or describing the contents (see Table 2).

In phases 3 and 4 we constructed potential themes from codes and reviewed the themes against the extracted data. We reviewed all coded data from interviews for each question to generate themes, collapsing potential themes together or splitting them into multiple different themes. Example: For question, “What is the role of end user in your Scrum process?”, we noticed that the codes focused on either involving end users in the Scrum process or no involvement of the end users at all. We then constructed one theme containing all codes relating to no end user involvement in the Scrum process and one theme relating to involving end users. We further divided the theme involving end user into several different themes: involving end users, involving end users through product owner, involving end users through customer, involving end users through user representative, involving end users through sales or marketing team.

In phase 5 we defined and named the themes addressing our research questions. We also noticed some identified themes can be subthemes of another theme. Example: involving end users through the product owner, involving end users through the customer, involving end users through user representative, and involving end users through the service or marketing team can be described as subthemes of a broader theme “indirect involvement of end users”.

4. Results

Here the interview results will be presented focusing on two major areas: 1) the role of end users in Scrum, 2) the importance of usability and usability evaluation techniques used in practice.

4.1. Involvement of end users in sprint review session

According to the Scrum guide, a sprint review is one of the mandatory events to gather feedback from key stakeholders (which includes end users) but in practice, involving end users is not necessary. In sprint reviews, feedback sessions and testing are often done without any involvement of end user. Instead, domain experts are playing the role of an end user. In almost all cases, major participants in sprint review are the Scrum team and internal stakeholders or customer representatives who play the role of an end user. In this case, it is hard to determine how relevant the feedback is with respect to the needs of real end users [ 26 ]. Almost half of the interviewees (five out of eleven) answered that they are only involving internal stakeholders in the sprint review session. Three interviewees had two different sessions but only on demand. One is for the internal stakeholders and another session is for the customers or end users. Three interviewees informed us that they are involving external stakeholders in the sprint review session. Stakeholders are those groups of people who are impacted by the product outcome. Based on our analysis, we identified the following categories of stakeholders from interviews. The categorization is necessary to understand different types of stakeholder interviewees addressed during the interview:

External stakeholders: Customers who finance the project and end users who use the product or benefit from the product. In many cases customers and end users can be the same group of people.

Internal stakeholders: Management of the company, Scrum team, product owners, program manager, functional manager, supervisors, customer support people, UX designer. In many cases an internal stakeholder can also be a customer.

4.2. Defining the term “end user” and “customer”

Customer and end user are different terms, and both have different roles in the development process. Scrum teams and literature sometimes do not specify the differences between a customer and an end user in a project. From the previous research [ 25, 18 ] we identified that agile practitioners are not always clear on the role of end user and customer. The purpose of defining “end user” and “customer” was to determine the participant’s understanding of these terms. Based on the answers almost all participants had a proper understanding and clear distinction between the two terms “customer” and “end user”. After defining end user and customer, all interviewees were asked about the role of the end user in their Scrum process. Analysis of the answers resulted in three different themes: a) Indirect involvement of end users b) No involvement of end users c) Direct involvement of end users. Four out of eleven interviewees mentioned that they are indirectly involving end users through a middle person such as a product owner, sales team, UX designers, customers, or user representatives. However, one interviewee also mentioned that even though end users are not involved in the Scrum process, they have close contact with them. Almost half of the interviewees (5 out of 11) replied they are having no end user involvement in their Scrum process. Some are involved, but only in special cases: “I would say that no, not in daily, like all the sprints that we have. But it will be more of a special case when we have a special need. So, it is more of this special occasion when we decide that, hey, OK, now, we want to talk with them.” “But in the scrum process we don't involve the end user actually”.

Only 2 interviewees were directly involving end users in their process however one of them stated that they are involving end users only when they need, not on a regular basis: “We haven't gotten them involved in the like every Sprint, so to speak, but there when we do something some kind of feature a little bit bigger thing we usually uh set up some kind of pilot group and we talk to them over teams and sort of gather their feedback from that”

4.4. Ensuring end users' feedback during the development process

We already noticed from the results that most of the interviewees are not involving end users in their Scrum process, so how do they ensure their feedback is heard? Everyone described their own different approaches. We categorized them into two sections: 1) getting feedback from end users; 2) getting feedback from user representatives: “but their customer make decision should we do the changes or not? We have these different persons who are basically listening the users.” 8 participants were collecting feedback from user representatives. Only 3 participants mentioned that feedback is coming directly from end users.

4.5. Importance of usability, user experience and benefits of involving end users in the Scrum process

Even though from the responses it is apparent that end users are seldomly involved in the Scrum process, almost all of the participants (8 out of 11) agreed that usability or UX is very important, and their priority is either high or very high: “it is always about providing the user with the best experience. So, it’s definitely the most important thing.” “I would say that it should not be the highest but one of the highest. If it doesn’t satisfy a customer, then what’s the point anyway.” However, it was also pointed out from the conversation that the priority of usability or UX also varies from case to case: “let's say if the customer or end user is some external customer or external client, then I would say the user experience should come first but for internal customers the situation is a bit different.” Furthermore, participants were asked about their opinions on the benefits of involving end users and two themes emerged from the analysis: 1) Enhanced User Understanding and 2) Enhanced Product Quality.

4.6. Techniques or methods used to collect feedback or evaluate the usability of the product

When asked about the methods used for usability evaluation and collecting feedback, we got quite diverse answers. We tried to sort the answers into two categories: 1) formal usability evaluation; 2) informal usability evaluation (no systematic plan or approach sometimes based on individuals’ own initiative). Only 3 participants mentioned that they are using some type of process although two of these participants are not using any techniques in the current project. They referred to a past project where they have used formal usability techniques such as questionnaires, or feedback tools. Only one participant was using formal methods for usability testing. The rest of the participants didn’t have any systematic plan or approach to collect feedback or evaluate usability. Mostly they were using team conversations and sending emails: “Uh, yeah, just uh, we are sometimes requesting this information in teams.” Some also mentioned initiating meetings when needed: “Uh is the informal techniques, so if we need it then we initiate a meeting request and then book a calendar and then get the feedback from them”.

4.7. Product demonstration with end user

Product demonstration is one of the core values of Agile to understand end user needs. Almost all interviewed individuals informed us that they are having either regular or irregular product demonstrations. But our goal was to find out if they are having these demonstrations with end users or not. Analysis showed that only three interviewees were involving end users in product demonstrations, where one company was having regular product demos and another two had irregular product demos. The rest of the interviewees had product demonstrations with no end user involvement.

4.8. Main challenges of involving end users in Scrum process

We asked participants to point out the challenges of involving end users in Scrum from their own experiences. It was interesting to see that there wasn’t that much diversity in the answers. Almost everyone had similar opinions. There were mainly two types of answers: 1) User characteristics, expertise, and background; 2) Scrum’s limitation in facilitating optimal end user engagement. According to the interviewees, choosing the right end user is very important and a big challenge: “I would say it depends on the product and what you consider as end user and it it's also very important to choose the right end users.” “You need to get involved the right people on the right time and on the right place.” Some interviewees also pointed out one issue and that is lack of knowledge and experience. Sometimes you need to train the general end users to get meaningful feedback. Also, the feedback should not come from just one individual’s opinion. Feedback should come from the whole user group. So, it is always easier to involve someone knowledgeable and experienced who’s able to represent end user groups such as super users or user representatives. The other challenges mentioned by interviewees are all related to Scrum’s limitation in facilitating optimal end user engagement. The most common challenge was not being able to include end users in the sprint review process. According to the interviewees, the Scrum sprint review is intended to be only for the internal team members. Involving general users is often not possible as these sessions are mostly technical: “they're talking sort of user language and we're talking developer language. So, if you involve them in the same kind of in this well in the Scrum, scrum day-to-day or Sprint week to week thing, then that could be a little bit challenge.” One interviewee mentioned that it is the role of the product owner to translate the technical language and have conversations with end users. Involving end users in sprint review just will create a chaotic environment and also might delay the whole process. Furthermore, one interviewee added that Scrum focuses more on delivery so fitting user’s perspective in two to three weeks sprints is hard. End user involvement needs different planning.

5. Discussion

5.1. Involvement of end user is not mandatory, and a customer maybe used to represent an end user

There are several implications for Scrum due to low user involvement, these are (1) software testing: In Scrum, testing is done by the product owner and development team instead of the user, (2) Poor software quality: The user might not be satisfied from the software as no testing has been performed, (3) No feedback: There are no feedback sessions after sprints. A feedback session takes place after the software has been created. As a result, the Scrum team has little idea if they are going in the right direction or not [ 15 ]. In our interviews we asked interviewees about the role of end users in their Scrum process and whether they are involved in sprint review sessions. The results show that the majority of the interviewees are not involving end users in their Scrum process. Interviewees who agreed that they are involving end users mostly having indirect communication with the end user through product owners, user representatives, or customers. From studies, it is indicated that customer representatives often have daily contact with the developers which can make them too familiar with the inner software working process. So, they cannot truly represent an actual end user during the feedback session [ 21 ]. According to the Scrum guide it is true that the product owner should be the link between end user and development team. But in practice this is not always true. One study [ 19 ] showed that some product owners think that identifying end users need and communicating those needs with development team is not product owner’s responsibility but a team effort. Also, product owners in Scrum are mostly overwhelmed with marketing and sales concerns and often lack formal education and skills to design effective user experiences [ 17 ]. Sprint review sessions are mainly for the internal Scrum team or internal stakeholders. Only a few mentioned they involve external stakeholders, but external stakeholders are not always end users. According to one of these interviewees, their external stakeholders are experts from an end user organization. So, our study results stay partially in line with [ 2, 26, 28, 15 ] that product owners, user representatives, or customers play the role of end users and participate in the Scrum sprint review. End user involvement is not mandatory in the Scrum process. We pointed out from our interviews that some of the interviewees are developing products for internal customers who are also end users. When end users are internal customers then the focus is more on functionality rather than usability. Moreover, some interviewees were working in sectors where UCD is not important, such as the telecommunication sector where the priority is to have a system that works as a mediator to other systems. Optimizing something for human use is not always a key requirement. So, the necessity of involving end users in the Scrum process varies case by case.

5.2. Product demonstration occurs with no end user involvement

A product demonstration can be useful after short development iterations to better understand user needs. However, even though product demonstration is one of the core values of agile, in practice it is not happening. We found out from our study that in most cases, product demonstrations happen without any involvement of the end user. In few cases, users are involved but the session is not regular. Only when there’s something big to demonstrate. Based on our results reasons behind not having a product demonstration are: 1) the outcome of a 2–4-week sprint lacks substantial significance for demonstration; 2) thinking of the product demo as a technical event only for the internal stakeholders; 3) lack of communication with end users.

5.3. Low focus on usability and usability is often ignored

Usability is not something that is only related to the aesthetic or visual design of the interface. Usability has a deep relationship with Human-Computer Interaction (HCI). There are several human factors in HCI and if they are implemented properly the usability of software could be improved. Studies have shown that even completed software increments have sometimes been rejected by the end user due to poor usability [ 29 ]. Our results contradict the previous research outcome [ 13 ], [ 29 ] on the importance of usability among Scrum practitioners as our study shows that almost all the participants agreed that usability or user experience is very important. But no matter how important usability or user experience is according to the interviewee’s answers, their focus and necessity in the Scrum process vary based on the product and customer. As mentioned before, end users and customers can be the same group of people. The customer can be an internal or external stakeholder. The priority of usability depends on the nature of the customer. For external stakeholders, the priority is high: “let’s say if the customer or end user is some external customer or external client, then I would say the user experience should come first. But for internal customers the situation is a bit different. The expectations are different, so they just want something that works for them, so they don’t care about the usability or such.” The same goes for the product a Scrum team is developing or the service they are providing. For the team who are developing products to support internal stakeholders, usability is not the top priority. But if the team is developing something for an external stakeholder such as a mobile app for a bank’s general users, then the priority is high. Based on the interviewee’s answers, UX or usability is not important or a top priority task in the case of internal stakeholders who are also the customer and end users. According to [ 13 ], the first two factors of usability are: Effectiveness – Can the users solve their tasks with the software? Efficiency – Can the user solve their tasks without major problems? - means the product a team is developing should be accessible to the end user regardless of whether the end user is an internal or external stakeholder. This leads us to a new argument that Scrum practitioners do not always have a very good understanding of UX and usability.

A common reason for neglecting usability issues in Scrum is having a short time frame. Scrum sprints are very short, and the focus is on the delivery. According to [ 6, 25, 17 ], Scrum practitioners do not have enough time to take usability activities into account. Even if usability is in the backlog, it is not on the top and is often ignored as functionality is the top priority. Now it can be asserted that it’s the product owner’s responsibility to enhance product quality by focusing on usability aspects. It is indeed factual that the product owner is an indirect leader and has some authority to make decisions. However, a product owner rarely wants to be the only person making decisions for the whole team. This was identified as one of the challenges of a product owner. A product owner is often guided by the development team in what is technically right. Another challenge is to act as a communication link between end users and developers. Because there are development teams who do not know who their end users are [ 19 ]. It is challenging to prioritize usability when there’s no clear understanding of the end users.

The output from a 2–4-week sprint is minor. So, there’s no point in evaluating in a quantitative way. We found the same from our study by showing one of the main challenges of involving end users in Scrum by professionals is its own limitations such as not having any proper feedback session with end users: “that could be then a nice thing that maybe we could have some lightweight user involvement methods that we could use inside the scrum process”, short sprint cycles (usually 2-3 weeks): “It might take a bit more time, so it doesn't always fit into the basic cycle of two or three weeks”, focusing more on output not usability: “well, scrum process isn't really it…it doesn't really take usability aspects or UX into account much.” Professionals practicing Scrum often do not follow any systematic approach to usability. Scrum practitioners rely on informal qualitative evaluation. There’s a difference between the top usability techniques and usability techniques commonly used by Scrum practitioners. Even though some techniques are effective, they are not being used in practical situations. Our results show that the commonly used usability technique by the participants is informal usability evaluation. Only one participant was using a specific user testing method (formal usability evaluation) in their ongoing project. Our result stays consistent with practitioners’ top and frequently used usability techniques [ 11 ] which discussed that formal usability techniques are not used by practitioners even though practitioners think formal usability techniques are highly effective [ 18 ].

6. Limitations

The first limitation of this study is the sample size of the data. 11 interviews might be considered relatively small to generalize the findings to a larger population of Scrum practitioners or organizations. Also, the perspective of 11 individuals may not cover the diversity within the Scrum community. Secondly, the data was analyzed by one researcher and reviewed by two other researchers. Many aspects of the result might have been influenced by the researcher’s assumptions, biases, and perspectives. Lastly, project type can influence the need to involve end user in the Scrum process. The findings of the study may be limited to projects that may not prioritize end user involvement such as infrastructure development or internal process improvement.

7. Conclusion

This study investigates the role of end users and usability in Scrum processes by conducting interviews with Scrum practitioners. The study addresses our research question by identifying four novel Scrum anti patterns. Anti-patterns identified in our research can provide insights to the Scrum practitioners in improving their Scrum process by 1) reflecting on their process and becoming aware of any harmful practices; 2) having a clear understanding of different stakeholders (internal, external) of the product, what are their roles and value; 3) reconsidering the role of a customer and end user in their Scrum process. Engage the right stakeholder at the right time; 4) seek ways to ensure end user’s feedback such as by introducing frequent product demonstrations with stakeholders. Sprint review is an event to engage with different stakeholders and gather feedback. Our recommendation is to avoid turning sprint review into a technical event only for the internal stakeholders. The outcome of a short sprint may be a minor update, in that case, practitioners should set a milestone where the increment would be worth testing. Involve end users from the beginning of the process when necessary to ensure objectives aligned with end user needs; 5) focusing on usability regardless of whether the product is for an internal stakeholder or external stakeholder; 6) using formal usability techniques (such as formal usability evaluation, prototyping, workshops, interviews, surveys) instead of getting feedback from online conversations; 7) managing time effectively so that usability activities cannot be ignored due to time limitation. The study contributes to the literature regarding usability in Scrum and extends the line of research by highlighting a significant gap: how end users are involved in the Scrum process, why end users are often neglected and what are the challenges of involving end users? Furthermore, we discussed certain limitations experienced by Scrum practitioners and the rationale behind specific practices. The outcome we derived from the interviews gives a good picture of why end user involvement and usability are often ignored and difficult in many organizations. The field of research is building a deeper understanding of the challenges and as future research, specific strategies such as lightweight methods for integrating usability aspects and end user evaluation within the Scrum framework, to mitigate the identified challenges are needed.

Acknowledgements

We thank all the interviewees who shared their knowledge and participated in this study. Further we thank all individuals from academia and industry who contributed.

References

[1] [2] Almeyda , S. , Zapata Del Río , C. and Cohn , D.

( 2021 ) 'Integration of User Experience and Agile Techniques for Requirements Analysis: A Systematic Review' , Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) . Springer International Publishing. doi: 10 .1007/978-3- 030 -78221-4_ 13 .

Almughram , O. and Alyahya , S. ( 2017 ) ' Coordination support for integrating user centered design in distributed agile projects' , Proceedings - 2017 15th IEEE/ACIS International Conference on Software Engineering Research, Management and Applications , SERA 2017 , pp.

229- 238 . doi: 10 .1109/SERA. 2017 . 7965732 .

[3] Anwar , S. , Motla , Y.H. , Siddiq , Y. , Asghar , S. , Hassan , M.S. and Khan , Z.I. , 2014 , December. ' User-centered design practices in scrum development process: A distinctive advantage?' . In 17th IEEE International Multi Topic Conference 2014 (pp. 161 - 166 ). IEEE.

[4] Flouri , K. and Berger , H., 2010 . ' Agile Development-Scrum adopted in practice but not in principle .'

[5] Blomkvist , S. , 2005 . ' Towards a model for bridging agile development and user-centered design'. In Human-centered software engineering-integrating usability in the software development lifecycle (pp. 219 - 244 ). Dordrecht: springer Netherlands.

[6] Cajander , Å. , Larusdottir , M. and Gulliksen , J. ( 2013 ) ' Existing but not explicit - The user perspective in scrum projects in practice' , Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 8119 LNCS(PART 3) , pp. 762 - 779 . doi: 10 .1007/978-3- 642 -40477-1_ 52 .

[7] Curcio , K. , Navarro , T. , Malucelli , A. and Reinehr , S. , 2018 . ' Requirements engineering: A systematic mapping study in agile software development . ' Journal of Systems and Software , 139 , pp. 32 - 50 .

[8] Schwaber , K. and Sutherland , J. ( 2020 ) 'Scrum Guide V7', Agile Metrics : Agile Health Metrics for Predictability, (November ), pp. 133 - 152 .

[9] Helmy , W. , Kamel , A. and Hegazy , O. ( 2012 ) ' Requirements engineering methodology in agile environment' , International Journal of Computer Science Issues , 9 ( 5 5-3 ), pp. 293 - 300 .

[10] Inayat , I. , Salim , S.S. , Marczak , S. , Daneva , M. and Shamshirband , S. , 2015 . ' A systematic literature review on agile requirements engineering practices and challenges .' Computers in human behavior, 51 , pp. 915 - 929

[11] Jia , Y. , Larusdottir , M. K. and Cajander , Å. ( 2012 ) ' The usage of usability techniques in scrum projects' , Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 7623 LNCS, pp. 331 - 341 . doi: 10 .1007/978-3- 642 - 34347-6_ 25 .

[12] Lárusdóttir , M. , Cajander , Å. and Gulliksen , J. ( 2014 ) ' Informal feedback rather than performance measurements - User-centred evaluation in Scrum projects' , Behaviour and Information Technology , 33 ( 11 ), pp. 1118 - 1135 . doi: 10 .1080/0144929X. 2013 . 857430 .

[13] Larusdottir , M. K. , Haraldsdottir , O. U. and Mikkelsen , B. A. ( 2009 ) 'User involvement in icelandic software industry' , CEUR Workshop Proceedings , 490 , pp. 0 - 1 .

[14] Lundström , M. , Åberg , J. and Blomkvist , J. ( 2015 ) 'Perceptions of software developers' empathy with designers' , ACM International Conference Proceeding Series, 2015 -Janua, pp. 239 - 249 . doi: 10 .1145/2783446.2783563.

[15] Rahayu , P. , Sensuse , D.I. , Fitriani , W.R. , Nurrohmah , I. , Mauliadi , R. and Rochman , H.N. , 2016 , October. ' Applying usability testing to improving Scrum methodology in develop assistant information system . ' In 2016 International Conference on Information Technology Systems and Innovation (ICITSI) (pp. 1 - 6 ). IEEE.

[16] Scrum , A. , 2013 . What is Scrum?. We'Ve Been Covering Scrum in Such Detail It Makes You Sick. and Now

[17] Singh , M. ( 2008 ) 'U-SCRUM: An agile methodology for promoting usability' , Proceedings - Agile 2008 Conference , pp. 555 - 560 . doi: 10 .1109/Agile. 2008 . 33 .

[18] Ogunyemi , A.A. , Lamas , D. , Lárusdóttir , M.K. and Loizides , F. , 2019 . ' A systematic mapping study of HCI practice research .' International Journal of Human-Computer Interaction , 35 ( 16 ), pp. 1461 - 1486 .

[19] Kristinsdottir , S. , Larusdottir , M. and Cajander , Å., 2016 . ' Responsibilities and challenges of product owners at spotify-an exploratory case study.' In Human-Centered and Error- Resilient Systems Development: IFIP WG 13.2/13 .5 Joint Working Conference, 6th International Conference on Human-Centered Software Engineering, HCSE 2016 , and 8th International Conference on Human Error, Safety, and System Development , HESSD 2016 , Stockholm, Sweden, August 29-31 , 2016 , Proceedings 8 (pp. 3 - 16 ). Springer International Publishing.

[20] Schön , E.M. , Winter , D. , Escalona , M.J. and Thomaschewski , J. , 2017 . Key challenges in agile requirements engineering . In Agile Processes in Software Engineering and Extreme Programming: 18th International Conference, XP 2017 , Cologne, Germany, May 22-26, 2017 , Proceedings 18 (pp. 37 - 51 ). Springer International Publishing.

[21] Teka , D. , Dittrich , Y. and Kifle , M. ( 2018 ) ' Adapting lightweight user-centered design with the scrumbased development process' , Proceedings - International Conference on Software Engineering , pp. 35 - 42 . doi: 10 .1145/3195528.3195530.

[22] Kikitamara , S. and Noviyanti , A. A. ( 2018 ) 'A conceptual model of user experience in scrum practice' , Proceedings of 2018 10th International Conference on Information Technology and Electrical Engineering: Smart Technology for Better Society, ICITEE 2018 , pp. 581 - 586 . doi: 10 .1109/ICITEED. 2018 . 8534905 .

[23] Alhazmi , A. and Huang , S. , 2020 , May. ' Integrating design thinking into scrum framework in the context of requirements engineering management .' In Proceedings of the 3rd International Conference on Computer Science and Software Engineering (pp. 33 - 45 ).

[24] Stettina , C.J. and Smit , M.N. , 2016 . ' Team portfolio scrum: an action research on multitasking in multi-project scrum teams .' In Agile Processes, in Software Engineering, and Extreme Programming: 17th International Conference, XP 2016, Edinburgh , UK, May 24 -27, 2016 , Proceedings 17 (pp. 79 - 91 ). Springer International Publishing.

[25] Law , E.L.C. and Lárusdóttir , M.K. , 2015 . ' Whose experience do we care about? Analysis of the fitness of scrum and kanban to user experience .' International Journal of Human-Computer Interaction , 31 ( 9 ), pp. 584 - 602 .

[26] Bordin , S. and De Angeli , A. , 2016 . ' Focal points for a more user-centred agile development .' In Agile Processes, in Software Engineering, and Extreme Programming: 17th International Conference, XP 2016, Edinburgh , UK, May 24 -27, 2016 , Proceedings 17 (pp. 3 - 15 ). Springer International Publishing.

[27] Hoda , R. , Noble , J. and Marshall , S. ( 2011 ) ' The impact of inadequate customer collaboration on self-organizing Agile teams' , Information and Software Technology , 53 ( 5 ), pp. 521 - 534 . doi: 10 .1016/j.infsof. 2010 . 10 .009.

[28] Eloranta , V. P. , Koskimies , K. and Mikkonen , T. ( 2016 ) 'Exploring ScrumBut - An empirical study of Scrum anti-patterns' , Information and Software Technology , 74 , pp. 194 - 203 . doi: 10 .1016/j.infsof. 2015 . 12 .003

[29] Rafi , U. , Mustafa , T. , Iqbal , N. and Zafar , W.U.I. , 2015 . ' US-Scrum: a methodology for developing software with enhanced correctness, usability and security .' Int J Sci Eng Res , 6 ( 9 ), p. 377 .

[30] Braun , V. , & Clarke , V. ( 2012 ). Thematic analysis . In H. Cooper, P. M. Camic , D. L. Long , A. T. Panter , D. Rindskopf , & K. J. Sher (Eds.), APA handbook of research methods in psychology , Vol. 2 . Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57 - 71 ). American Psychological Association. https://doi.org/10.1037/ 13620 - 004

[31] Herdika , H.R. and Budiardjo , E.K. , 2020 , September. Variability and commonality requirement specification on agile software development: Scrum, xp, lean, and kanban . In 2020 3rd International Conference on Computer and