Software startups seek to apply strategies to provide a more meaningful User eXperience (UX) and generate a sustainable business. However, the lack of knowledge or limited resources are challenges in incorporating UX practices in the software development. This PhD project aims to develop a lean framework to empower professionals working in software startups with UX Research activities and encourage longitudinal research about UX (i.e., Long-Term UX). The methodology adopts Grounded Theory as a method for qualitative data analysis. Successive and incremental comparisons between results from literature and data collected from the field studies will result in the framework. The framework evaluation will be conducted with startups to mature the solution. We hope to present the framework in web format as an online catalog to support the activities on UX Research and Long-Term UX.
Established companies and startups have shown interest in integrating User eXperience (UX)
practices in the agile software development [
Studies often mention UX as necessary for software development in startups, but they do not
explore the integration of UX and agile practices [
Dificulties reported in terms of collecting and using user feedback are failure factors faced
by startups [
Longitudinal research is ideal for studying how and when users transition from novice to expert, understanding abandonment or adoption rates, comfort with technology, productivity, and evolution of user perceptions [14]. Understanding these issues enables us to recognize the user experience attributes that change in terms of temporality [15]. For instance, a product’s quality (reliability) is bound to increase relative importance with prolonged use [15]. However, the feeling of novelty is an aspect that quickly fades after the first interaction, while over time, the product’s value to the user can emerge [16]. Software practitioners declare that Long-Term UX studies emerged results relevant for (i) comparing the results with previous knowledge, (ii) understanding the change in UX over time, (iii) helping to decide future work, (iv) designing and developing new products, and (v) updating current products [17]. Therefore, Long-term UX can to support the decision-making and minimize the risk of product failure [17].
Considering the literature we posed the following research problem: The conduction of Long-Term UX research in a fast-paced environment of software startups provides UX data to support decisions about a software product/service. Taking into account the research problem, this PhD project aims to respond faced under two research questions: (i) how can Long-Term UX be adopted by software startups, considering their limited human resources (i.e., without UX professionals) and temporal factors (i.e., agile software development)? and (ii) which UX data collected from Long-Term UX practices can support startup practitioners in decision-making during software development?
Taking into account the discussion is possible to justify this research problem. Startup
characteristics (e.g., limited resources, small teams, and fast deliveries) [
Methods and techniques can be applied to support UX practices, making them systemic and
more likely to be successful [
Using practices that do not capture user data only in a single experience can minimize the
startup practitioners’ dificulty in handling and utilizing feedback received [ 18, 23]. Solutions
reported in the literature unite agile software development with UX practices [
The main goal is to investigate UX Research practices, techniques, and tools commonly applied in software startups. This PhD project aims to propose a framework to facilitate the use of Long-Term UX research in software startups. This proposal intends to help startup practitioners decide which UX-related data to consider in their software projects and which techniques they can apply to collect and analyze long-term data. Specific goals are: (i) identify UX Research practices that are applied by adopting a systematic process (i.e., formal practices) or not (i.e., informal practices); (ii) classify UX Research and Long-Term UX practices applied by the software industry through scientific literature; (iii) find out UX Research and Long-Term UX practices applied by software startups from field studies; (iv) systematize a framework on Long-Term UX to support startup practitioners in data collection and analysis activities; (v) evaluate the framework with startup practitioners to refine the proposal.
This project is following a qualitative methodology (see Fig. 1) which use Grounded Theory (GT) as the core method. The GT method aims to generate a theory or solution grounded in analyzing the experiences and practices lived by people in a context [28]. practices applied by the software industry. From the literature and the findings of the first ifeld study, we will cross-reference the data to identify common actions and characteristics about: Long-Term UX practices, UX methods and techniques, and users’ information (UX data). This cross-referencing of data will allow me to reach grounded categories and understand their relationships. Consequently, this knowledge will be matured with new results - obtained in the second field study - to consolidate the framework’s first version. We will return to stage (D) as long as the framework’s evaluations are carried out to generate updates in the proposal.
we will carry out the Proposal Evaluation (E) in software startups. we plan to put the framework into practice by using it in agile software development. Each software startup will use the framework between 1 and 2 months. We will collect software teams’ feedback about the framework usage, including the perceived usefulness and ease of use.
The Communication of Results (F) has been initiated and occurs in parallel to the previous stages. We are preparing a journal paper with the results of the SLR. We also published a paper in XXI Brazilian Symposium on Human Factors in Computing Systems, which discusses the findings on Long-Term UX practices in depth.
This section presents the planned activities until the end of this research and timeline (see Fig. 2). Each activity is represented through an identifier (e.g., A01, A02), and the color of each activity follows the stage’s color that it will occur (see Fig. 1). Marking with an “X” in space represents that an activity will be carried out in a month and year. We eliminated the activities about (A) and (B) stages because they finished. A01: Qualitative analysis on 1st field study, supported by SLR results (2nd GT level); A02: Writing up the partial results of the GT, with codes and emerging themes; A03: Conducting the 2nd field study with software startups and organizing the data collected; A04: Qualitative analysis on 2nd field study, supported by previous results (3rd GT level); A05: Conception of the framework resulting from the GT, with explanations of the practices, artifacts and types of data that underpin the proposal;
A06: Submit the project on the framework evaluation to the UFSCar Ethics Committee and contact startups interested in participating in the evaluation;
A07: Conducting the 1st evaluation study of the framework with startup “A” and organizing the data collected;
A08: Conducting the 2nd evaluation study of the framework with startup “B” and organizing the data collected;
A10: Framework documentation with the updated proposal (final version); A11: Communication of results (i.e., publication of papers, reports, presentations); A12: Writing of the qualification of PhD project and examination;
A13: Thesis writing and defense.
Two preliminary results were produced in this project. The first results are of the SLR, with the reduced presentation given the limited pages. In addition, we have partial results from the field study. Regarding SLR results, there are 38 UX Research practices (formal and informal) applied by the software industry and cataloged in 6 groups (see Fig. 3). The rectangle size reports the number of occurrences per practice. Thus, Practices on Research Planning and Practices on Collecting Data with Users were the groups of practices most frequently. The identification of these practices allows us to generate further studies for: i) understand which UX Research practices software startups perform and which they really need to carry out - either at the business or software development level, and ii) develop solutions that fulfill these practices and provide ways for small teams with little knowledge of UX to apply Long-Term UX research.
■ We identified 52 methods and techniques used in UX Research practices. Software practitioners commonly apply personas, metrics analysis, interviews, usability tests, and A/B tests. However, qualitative analysis, contextual inquiry, and ethnographic study are still little explored; ■ The software industry obtains 12 types of users’ information (UX data). These UX data are obtained through 20 of the 38 UX Research practices. Activities such as combine quantitative and qualitative analysis on user data and analyze metrics and quantitative data are often applied to get this variety of users’ information. Nevertheless, capturing the user frequency of use, user culture and habits, and user-perceived utility are still complex;
■ We identified that 15 of the 38 UX Research practices present a relationship with the Long-Term UX research moments. However, performing research and evaluation with users before the first use of the product ( Anticipated UX ) is a rare activity in the software industry. Most UX Research practices focus on getting feedback after usage (Episodic UX ).
■ These findings supported elaborating of recommendations that are currently being adjusted. The recommendations present a summary of practices found, the methods and techniques used, and the contexts in which they were applied. We clarify how the software industry applies such recommendations and the benefits of carrying them out.
Regarding the analysis of the interviews (first field study), we expect to identify UX Research practices that: (i) to report user needs or expectations, (ii) to obtain user feedback or evaluation, (iii) to analyze user feedback and generate insights into the product, and (iv) to communicate research results and insights. We are also looking at what Long-Term UX moments these practices are applied, besides what UX data is helpful to software startups.
Considering the interviewees’ profiles and the first insights on partial results, we can expose two hypotheses. The first is that software startups execute longitudinal research practices but do not necessarily follow the Long-Term UX cycle [13, 19, 21, 20]. The literature presents that Cumulative UX is characterized by sequential inquiry of all previous research moments (see Fig. 4-A). However, part of the interviews suggest that software startups may work with a longitudinal view, but their inquiries do not cover all the research moments (see Fig. 4-B). For example, obtaining a longitudinal view of their users would be possible considering only inquiries made in Episodic UX.
The second hypothesis is that startup practitioners with positions or roles dedicated to administration or marketing get directly involved with UX Research activities. It is possible that this practitioner also has UX Research responsibilities, besides performing activities to extract a longitudinal view of users.
This PhD project expects to contribute to state of the art on Long-Term UX. Longitudinal studies
on UX evaluation reported in literature assessed users’ perceptions by focusing on specific
times (e.g., Episodic UX) rather than assessing how their perceptions changed over time [16].
Measuring the usefulness of Long-Term UX evaluation results to work practice still receives
little attention in the literature [17]. Therefore, this PhD can contribute to the Long-Term UX
gaps. Most academic researchers concentrate on investigating UX from a theoretical perspective,
while software practitioners need tools and methods that make UX feasible and assessable [25].
This PhD research also generates knowledge on Long-Term UX practices commonly applied in
software startups. Knowing what Long-Term UX research practices are applied and how UX
data are relevant for product development can diminish the gap between theory and practice in
Long-Term UX. According to the benefits that Long-Term UX research can generate for software
startups (i.e., creating value from users, scaling up the business, and retaining users over time
[
We thank the support of grant #2020/00615-9 and grant #2020/11441-1, São Paulo Research Foundation (FAPESP), and grant 313312/2019-2, Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq - Brazil). [13] V. Roto et al., User experience white paper: Bringing clarity to the concept of user experience, Dagstuhl seminar on demarcating user experience 1 (2011). [14] C. Courage, J. Jain, S. Rosenbaum, Best Practices in Longitudinal Research, in: CHI ’09
Extended Abstracts on Human Factors in Computing Systems, ACM, 2009, p. 4791–4794. [15] A. Pohlmeyer, Identifying Attribute Importance in Early Product Development. Exemplified by Interactive Technologies and Age, Doctoral thesis, Technische Universität Berlin, 2012. [16] P. Marti, I. Iacono, Anticipated, momentary, episodic, remembered: the many facets of User eXperience, in: Federated Conference on Computer Science and Information Systems, IEEE, 2016, pp. 1647–1655. [17] J. Varsaluoma, F. Sahar, Usefulness of Long-Term User Experience Evaluation to Product Development: Practitioners’ Views from Three Case Studies, in: 8th Nordic Conference on Human-Computer Interaction, ACM, 2014, p. 79–88. [18] L. Hokkanen, K. Väänänen-Vainio-Mattila, UX Work in Startups: Current Practices and Future Needs, in: Agile Processes in Software Engineering and Extreme Programming, Springer, 2015, pp. 81–92. [19] S. Kujala et al., UX Curve: A method for evaluating long-term user experience, Interacting with computers 23 (2011) 473–483. [20] D. Biduski et al., Assessing long-term user experience on a mobile health application through an in-app embedded conversation-based questionnaire, Computers in Human Behavior 104 (2020) 106169. [21] S. Kujala et al., Lost in Time: The Meaning of Temporal Aspects in User Experience, in:
Human Factors in Computing Systems (CHI EA ’13), ACM, 2013, p. 559–564. [22] M. von Wilamowitz-Moellendorf, M. Hassenzahl, A. Platz, Dynamics of user experience: How the perceived quality of mobile phones changes over time, in: Workshop at the 4th Nordic Conference on Human-Computer Interaction, 2006, pp. 74–78. [23] L. Hokkanen, K. Kuusinen, K. Väänänen, Minimum viable user experience: A framework for supporting product design in startups, in: Agile Processes in Software Engineering and Extreme Programming, Springer, 2016, pp. 66–78. [24] K. Kuusinen, BoB: a framework for organizing within-iteration UX work in agile development, in: Integrating User-Centred Design in Agile Development, Springer, 2016. [25] F. Lachner et al., Quantified UX: Towards a Common Organizational Understanding of
User Experience, in: 9th Nordic Conference on Human-Computer Interaction, ACM, 2016. [26] J. Gothelf, Lean UX: Applying lean principles to improve user experience, "O’Reilly Media,
Inc.", 2013. [27] R. Hartson, P. S. Pyla, The UX book: Agile UX design for a quality user experience, Morgan
Kaufmann, 2018. [28] J. W. Creswell, C. N. Poth, Qualitative Inquiry and Research Design: Choosing Among
Five Approaches, 4 ed., SAGE Publications, Inc, 2017. [29] B. Kitchenham, S. Charters, Guidelines for performing Systematic Literature Reviews in
Software Engineering, Technical Report, Keele University and Durham University, 2007. [30] T. Dybå, T. Dingsøyr, Empirical studies of agile software development: A systematic review, Information and Software Technology 50 (2008) 833–859. [31] S. Genome, Global Startup Ecosystem Report, 2019. Startup Genome.