Long-Term UX framework: supporting software startups in UX Research Suéllen Martinelli1,* , Luciana Zaina1,† 1 Federal University of São Carlos (UFSCar), Brazil Abstract Software startups seek to apply strategies to provide a more meaningful User eXperience (UX) and generate a sustainable business. However, the lack of knowledge or limited resources are challenges in incorporating UX practices in the software development. This PhD project aims to develop a lean framework to empower professionals working in software startups with UX Research activities and encourage longitudinal research about UX (i.e., Long-Term UX). The methodology adopts Grounded Theory as a method for qualitative data analysis. Successive and incremental comparisons between results from literature and data collected from the field studies will result in the framework. The framework evaluation will be conducted with startups to mature the solution. We hope to present the framework in web format as an online catalog to support the activities on UX Research and Long-Term UX. Keywords Software Engineering, Agile Software Development, User eXperience, UX Research, Long-Term UX, Grounded Theory, Software Startup, Software Industry 1. Research Problem Established companies and startups have shown interest in integrating User eXperience (UX) practices in the agile software development [1]. Startups operate with small software teams, use new technologies with little knowledge, work with high uncertainty about customers and market conditions, and have a high failure rate [2, 3]. Software startups are newly created companies that produce software products [3] or make intense use of software to manage their activities [4]. Startups differ from established companies by their aim to quickly scale up, changing the business model to grow in the market [5, 3]. UX knowledge and practice might bring benefits to scale up the business of software startups [6]. For instance, a good UX can maximize the product’s value to the customer [7, 8] and create competitive advantages for the business (e.g., increasing the number of users, identifying new market segments) [6, 5]. Working UX from the beginning of a software project can increase the chances of success of the products developed by software startups [7, 5]. Studies often mention UX as necessary for software development in startups, but they do not explore the integration of UX and agile practices [8, 3]. Besides, the lack of resources is one of the main reasons software startups do not spend their resources with UX [5]. Software startups ICSOB 2022: 13th International Conference on Software Business (PhD Retreat), November 08–11, 2022, Bolzano, Italy * Corresponding author, and PhD student in Computer Science. † She supervise this research. $ suellen.martinelli@estudante.ufscar.br (S. Martinelli); lzaina@ufscar.br (L. Zaina) € http://lattes.cnpq.br/9978120143458461 (S. Martinelli); http://uxleris.net/lzaina/ (L. Zaina)  0000-0002-4421-2940 (S. Martinelli); 0000-0002-1736-544X (L. Zaina) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) need to be fast to develop and deliver a Minimum Viable Product (MVP) to the market; therefore, they end up neglecting the application of UX practices [5]. Hassenzahl (2018) advocates content- oriented UX as a possibility to operationalize UX practices during product development. Content- oriented UX model contains three levels (Why, What, and How) and a goal, the wellbeing of users. This model assumes that the “Why” must be determined first, setting needs and envisioned experience. Subsequently, to become able to choose functionality (“What”) and to determine the appropriate form and interaction (“How”) in line with the experience. Finally, it can provide enjoyable and meaningful everyday experiences (“Wellbeing”) [9]. However, models are reductions and require inquiries into UX practices and experience [9]. Difficulties reported in terms of collecting and using user feedback are failure factors faced by startups [10, 5]. The main difficulty is analyzing and translating user data into meaningful information for product development [5]. These difficulties can be addressed by UX Research, an area that executes systemic research and evaluations into the user experience [11]. UX Research aims to collect, analyze and interpret user data to provide insights for product development [11, 12]. Thus, UX Research practices are attitudes, actions, or activities about research and evaluation that practitioners need to perform to understand the UX [1]. Although UX Research can be seen as a relevant area, the Long-Term UX investigations with users are useful for exploring the UX factors (i.e., context around the user, the user’s state, and product properties) in different time spans of UX [13]. Long-term UX classifies the UX evaluation in time spans for: getting a user’s expectations before the first use (Anticipated UX ); understanding perceived changes during the user’s interaction with the product (Momentary UX ); evaluating an episode of use after an interaction event (Episodic UX ); and to gather the results of previous research and user recollections after having used the product for broader time (Cumulative UX ) [13]. Longitudinal research is ideal for studying how and when users transition from novice to expert, understanding abandonment or adoption rates, comfort with technology, productivity, and evolution of user perceptions [14]. Understanding these issues enables us to recognize the user experience attributes that change in terms of temporality [15]. For instance, a product’s quality (reliability) is bound to increase relative importance with prolonged use [15]. However, the feeling of novelty is an aspect that quickly fades after the first interaction, while over time, the product’s value to the user can emerge [16]. Software practitioners declare that Long-Term UX studies emerged results relevant for (i) comparing the results with previous knowledge, (ii) understanding the change in UX over time, (iii) helping to decide future work, (iv) designing and developing new products, and (v) updating current products [17]. Therefore, Long-term UX can to support the decision-making and minimize the risk of product failure [17]. Considering the literature we posed the following research problem: The conduction of Long-Term UX research in a fast-paced environment of software startups provides UX data to support decisions about a software product/service. Taking into account the research problem, this PhD project aims to respond faced under two research questions: (i) how can Long-Term UX be adopted by software startups, considering their limited human resources (i.e., without UX professionals) and temporal factors (i.e., agile software development)? and (ii) which UX data collected from Long-Term UX practices can support startup practitioners in decision-making during software development? Taking into account the discussion is possible to justify this research problem. Startup characteristics (e.g., limited resources, small teams, and fast deliveries) [2] can insert obstacles to the practitioners conducting Long-Term UX. Furthermore, software startups cannot afford to hire UX specialists or professionals dedicated only to UX Research activities [18]. We identified in the literature that the software industry applies internal workshops and study groups about UX Research, working skills research in developers and designers who do not know about UX [1]. This alternative, especially in startups, encourages developers and other practitioners to apply UX Research practices. The need of having Long-Term UX methods and techniques more easier to be applied is another topic [19]. Longitudinal studies should present Long-Term UX techniques that are less costly to facilitate UX Research practices in agile software development [1, 20]. The software industry uses fewer techniques specific to Long-Term UX (i.e., iScale and AttrakDiff). There is an effort to develop longitudinal studies, even with common collection techniques (such as interviews, surveys online, observations, and focus groups) [17]. Finally, Long-Term UX research generates results (UX data) about the cause and effect relationships of users’ actions in each time span [13, 21, 17]. But, the Academy still has little explored how this UX data impacts the business or decision-make during software development. 2. Knowledge Gap Methods and techniques can be applied to support UX practices, making them systemic and more likely to be successful [1]. Particularly, Long-Term UX practices can be supported by several methods and techniques to continuous getting user feedback. These methods and techniques can be specific to Long-Term UX or not. Techniques such as interviews, surveys, and focus groups are welcome, but others like CORPUS, iScale, and UX Curve are dedicated to extracting a longitudinal view [20, 19, 17]. Nevertheless, these methods and techniques of Long-Term UX may require a substantial amount of time and resources [17]. Retrospective techniques are less costly and may be suitable for software startups, while repeated longitudinal surveys present more reliable results but generate higher expenses [16]. Some studies address the Long-Term UX work in the software industry [22, 16, 17], but we did not find studies that specifically attend to the software startups’ environment. Using practices that do not capture user data only in a single experience can minimize the startup practitioners’ difficulty in handling and utilizing feedback received [18, 23]. Solutions reported in the literature unite agile software development with UX practices [23, 10, 24]. However, they do not give autonomy for startup practitioners to select UX Research practices - besides tools and techniques - that best fit their needs during product development. There are also proposals focused on Long-Term UX [19, 21, 25]. The software industry can also use consolidated approaches, such as Lean UX [26] and Agile UX [27], that focus on incorporating UX activities in agile environments. However, we do not find in the literature specific frameworks for conducting Long-Term UX research that considers the software startups’ characteristics. On the other hand, the literature suggests that solutions about Long-Term UX can increase software startups’ competitiveness and to retain users over time [17, 10, 18]. 3. Research Goals The main goal is to investigate UX Research practices, techniques, and tools commonly applied in software startups. This PhD project aims to propose a framework to facilitate the use of Long-Term UX research in software startups. This proposal intends to help startup practitioners decide which UX-related data to consider in their software projects and which techniques they can apply to collect and analyze long-term data. Specific goals are: (i) identify UX Research practices that are applied by adopting a systematic process (i.e., formal practices) or not (i.e., informal practices); (ii) classify UX Research and Long-Term UX practices applied by the software industry through scientific literature; (iii) find out UX Research and Long-Term UX practices applied by software startups from field studies; (iv) systematize a framework on Long-Term UX to support startup practitioners in data collection and analysis activities; (v) evaluate the framework with startup practitioners to refine the proposal. 4. Methodology This project is following a qualitative methodology (see Fig. 1) which use Grounded Theory (GT) as the core method. The GT method aims to generate a theory or solution grounded in analyzing the experiences and practices lived by people in a context [28]. Figure 1: Research Methodology The investigation of the State of the Art Research (A) was completed. The themes investi- gated during the (A) stage were discussed in Section 1. The (A) stage receives updates as this research progresses to the later stages. The second stage, the Systematic Review Literature SLR (B), also was completed. The SLR finished with 45 papers selected among 634 papers obtained in five search engines. We inves- tigated the UX Research practices applied by the software industry, just as the methods and techniques used in these UX Research practices. The SLR also investigated which moments of Long-Term UX have applied the UX Research practices (more details in Section 6). The SLR followed the guidelines by Kitchenham and Charters [29]. We calculated the Kappa coefficient and applied a quality asseeement [30]. The SLR used open and closed coding to analyze text fragments (snippets) from selected papers. This analysis is equivalent to the 1st GT level. The Field Studies (C) is ongoing. we interviewed 11 practitioners (e.g., CEOs, developers, UX designers, and UX researchers) from 5 software startups. we are analyzing the responses about the UX Research practices applied by startup practitioners and how these practices can be connected to Long-Term UX (more details in Section 6). The SLR’s results are used in the qualitative analysis of the interviews. The SLR’s results operate as codes during the closed coding. Other codes used in the closed coding are the characteristics of software startups [2] and the Long-Term UX research moments [13]. This strategy provides a theoretical-practical background to develop the qualitative analysis, respective to the 2nd GT level. we will conduct a further field study with other software startups, which includes getting feedback through interviews and non-participant observations. In the next field study, it is expected to collect data from startups in the same life cycle stage (e.g., in the growth or scale-up stage) [31]. The Framework Development (D) follows the coding rounds present in the GT. The (D) stage started with the SLR’s results and the emerging recommendations about the UX Research practices applied by the software industry. From the literature and the findings of the first field study, we will cross-reference the data to identify common actions and characteristics about: Long-Term UX practices, UX methods and techniques, and users’ information (UX data). This cross-referencing of data will allow me to reach grounded categories and understand their relationships. Consequently, this knowledge will be matured with new results - obtained in the second field study - to consolidate the framework’s first version. We will return to stage (D) as long as the framework’s evaluations are carried out to generate updates in the proposal. we will carry out the Proposal Evaluation (E) in software startups. we plan to put the framework into practice by using it in agile software development. Each software startup will use the framework between 1 and 2 months. We will collect software teams’ feedback about the framework usage, including the perceived usefulness and ease of use. The Communication of Results (F) has been initiated and occurs in parallel to the previous stages. We are preparing a journal paper with the results of the SLR. We also published a paper in XXI Brazilian Symposium on Human Factors in Computing Systems, which discusses the findings on Long-Term UX practices in depth. 5. Timeline This section presents the planned activities until the end of this research and timeline (see Fig. 2). Each activity is represented through an identifier (e.g., A01, A02), and the color of each activity follows the stage’s color that it will occur (see Fig. 1). Marking with an “X” in space represents that an activity will be carried out in a month and year. We eliminated the activities about (A) and (B) stages because they finished. Figure 2: Timeline with next stages A01: Qualitative analysis on 1st field study, supported by SLR results (2nd GT level); A02: Writing up the partial results of the GT, with codes and emerging themes; A03: Conducting the 2nd field study with software startups and organizing the data collected; A04: Qualitative analysis on 2nd field study, supported by previous results (3rd GT level); A05: Conception of the framework resulting from the GT, with explanations of the practices, artifacts and types of data that underpin the proposal; A06: Submit the project on the framework evaluation to the UFSCar Ethics Committee and contact startups interested in participating in the evaluation; A07: Conducting the 1st evaluation study of the framework with startup “A” and organizing the data collected; A08: Conducting the 2nd evaluation study of the framework with startup “B” and organizing the data collected; A09: Refinement of the framework with the results of the last two evaluations; A10: Framework documentation with the updated proposal (final version); A11: Communication of results (i.e., publication of papers, reports, presentations); A12: Writing of the qualification of PhD project and examination; A13: Thesis writing and defense. 6. Preliminary Results Two preliminary results were produced in this project. The first results are of the SLR, with the reduced presentation given the limited pages. In addition, we have partial results from the field study. Regarding SLR results, there are 38 UX Research practices (formal and informal) applied by the software industry and cataloged in 6 groups (see Fig. 3). The rectangle size reports the number of occurrences per practice. Thus, Practices on Research Planning and Practices on Collecting Data with Users were the groups of practices most frequently. The identification of these practices allows us to generate further studies for: i) understand which UX Research practices software startups perform and which they really need to carry out - either at the business or software development level, and ii) develop solutions that fulfill these practices and provide ways for small teams with little knowledge of UX to apply Long-Term UX research. Figure 3: Treemap with occurrences by UX Research practices and groups Other results identified in the SLR were: ■ We identified 52 methods and techniques used in UX Research practices. Software practi- tioners commonly apply personas, metrics analysis, interviews, usability tests, and A/B tests. However, qualitative analysis, contextual inquiry, and ethnographic study are still little explored; ■ The software industry obtains 12 types of users’ information (UX data). These UX data are obtained through 20 of the 38 UX Research practices. Activities such as combine quantitative and qualitative analysis on user data and analyze metrics and quantitative data are often applied to get this variety of users’ information. Nevertheless, capturing the user frequency of use, user culture and habits, and user-perceived utility are still complex; ■ We identified that 15 of the 38 UX Research practices present a relationship with the Long-Term UX research moments. However, performing research and evaluation with users before the first use of the product (Anticipated UX ) is a rare activity in the software industry. Most UX Research practices focus on getting feedback after usage (Episodic UX ). ■ These findings supported elaborating of recommendations that are currently being adjusted. The recommendations present a summary of practices found, the methods and techniques used, and the contexts in which they were applied. We clarify how the software industry applies such recommendations and the benefits of carrying them out. Regarding the analysis of the interviews (first field study), we expect to identify UX Research practices that: (i) to report user needs or expectations, (ii) to obtain user feedback or evaluation, (iii) to analyze user feedback and generate insights into the product, and (iv) to communicate research results and insights. We are also looking at what Long-Term UX moments these practices are applied, besides what UX data is helpful to software startups. Considering the interviewees’ profiles and the first insights on partial results, we can expose two hypotheses. The first is that software startups execute longitudinal research practices but do not necessarily follow the Long-Term UX cycle [13, 19, 21, 20]. The literature presents that Cumulative UX is characterized by sequential inquiry of all previous research moments (see Fig. 4-A). However, part of the interviews suggest that software startups may work with a longitudinal view, but their inquiries do not cover all the research moments (see Fig. 4-B). For example, obtaining a longitudinal view of their users would be possible considering only inquiries made in Episodic UX. Figure 4: (A) Long-Term UX cycle in the literature [13] and (B) possible Long-Term UX cycle in the software startups The second hypothesis is that startup practitioners with positions or roles dedicated to administration or marketing get directly involved with UX Research activities. It is possible that this practitioner also has UX Research responsibilities, besides performing activities to extract a longitudinal view of users. 7. Expected Contributions This PhD project expects to contribute to state of the art on Long-Term UX. Longitudinal studies on UX evaluation reported in literature assessed users’ perceptions by focusing on specific times (e.g., Episodic UX) rather than assessing how their perceptions changed over time [16]. Measuring the usefulness of Long-Term UX evaluation results to work practice still receives little attention in the literature [17]. Therefore, this PhD can contribute to the Long-Term UX gaps. Most academic researchers concentrate on investigating UX from a theoretical perspective, while software practitioners need tools and methods that make UX feasible and assessable [25]. This PhD research also generates knowledge on Long-Term UX practices commonly applied in software startups. Knowing what Long-Term UX research practices are applied and how UX data are relevant for product development can diminish the gap between theory and practice in Long-Term UX. According to the benefits that Long-Term UX research can generate for software startups (i.e., creating value from users, scaling up the business, and retaining users over time [6, 8, 10]), our main contribution is the proposed framework. The framework will guide startup practitioners in choosing Long-Term UX practices and techniques that attend the software teams’ needs during product development. The framework’s final version will be available in an interactive web format for startup practitioners to query Long-Term UX practices and techniques, just as to query which UX data should be obtained. Acknowledgments We thank the support of grant #2020/00615-9 and grant #2020/11441-1, São Paulo Research Foun- dation (FAPESP), and grant 313312/2019-2, Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq - Brazil). References [1] P. Kashfi, R. Feldt, A. Nilsson, Integrating UX principles and practices into software development organizations: A case study of influencing events, Journal of Systems and Software 154 (2019) 37–58. [2] N. Paternoster et al., Software development in startup companies: A systematic mapping study, Information and Software Technology 56 (2014) 1200–1218. [3] J. Choma et al., Influences of UX factors in the Agile UX context of software startups, Information and Software Technology 152 (2022) 107041. [4] C. Giardino et al., What Do We Know about Software Development in Startups?, IEEE Software 31 (2014) 28–32. [5] J. Saad et al., UX work in software startups: A thematic analysis of the literature, Informa- tion and Software Technology 140 (2021) 106688. [6] L. Hokkanen, Y. Xu, K. Väänänen, Focusing on User Experience and Business Models in Startups: Investigation of Two-Dimensional Value Creation, in: 20th International Academic Mindtrek Conference, ACM, 2016, p. 59–67. [7] E. Klotins, M. Unterkalmsteiner, T. Gorschek, Software engineering in start-up companies: An analysis of 88 experience reports (2019) 68–102. [8] K. Kuusinen et al., From Startup to Scaleup: An Interview Study of the Development of User Experience Work in a Data-Intensive Company, in: Human-Centered Software Engineering, Springer, 2019, pp. 3–14. [9] M. Hassenzahl, The Thing and I (Summer of ’17 Remix), in: Funology 2: From Usability to Enjoyment, Springer, 2018, pp. 17–31. [10] L. Hokkanen, M. Leppänen, Three patterns for user involvement in startups, in: 20th European Conference on Pattern Languages of Programs, ACM, 2015, pp. 1–8. [11] S. Farrell, UX Research Cheat Sheet, 2017. Nielsen Norman Group (NNGroup). [12] K. Pazitka, The UX Research Methods Every Designer Needs To Know, 2019. Career- Foundry. [13] V. Roto et al., User experience white paper: Bringing clarity to the concept of user experience, Dagstuhl seminar on demarcating user experience 1 (2011). [14] C. Courage, J. Jain, S. Rosenbaum, Best Practices in Longitudinal Research, in: CHI ’09 Extended Abstracts on Human Factors in Computing Systems, ACM, 2009, p. 4791–4794. [15] A. Pohlmeyer, Identifying Attribute Importance in Early Product Development. Exemplified by Interactive Technologies and Age, Doctoral thesis, Technische Universität Berlin, 2012. [16] P. Marti, I. Iacono, Anticipated, momentary, episodic, remembered: the many facets of User eXperience, in: Federated Conference on Computer Science and Information Systems, IEEE, 2016, pp. 1647–1655. [17] J. Varsaluoma, F. Sahar, Usefulness of Long-Term User Experience Evaluation to Product Development: Practitioners’ Views from Three Case Studies, in: 8th Nordic Conference on Human-Computer Interaction, ACM, 2014, p. 79–88. [18] L. Hokkanen, K. Väänänen-Vainio-Mattila, UX Work in Startups: Current Practices and Future Needs, in: Agile Processes in Software Engineering and Extreme Programming, Springer, 2015, pp. 81–92. [19] S. Kujala et al., UX Curve: A method for evaluating long-term user experience, Interacting with computers 23 (2011) 473–483. [20] D. Biduski et al., Assessing long-term user experience on a mobile health application through an in-app embedded conversation-based questionnaire, Computers in Human Behavior 104 (2020) 106169. [21] S. Kujala et al., Lost in Time: The Meaning of Temporal Aspects in User Experience, in: Human Factors in Computing Systems (CHI EA ’13), ACM, 2013, p. 559–564. [22] M. von Wilamowitz-Moellendorff, M. Hassenzahl, A. Platz, Dynamics of user experience: How the perceived quality of mobile phones changes over time, in: Workshop at the 4th Nordic Conference on Human-Computer Interaction, 2006, pp. 74–78. [23] L. Hokkanen, K. Kuusinen, K. Väänänen, Minimum viable user experience: A framework for supporting product design in startups, in: Agile Processes in Software Engineering and Extreme Programming, Springer, 2016, pp. 66–78. [24] K. Kuusinen, BoB: a framework for organizing within-iteration UX work in agile develop- ment, in: Integrating User-Centred Design in Agile Development, Springer, 2016. [25] F. Lachner et al., Quantified UX: Towards a Common Organizational Understanding of User Experience, in: 9th Nordic Conference on Human-Computer Interaction, ACM, 2016. [26] J. Gothelf, Lean UX: Applying lean principles to improve user experience, "O’Reilly Media, Inc.", 2013. [27] R. Hartson, P. S. Pyla, The UX book: Agile UX design for a quality user experience, Morgan Kaufmann, 2018. [28] J. W. Creswell, C. N. Poth, Qualitative Inquiry and Research Design: Choosing Among Five Approaches, 4 ed., SAGE Publications, Inc, 2017. [29] B. Kitchenham, S. Charters, Guidelines for performing Systematic Literature Reviews in Software Engineering, Technical Report, Keele University and Durham University, 2007. [30] T. Dybå, T. Dingsøyr, Empirical studies of agile software development: A systematic review, Information and Software Technology 50 (2008) 833–859. [31] S. Genome, Global Startup Ecosystem Report, 2019. Startup Genome.