In a scenario of investments in software development strategies and approaches to meet new market demands, it has been a great challenge for corporations to maintain a system/software architecture fully internalized to its organization [1]. For this reason, some companies have invested in opening their architectures to allow third-party developers to collaborate in producing their components over a common technological platform. That practice defines the notion of software ecosystem (SECO) [2].
In this environment, interfaces that support the Web (e.g., Web portals, forums, social networks etc.) play an important role within SECO, as they facilitate access to information and interaction with developers on the platform. These interfaces form the so-called SECO portals [3]. As examples, it is possible to mention mobile SECO portals Android 1 and iOS 2 that make available to their developers' community a set of tools, documents, and support channels to assist them in building their usage tools.
SECO portals are fundamental elements for a keystone, i.e., an organization that manages a common technological platform, to maintain its sustainable ecosystem in the market over time. These web interfaces are one of the main mechanisms to attract and engage third-party developers in SECO. They use SECO portals to be aware of the processes and elements (e.g., documentation files, source code, forums etc.) that are part of the SECO platform. Therefore, they can learn how to contribute to SECO [4].
One of the factors that engage third-party developers in SECO is directly related to their experience on its portals. It will result in a type of user experience (UX), giving rise to the concept of developer
The 15th International Conference on Software Business (ICSOB 2024), November 18-20, 2024, Utrecht, the Netherlands * Corresponding author.
rodrigo.zacarias@edu.unirio.br (R. O. Zacarias); rps@uniriotec.br (R. P. d. Santos); p.lago@vu.nl (P. Lago) 0000-0003-0005-4669 (R. O. Zacarias); 0000-0003-4749-2551 (R. P. d. Santos); 0000-0002-2234-0845 (P. Lago) experience (DX) [5]. Fagerholm and Münch [6] define DX as a broader concept that captures how developers feel about, think about, and value their work. In other words, DX refers to a term that explains how developers engage with the software development process, both from a technical and a social perspective [5]. In the context of SECO, DX is a coefficient both for the performance of developers and for keeping developers actively contributing to a SECO platform [7]. However, software developers have reported difficulties in accessing, using, and understanding the information provided by keystones on their SECO portals [4]. This situation is related to the lack of transparency of SECO portals. In this context, transparency can be defined as the condition that allows the individual access, ease of use, quality of content, understanding, and auditing of information of interest [8,9]. From a software perspective, transparency also encompasses openness, clarity, and visibility of the mechanisms, processes, and actions of software applications [10]. This problem contributes to increased barriers to entry for SECO and a cumbersome start for newcomer third-party developers [2,11]. It affects DX in SECO, which can lead to a lack of interest and engagement among third-party developers to the platform. In more severe cases, this can culminate in the SECO "death", due to the difficulty of retaining developers who contribute frequently and attracting new ones [12].
In this context, this research aims to investigate the relationship between DX and transparency to develop a framework integrated into a set of guidelines for improving transparency in SECO portals from a DX perspective. The framework for Transparency in SECO Portals (T-SECOP) will consist of: (i) a checklist for assessing transparency, (ii) a computational tool for monitoring DX, and (iii) a mechanism for collecting feedback from the developers community about DX. The basis for these artifacts will be guidelines for improving transparency, which will be elaborated on throughout this research. Under the aegis of the Design Science Research (DSR) method [13], our research method is comprised of three phases: characterization, development, and evaluation. As a contribution, we intend to offer a framework with a set of artifacts to support researchers, from universities and research institutes, and designers and managers from organizations within SECO in diagnosing problems related to transparency in their portals and obtaining suggestions for actions to minimize these problems.
The remainder of this paper is organized as follows: Section 2 discusses related work and the knowledge gap; Section 3 describes the research method and the evaluation plan of the results; Section 4 depicts the partial results achieved; Section 5 presents the planned research timeline; and, finally, Section 6 presents expected contributions of the thesis.
According to Ofem et al. [14], research on software transparency has progressed slowly. There are few studies focused on software transparency, especially in software engineering. Isong et al. [10] argue that transparency is a complex concept and is rarely considered from a theoretical point of view. It is necessary to explain transparency in terms of factors that help its achievement. Obie et al. [15] highlight that exploring the stakeholders' perspective can contribute to generating guidelines to operationalize transparency more easily. In our systematic mapping study (SMS) on transparency in SECO, we have not identified solutions that approached this knowledge gap [16]. Our framework integrated with guidelines for improving transparency aims to manage these issues.
As related work, Cataldo and Herbsleb [17] introduce the concept of interface translucence as an architectural mechanism that seeks to overcome challenges faced by transparency, a critical principle in SECO, and has the potential to lead to information overload. They present an application of this concept in the context of architecting and implementing a software system. As challenges, they point out that new measures of transparency should be empirically evaluated to understand their impact in the technical and socio-organizational contexts. Meireles et al. [3] propose an instrument for assessing transparency in SECO portals, called ITRANSPE. It suggests some corrective measures that can be implemented to increase the transparency level of the SECO portal. As research challenges, they point to the need to assess transparency in portals from other perspectives, for example, the user's perspective; and evaluate cases of SECO portals that are sensitive to the organization's business, for example, when the degree of transparency is not required in its entirety.
Lnenicka and Anastasija [18] explore the role of open data portals in supporting these efforts and provide findings regarding the features in the design of these data infrastructures. They propose a list of the categories and corresponding features of open data portals that should constitute the checklist of the portal aiming to achieve the highest level of transparency. As research challenges, they point out that transparency depends not only on how visible the information is, but also on how understandable it is to users. Therefore, it is necessary to investigate a form of design that facilitates the understanding of the information available on portals. Tuomi [19] investigates various existing usability evaluation methods (UEM) in an exploratory study and uses the findings to develop a framework for examining and adapting UEM for DX evaluation. The result of this study is a multi-stage framework for UEM analysis and adaptation, consisting of diagrams, tables, and textual descriptions. The framework development draws from design science principles. This thesis was limited in scope to a specific research approach that did not include empirical studies or actual user research, but instead opted for simulation. Therefore, empirical approaches are necessary to evaluate and enhance the findings and validity of the framework.
Comparing these works with the T-SECOP proposal, we can highlight artifacts of our framework intended to address some of the challenges pointed out in the works. As we are using the DSR methodology to develop our framework, our research process will address theoretical rigor and empirical evaluation, a need highlighted in works related to research on transparency. We intend to extend the principles addressed by Meireles et al. [3], bringing an approach that promotes transparency in the environment considering the user's perspective, in this case, the software developer.
As mentioned by Lnenicka and Anastasija [18], this proposal also starts from the view that transparency is related to the design of portals. Therefore, the guidelines that will be developed will bring suggestions on how to improve the portal interface elements to facilitate access, ease of use, quality of content, understanding, and auditing of information provided. Finally, with the T-SECOP development proposal, we hope to advance the understanding of how to improve DX, especially in SECO. Tuomi [19] claims that this is an emerging research topic and that approaches to systematizing its evaluation are still lacking. They direct to use of the existing strategies of the UX area for this context.
According to Hevner and Chatterjee [13], in DSR, a practical problem is responsible for guiding the research, and other practical problems and questions about knowledge will arise. There is no detailed process for performing DSR. However, a key insight can be gained by identifying and understanding the existence of three DSR cycles: Relevance, Rigor, and Design. The Relevance Cycle bridges the contextual environment of the research project with the design science activities. The Rigor Cycle connects the design science activities with the knowledge base of scientific foundations, experience, and expertise that informs the research project. The central Design Cycle iterates between the core activities of building and evaluating the design artifacts and processes of the research [20]. The DSR cycles are often iterative, and the activities and the studies performed intertwine.
To guide this doctoral research, we elaborated three research questions (RQ): (RQ1) "How does transparency in SECO portals affect DX?"; (RQ2) "How to check the appropriate level of transparency of a SECO portal from a DX perspective?"; and (RQ3) "What guidelines are needed to improve transparency in SECO portals?". Figure 1 illustrates the steps, activities, and their respective products in the research proposal.
In the Characterization Phase, we performed the Rigor Cycle. We aimed to investigate the scientific literature and the real context to form a knowledge base of conditioning factors for transparency in SECO, DX factors in SECO, evaluation approaches of DX, impacts of transparency on SECO sustainability, and ways of consuming information by developers. Currently, this phase is already completed. Conduct an experiment with developers in SECO portals. This study aimed to understand developers' behavior with SECO portals. To do so, we used the tracking tool T2-UXT (now UX-Tracking3 ) to monitor developers while interacting with SECO portals.
Conduct an SMS and field study on transparency in SECO. This study aimed to characterize transparency in SECO and identify possible issues, challenges, practices, and technologies in this research topic. We conducted an SMS and a field study to identify and analyze such factors. After investigating the literature, we interviewed software developers to characterize the importance of conditioning factors for transparency identified in their interaction with GitHub.
Conduct a survey and field study on consuming information about software development in SECO. This study aimed to investigate the factors associated with the information consumption forms that influence software developers' decisions about where to seek knowledge about a SECO technological platform. We conducted a survey and a field study with software developers to understand their information consumption in SECO portals.
Conduct an exploratory study on transparency as a sustainability goal in SECO. This study aimed to understand sustainability-driven decisions considering the transparency of SECO information and processes. To do so, we proposed an approach using decision maps to frame the impacts of the conditioning factors for transparency on SECO sustainability.
Conduct an SMS on DX factors in software platforms. This study aimed to investigate and characterize the factors that affect DX in software platforms. To do so, we reviewed the scientific literature to map and analyze the state-of-the-art of DX in development on software platforms.
Conduct an SMS and a Delphi study on DX in SECO. This study aimed to investigate DX factors and characterize their influence on software developers to adopt and keep contributing to a SECO. We conducted an SMS and a Delphi study with software developers to evaluate how DX factors influenced them to adopt and keep contributing to a SECO through a questionnaire.
This is the current stage of our research. In the Development Phase, we are performing the Design Cycle. We are conducting the following activities to develop the artifacts of T-SECOP. Develop a conceptual model on transparency and DX in SECO. We intend to develop a conceptual model to connect all the knowledge obtained in the characterization phase. The conceptual model will be the basis for the development of the T-SECOP framework artifacts.
Develop the guidelines for improving transparency in SECO portals. The guidelines will be developed to consider the experience reported by the developers in the previous studies, so that the desirable characteristics of transparency for the portals have a positive influence on DX. To develop guidelines, we will adapt the Methodology to Develop Usability/UX Heuristics by Quiñones et al. [21]. Since DX is directly related to UX and we want the basis of transparency guidelines to come from DX, we chose this strategy because it includes a set of systematized activities and a template for UX heuristics specification.
Create a checklist for assessing transparency in SECO portals. The checklist can be used by designers or researchers to systematically inspect the interfaces of the portals and identify possible problems of transparency, allowing the implementation of improvements in their design.
Based on the proposed guidelines, this tool will try to automate or semi-automate some types of portal transparency analysis from the DX perspective. Using this tool, researchers and practitioners will be able to obtain indicators based on eye tracking, mouse tracking, typed data, and browsing history, and use them to identify possible transparency problems based on developers interacting with the SECO portal.
Develop a mechanism for collecting feedback from developers on DX. The initial idea is to create a plugin that can be integrated into the portal that will be activated when the developer leaves the portal. The developer will be invited to answer a short questionnaire involving questions about DX and transparency. The elaboration of these questions will be guided by the transparency guidelines. Hereby, keystone managers will be able to get a report on developer feedback on their experience. Thus, they will be able to identify new demands or requirements more easily and will be able to implement changes that directly impact DX in the portal.
Develop T-SECOP framework. After developing the three artifacts, the T-SECOP framework will be developed. T-SECOP's initial design is based on the three dimensions of SECO: business, technical, and social. The checklist for assessing transparency will address the business dimension; the computational tool will address the technical dimension; and the feedback collection mechanism will address the social dimension. All artifacts will be centered on the transparency guidelines.
In the Evaluation Phase, we intend to conduct specific activities to verify and validate the artifacts of T-SECOP, according to Design and Relevance Cycles in DSR.
Conduct a focus group with experts to evaluate T-SECOP. The focus group will consist of experts meeting from areas of SECO, Human-Computer Interaction (HCI), and Software Engineering for a detailed framework presentation. This study aims to evaluate the architecture and functioning of the T-SECOP framework artifacts, besides the applicability of the guidelines for improving transparency in SECO portals. Initially, a 90-minute session will be held. Feedback collected during the study will serve for the framework refinement.
Refine T-SECOP. Based on the feedback from the focus group, we will make adjustments to improve and refine the framework artifacts.
Conduct a case study with professionals from a real SECO. After refinement, we will conduct a case study with real SECO professionals. Initially, we intended to carry out this study at a research institute that develops solutions within a SECO. Currently, they need information (reports) about DX when using SECO partner platform tools to identify improvement possibilities that can be reported to the SECO keystone. In addition, the professionals from the research institute can use the T-SECOP framework to design the portal interfaces of their products. After a few weeks of using the framework, they will be interviewed by the researcher to verify the perception of ease of use and usefulness of T-SECOP artifacts. Feedback collected during the study will help in the framework evolution.
In this section, we present the results obtained in the studies of the Characterization Phase of our research method.
Study 1: T2-UXT: A Tool to Support Transparency Evaluation in Software Ecosystems Portals. This paper presented a tool to allow the evaluation of characteristics that contribute to a SECO portal transparency, called T2-UXT (now, UX-Tracking). To do so, this tool produced visual artifacts and indicators from the user's experience. The tool was evaluated with five volunteer developers, in the context of the SECO Android and iOS portals, based on the execution of five tasks. The results showed that T2-UXT generated indicators that support an investigation into the origin of inconsistencies in the transparency of SECO portals. This exploratory study provided the first insights into the relation transparency-DX and was published at the 34th Brazilian Symposium on Software Engineering (SBES'20) [22].
Study 2: Investigating Conditioning Factors for Transparency in Software Ecosystems. This study aimed to characterize transparency in SECO and identify conditioning factors, challenges, practices, and technologies in this research topic. To do so, we conducted an SMS and a field study. After investigating the literature, we selected 23 studies to analyze the state-of-the-art about transparency in SECO. Next, we interviewed 16 software developers to characterize the importance of conditioning factors for transparency identified in their interaction with GitHub. As a result, we presented a conceptual framework that structures all the knowledge about transparency in SECO obtained in both studies. This study was awarded as a distinguished paper at the 37th Brazilian Symposium on Software Engineering (SBES'23) [16] and an extended version was published in the Journal of Software Engineering Research and Development (JSERD) [23].
Study 3: Investigating Factors on Information Consumption by Software Developers on Software Ecosystem Portals. This study aimed to investigate the factors associated with the information consumption forms that influence software developers' decisions about where to seek knowledge about a SECO technological platform. To do so, we conducted a survey and a field study with developers to understand their information consumption in SECO portals. Based on the results of the 50 responses from the survey, refined and complemented with 14 interviews from the field study, we made four suggestions to favor developers' engagement and transparency in information consumption and interaction within SECO portals. This study was awarded as a distinguished paper at the 19th Brazilian Symposium on Information Systems (SBSI'23) [24] and an extended version was published in the Brazilian Journal of Information Systems (iSys) [4].
Study 4: Exploring Transparency as a Sustainability Goal in Software Ecosystems. This study aims to express transparency as a sustainability goal in SECO in a systematic way. To do so, we propose an approach that uses decision maps to make sustainability-driven decisions considering the transparency of SECO information and processes. This study was published at the 12th ACM/IEEE International Workshop on Software Engineering for Systems-of-Systems and Software Ecosystems (SESoS'24), co-located with the IEEE/ACM 46th International Conference on Software Engineering (ICSE'24) [25].
Study 5: What Affects Developer Experience in Software Platforms? This study aimed to investigate and characterize the factors that affect DX in software platforms. After investigating the literature, we selected 35 studies to analyze the state-of-the-art of DX in development on software platforms. As the main contribution, we provided a set of factors organized into six categories that affect DX in software platforms. This set of DX factors can be used as a reference guide to support researchers and practitioners in improving the development processes of the products and services on software platforms. This study was published at the 15th International Conference on Software Business (ICSOB'24) [26].
Study 6: Exploring Developer Experience Factors in Software Ecosystems. This study aimed to investigate DX factors and characterize their influence on software developers to adopt and keep contributing to a SECO. To do so, we conducted an SMS and selected 29 studies to analyze the state-ofthe-art. Next, we conducted a Delphi study with 21 software developers to evaluate how 27 DX factors, identified in our SMS, influenced them to adopt and keep contributing to a SECO. The factors "financial costs for using the platform", "desired technical resources for development", "low barriers to entry into the applications market", and "more financial gains" have the most influence on developers to adopt and keep contributing to a SECO. We concluded that DX is crucial to the dynamics of SECO, promoting its health and sustainability. This study is under evaluation at the Journal of Systems and Software (JSS).
This section presents the planned activities of the research until the thesis defense. This is a mid-stage PhD submission to the retreat. Each activity is written down in an action column and marked with an "X" in the cell representing that an activity will be carried out in a certain period (see Table 1). Currently, we are developing the conceptual model on transparency and DX in SECO and the set of guidelines for improving transparency in SECO portals. In addition, we are working on the submission procedures for upcoming studies to our Research Ethics Committee.
These are the main expected scientific and technological contributions of this doctoral research project:
• A conceptual model on transparency and DX in SECO;
• Set of guidelines for improving transparency in SECO portals from DX; • Computational tool for tracking the interaction of developers in SECO portals (an extension of UX-Tracking); • Transparency assessment checklist in SECO portals; • Mechanism to generate reports through developer feedback on DX; • T-SECOP: a framework to provide a set of guidelines for improving transparency in SECO portals from a DX perspective.
In this research, we have been collaborating with Professor Patricia Lago from Vrije Universiteit Amsterdam and researchers from the Federal University of Pará (UFPA) and the University of São Paulo (USP) to develop artifacts based on their expertise on HCI (one key area involved in this research).
This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior -Brazil (CAPES) -Finance Code 001 and Grant 88887.928989/2023-00, CNPq (Grant 316510/2023-8), FAPERJ (Grant E-26/204.404/2024), and UNIRIO.