The purpose of this live study is to investigate the impact of emotional states on the e ciency of correcting defects within a Conceptual Model veri cation context. We plan to use the CoSTest tool for verifying defects and measure the experimented emotions during the testing tasks through I-PANAS-SF and SUDS instruments. Furthermore, the design, overview and potential threats of this experiment are presented.
1.1
Emotions have an important role in our life and their in uence on our work is not an exception. According to Weiss and Cropanzano [WC96], emotions and moods have e ects on work performance, a ecting the duration of activities and the perceived experience. This situation is not only present in industrial activities; some researchers have been analyzing the in uence and importance of emotions in software engineering [Wro13, CSdlB+14, MF15, RFB+19], being the software development process one of the principal centers of interest.
Wrobel [Wro13] analyzed the in uence of the emotional state on the performance of programming tasks collecting data through a questionnaire and an interview for each participant, the Job Emotions Scale (JES) was applied to measure human emotions. Romano et al. [RFB+19] also investigated in this area, analyzing emotions when novice developers apply the TDD (Test-Driven Development) approach on change tasks; in a similar way, they collected data through self-reported emotions by the participants. In the same way, we propose to analyze the in uence of the emotional state on the work productivity based on correcting defects in conceptual models.
In line with this notion, according to Granda et al. [GCVP15], defects (e.g., missing, wrong and unnecessary elements) in conceptual models (e.g., UML class diagram) can be located in several ways through Validation & Veri cation techniques, which use a detection mechanism (based on rules, metrics, and modelling conventions) for this purpose. According to the technique nature, this can be statically or dynamically supported by a tool and they can have di erent scope and limitation depending on its purpose (i.e., detect, prevent and resolve). 1.2
In this live study proposal, we start from the hypothesis that the emotional state of workers in uences on their productivity when they correct defects during the software development process [WC96]. It could be in uenced by the user pro le (e.g., background, preferences, experience, personality) or the way how the emotions are experiment for each one. Moreover, we focus on investigating the usefulness and user experience of using CoSTest tool [GCVP17a], which is a tool created to help in the tasks of defect detection and correction.
Therefore, in order to investigate 1) what emotions are experienced by the subjects during the correction of defects using the tool, and 2) how useful the testing tool is perceived for supporting the veri cation of conceptual models, we present the design and plan of a live study to be conducted with the REFSQ attendees. 2
In this section, we present the scenario that illustrates the context of the use of CoSTest tool [GCVP17a], which takes as input a requirements model based on communication analysis to generate test cases and validate a conceptual model (i.e., UML class diagram) evaluating its correctness and completeness.
The software development process includes di erent tasks, where testing and correction tasks are important to assure quality. Requirements engineers, analysts, and testers have an important role inside these tasks, using a UML modeling editor and the CoSTest tool to detect defects and verify the changes in the UML class diagram. Doing these tasks, they commonly experience a range of emotions (e.g., anger, frustration, stress, anxiety, happiness), that could be generated by the use of the tool; knowing these emotions let us nd potential feedback to correct and improve CoSTest tool. 3 3.1
The live study proposal aims to 1) analyze the in uence of emotional state on the e ciency for performing defect correction tasks, and 2) evaluate the utility and experience of using the CoSTest tool [GCVP17a] as a support in the correction of conceptual model defects.
From this goal, the following research questions are derived: RQ1: How does the emotional state in uence on e ciency for performing defect correction tasks? RQ2: Is the CoSTest tool perceived as useful to support the veri cation of conceptual models? Type of study: to investigate this research problem, we propose to conduct a quasi-experiment, where all participants have the same set of defects to be corrected and will use CoSTest1 as a support tool.
to automatically detect defects in conceptual models). Other factors which can impact the results are the selected Conceptual models (CM) (see Section 3.3.2 for more details about them) and the Defects injected into the CMs. By using the defects classi cation proposed in [GCVP15], we will use three defect types (i.e., missing, wrong and unnecessary elements) to inject six defects per CM (two for each defect type). As dependent variables: user emotional state that can be determined by analyzing the self-reported emotions through speci c questionnaires (see Section 3.3.1 for more details); and e ciency for correcting defects that represents the relationship between the total corrected defects and the total time assigned for correcting all defects. 3.2 3.2.1
We plan for 30 participants. Students, senior researchers, and practitioners are very welcome and need to bring their laptops to execute CoSTest tool, which is based on Java (jar le). In order to expedite the installation of CoSTest and avoid compatibility problems with Java versions, we will provide a virtual machine with all the software required in this study. Prior knowledge and experience on modeling UML-based conceptual schemas (i.e., class diagrams) using tools (e.g., UML2Tools editor2) is required. However, some tips to remember or 1https://costestproject2017.wordpress.com/ 2https://www.eclipse.org/modeling/mdt/?project=uml2tools reinforce its use will be included in the training phase. We choose REFSQ'20 to run this experiment thanks to the strong knowledge in testing and knowledge in model-driven development process of REFSQ's attendees. 3.2.2
The testing tasks are the key leverage point for practitioners (e.g., project managers, analysts, testers) who want to develop software systems with high quality level. In this context, we think that this study might bene t to participants by getting:
Knowledge and training on a tool to support the veri cation of conceptual models through test cases, which could be applied on their workplaces or development tasks.
Experience in running studies to analyze emotions.
Access to instruments for evaluating and measuring emotions, such as International Positive and Negative A ect Schedule Short-Form (I-PANAS-SF) and Subjective Units of Distress Scale (SUDS), which could be used for the participants in other studies based on human emotions. 3.3 3.3.1
We implement a web-based survey using the Qualtrics tool3, which is composed by the following parts: 1) a demographic and educational background questionnaire (e.g., sex, age, educational degree, domain expertise); 2) CoSTest training, we provide the required material to use CoSTest tool; 3) Relaxing, during this period we ask to participants get relaxed; 4) Testing task, we provide two conceptual models with di erent defects for correcting, also participants have an option to upload their responses; 5) Emotion state questionnaires; in order to understand the di erent emotions experimented by the participants during the contest, we use the I-PANASSF questionnaire and the SUDS scale. The I-PANAS-SF is a list of 10 adjectives used to describe di erent emotional states: 5 states of Positive A ect (PA) and 5 states of Negative A ect (NA). The PA scale measures activity and pleasure, while the NA scale relates to fear and stress [KWR11, Tho07]. The Subjective Units of Distress Scale (SUDS) can be used to measure the intensity of anxiety/distress [BOC+10]; this instrument rates on a scale from 0 to 10, so that subject marks a point on the range where he perceives his anxiety/distress state has been located during the di erent situations presented on the contest. 6) CoSTest evaluation, we propose di erent questions in order to investigate whether CoSTest tool was useful to test conceptual models, and 7) Experiment feedback, we collect feedback about the study. 3.3.2
CoSTest [GCVP17a] is a tool to verify automatically conceptual models using test cases from an existing approach to communication-oriented requirements engineering. In this live study , we will ask the participants to carry out two testing tasks (i.e., two conceptual models, CM1 and CM2) and run three test cases associated with each one by using the CoSTest tool. Participants will be asking to consider one test case at a time (starting with the rst one); if the result of the test case shows the conceptual model has a defect, they have to correct it and rerun the test case to make sure that the defect correction was done well. The participants will be able to perform the next test task when nishing the current one. The total time allocated to perform the two test tasks is 30 minutes, if the participant nish before ending time, he could try to generate new test cases in the tool to nd additional defects. The idea is to motivate participants to use the tool so that this will help to measure their perception of the usefulness of CoSTest.
The experimental objects consist of the speci cations of two conceptual models: CM1 is a Super Stationery (SS) system, where CM de nes the information system of a company that provides stationery and o ce material to its clients; and CM2 is a Photography Agency (PA) system where a CM de nes the information system that manages photographers and their photographic reports for distribution to newspaper publishers. Regardless of the experimental object, we provide the participants with the following experimental material: (i) a brief description for each information system modeled in CM1 and CM2; (ii) a template to create test cases using CoSTest; and (iii) an example test suite (each one with three test cases), developed by the authors to test each CM using CoSTest. We opted for SS and PA as experimental objects because they are often adopted to learn/practice CoSTest and were used in past empirical studies on Mutation Testing [GCVP17c]. 3.4 Experimenter Tasks Subject Tasks
Analysis and Interpretation
Report of
results
Data collection
Demographic and educational background
Perception on usefulness, experience and emotions Instructions (5 min)
CoSTest training
(30 min) Preparation
Relaxation (5 min)
Correction contest (30 min)
Questionnaire
(10 min) Experiment
Award (5 min)
Post-experiment
The study is composed of three phases, as shown in Figure 1. (i) Preparation: rst we explain details about the study and request to read and sign the informed consent form, where it is detailed the privacy and con dentiality terms; after we ask to participants with a demographic and educational background questionnaire ( ve minutes). As the CoSTest could be a new tool for the participants, we will give a training phase about how to use it (30 minutes). As next step, we need to uniform the emotional state of all participants (e.g., someone could came to the experiment already stressed) to avoid the in uence of previous emotions in our experiment; for that reason, participants are asked to stay quiet and watch a video during ve minutes to get relaxed. (ii) Experiment : in this phase, we provide di erent test cases for using the CoSTest tool (CMs are delivered in random order), participants will have one attempt to send their solution for each task. With the objective to generate competitiveness, during the experiment each time that a participant submit a correct solution of a testing task, we will give a globe (similar to ICPC4). After the contest, participants will be asked about their perceived emotions during the experiment, their experience using the CoSTest tool, and their feedback regarding the study (see Section 3.3.1 for more details about these questionnaires). (iii) Post-experiment : after to process all submissions of the participants, we will reward three participants who detect and correct more defects in the less time. 4
The di erent factors triggered by live study (e.g., place, settings) might a ect the observed variables. We mitigate this threat by performing the study in similar conditions for each participant (e.g., material, testing tasks, rules of contest). The tools (i.e. CoSTest and UML2Tools Editor) used in this experiment will be tested to adjust the settings on the virtual machine used for software installation. For this purpose, we will use 5 reference points, taken from other MCs, which are not part of this study. Additionally, our study could be negatively a ected if both artifacts (MCs and injected defects) were not properly selected. The MCs have been taken from other experiments that have been carried out with the tool [GCVP17b] and the defects are those generated by the mutation tool [GCVP17c]. Another threat is the emotions of the participants before starting this live study (due to some previous activities or experiences on the day) might a ect the perception of their emotional state during this study; to mitigate this threat, we have prepared a relaxing activity to uniform the emotions of all participants before starting this experiment.
This validity is regarding the generalization of our results outside the experiment setting; in this context, a possible threat could be the selection of participants. Nonetheless, it is mitigated by the REFSQ attendees, because they have di erent personalities, experiences and educational backgrounds, such as master/PhD students, senior researchers, and practitioners. Another possible threat is that the environment where this study will be carried out might not be a typical context for doing software work; to mitigate this threat, we will conduct the study in a quiet environment, trying to limit unnecessary distractions and trying to give comfortable work-spaces for the participants.
As part of this study, our instruments are based on questionnaires with self-reported responses and as consequence, participants could be afraid of giving information about their emotional states or personal information; however, this threat is mitigated through our privacy and con dentiality terms that specify their information and responses are going to be anonymous. Furthermore, the selected instruments are well known and have been used in other works to measure emotions [Epp16, AB18]. Some other possible threat is determining the correctness of a solution for a defect because it could a ect the measure of e ciency; this threat is mitigated by the CoSTest tool, which veri es automatically if the defect was solved successfully or not. 5
Franci Suni Lopez is a PhD candidate at National University of San Agustin of Arequipa. His main research focuses on using human emotions to empower the self-adaptation capability of software services. His research interest also includes software engineering for mobile development and self-adaptive software. He executed one empirical studies in the live study track of REFSQ (whose results have been published in [CFML19]) and an experiment at MEGSUS [SLCFMA18].
Angela Mayhua Quispe is researcher at La Salle University (Arequipa, Peru) and PhD candidate at National University of San Agustin of Arequipa. Her main research focuses on data visualization and pattern recognition from images and signals. She has a particular interest in analyzing physiological signals to recognize human emotions, with special emphasis on negative emotions.
Maria Fernanda Granda is a full professor and researcher of the Computer Science Department at University of Cuenca (Ecuador). She obtained her Ph.D in Computing in 2017 at the Universitat Politecnica de Valencia (Spain). Her main research focuses on Requirements Engineering, Software Testing & Quality and Model-driven Development. She has executed empirical studies using mutation testing and the results have been published in [GCVP17b], [GCVP17c] and [GCVP16].
This work has been supported by the KUSISQA project (014-2019-FONDECYT-BM-INC.INV) from the National Fund for Scienti c and Technological Development (FONDECYT-PERU) and World Bank, and the Direccion de Investigacion de la Universidad de Cuenca (DIUC) - Ecuador. Authors also want to acknowledge the anonymous reviewers and Live Study track chairs (Nelly Condori-Fernandez and Luisa Mich) for their valuable feedback. [AB18] [BOC+10]
A. Alhubaishy and L. Benedicenti. A ect and a ective trust in agile requirements engineering. In 2018 1st International Workshop on A ective Computing for Requirements Engineering (AffectRE), pages 9{12, Aug 2018.
Courtney L. Benjamin, Kelly A. O'Neil, Sarah A. Crawley, Rinad S. Beidas, Meredith Coles, and Philip C. Kendall. Patterns and predictors of subjective units of distress in anxious youth. Behavioural and Cognitive Psychotherapy, 38(4):497{504, 2010.
Nelly Condori-Fernandez, Denisse Mun~ante, and Franci Suni Lopez. Exploring users perception on security and satisfaction requirements of context-aware applications: An online survey. In 2019 Joint of International Conference on Requirements Engineering: Foundation for Software Quality [CSdlB+14] [RFB+19]
Broderick Crawford, Ricardo Soto, Claudio Leon de la Barra, Kathleen Crawford, and Eduardo Olgu n. The in uence of emotions on productivity in software engineering. In Constantine Stephanidis, editor, HCI International 2014 - Posters' Extended Abstracts, pages 307{310, Cham, 2014. Springer International Publishing.
Carrie Demmans Epp. English language learner experiences of formal and informal learning environments. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, LAK '16, page 231{235, New York, NY, USA, 2016. Association for Computing Machinery. Maria F. Granda, Nelly Condori-Fernandez, Tanja E. J. Vos, and Oscar Pastor. What do we know about the defect types detected in conceptual models? In 2015 IEEE 9th IEEE International Conference on Research Challenges in Information Science (RCIS), pages 88{99, May. 2015. Maria F. Granda, Nelly Condori-Fernandez, Tanja E. J. Vos, and Oscar Pastor. Mutation operators for uml class diagrams. In 2016 28th International Conference on Advanced Information Systems Engineering, volume 9694, pages 325{341, Jun 2016.
Maria. F. Granda, Nelly Condori-Fernandez, Tanja E. J. Vos, and Oscar Pastor. Costest: A tool for validation of requirements at model level. In 2017 IEEE 25th International Requirements Engineering Conference (RE), pages 464{467, Sep. 2017.
Maria F. Granda, Nelly Condori-Fernandez, Tanja E. J. Vos, and Oscar Pastor. E ectiveness assessment of an early testing technique using model-level mutants. In 2017 21st International Conference on Evaluation and Assessment in Software Engineering (EASE), pages 98{107, Jun. 2017.
Maria F. Granda, Nelly Condori-Fernandez, Tanja E. J. Vos, and Oscar Pastor. A model-level mutation tool to support the assessment of the test case quality. In 2017 Lecture Notes in Information Systems and Organisation, volume 11, pages 17{37, 2017.
Jahanvash Karim, Robert Weisz, and Sha q Ur Rehman. International positive and negative a ect schedule short-form (i-panas-sf): Testing for factorial invariance across cultures. Procedia - Social and Behavioral Sciences, 15:2016 { 2022, 2011. 3rd World Conference on Educational Sciences 2011.
Sebastian C. Muller and Thomas Fritz. Stuck and frustrated or in ow and happy: Sensing developers' emotions and progress. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, volume 1, pages 688{699, May 2015.
Simone Romano, Davide Fucci, Maria Teresa Baldassarre, Danilo Caivano, and Giuseppe Scanniello. An empirical assessment on a ective reactions of novice developers when applying testdriven development. https://arxiv.org/abs/1907.12290, July 2019. [Tho07]