Using Human Error Abstraction Method for Detecting and Classifying Requirements Errors: A Live Study Vaibhav Anu1, Gursimran Walia1, Gary Bradshaw2, Wenhua Hu3, Jeffrey C. Carver3 North Dakota State University1, Mississippi State University2, University of Ala- bama3 1 Introduction Inspections, a proven quality improvement approach [3, 7], are a process where a team of skilled individuals review a software artifact (e.g., requirements specification document) to identify faults. Traditional fault-based software inspections (like Fault Checklist inspection) focus inspectors’ attention on different type of faults (e.g., incor- rect or incomplete or ambiguous requirements) [7]. Even a faithful application of vali- dated fault-based techniques does not help inspectors in finding all faults. As a result, a larger part of (40%-50%) the development effort is spent fixing issues that should have been fixed in an earlier phase [3]. Hence, there is a real need to improve early fault detection and to help developers avoid the unnecessary rework. We hypothesize that inspections focused on identifying human errors (i.e., the underlying cause of faults) are better at identifying requirements problems when compared to inspections focused on faults (i.e., manifestation of human error). On those lines, our recent work [1, 5] uses a Cognitive Psychology perspective on human errors to improve the practice of requirements inspections. Human errors are understood as purely mental events, failings of human cognition in the process of prob- lem solving, planning, and acting. Errors, in turn, will produce faults, a physical mani- festation of the error. It is important that a clear distinction is made between human errors (mental events) vs program errors (related to coding or programmatic failures). To help inspectors in identifying human errors, the authors over the past two years, have worked on developing a Human Error Taxonomy (HET) that classifies human errors that commonly during requirements engineering [2]. Additionally, we have also developed a human error analysis framework called the Human Error Abstraction (HEA) method that can guide inspectors in analyzing and abstracting (i.e., extracting) human error information from requirements faults, a process referred to as Error Ab- straction (EA) by Psychologists. Description of HET and HEA appears in Section 2. We recently carried out a series of empirical studies to validate the effectiveness of human error based inspections supported by HET against FC based inspection at two different sites [1, 5]. While the results were promising, the subjects did not have a sup- porting framework to assist them while abstracting errors from the faults. Analogous to human error investigation frameworks in Psychology, this paper discusses the design and evaluation of the HEA method during the live study. $PQZSJHIUªGPSUIJTQBQFSCZJUTBVUIPST$PQZJOHQFSNJUUFEGPSQSJWBUFBOE BDBEFNJDQVSQPTFT 2 Background In this section we briefly describe human errors and method for abstracting human errors from the requirement faults which in turn can help find additional faults. (1) Human Error Based Requirements Inspections: Error based inspections [6], works by assisting inspectors to identify and extract human error information from faults found during FC inspection, and then to use the abstracted human error infor- mation to guide the re-inspection. Our prior studies [1, 5, 10] have shown that, error based inspections are a significant improvement over fault-based inspections. However, an inspectors’ ability to find faults using the error information is highly dependent on their ability to correctly identify human errors during the error abstraction (EA) process. Therefore, the goal of this study is to evaluate the usability of HEA method during the EA step so that their inspection effectiveness can be further improved. (2) Human Errors: To assist inspectors during the Error Abstraction (EA) leg of human error based inspections, we have developed a Human Error Taxonomy (HET) that classifies the most commonly occurring requirements phase human errors built around Reason’s psychological account of human errors [9]. The complete develop- ment process of the HET can be found in [2]. Fig. 1. Human Errors (Slips, Lapses and Mistakes) Reason’s well-respected human error classification system, classifies human errors into slips, lapses, and mistakes, as shown in Figure 1. According to Reason [9], when faced with a situation that demands problem-solving, human operators go through two major information-processing stages: planning stage and execution stage. The human error mechanisms associated with execution (or action) stage are called slips and lapses. The error mechanism associated with planning stage is called mistake. As illustrated in Figure 1, assume that our goal is to drive to the store and entails steps such as starting the car, backing down the driveway to the street, navigating the route, and parking in the store lot. If we put the wrong key in the ignition, we have committed a failure of execution known as a slip. If we forget to put the transmission into reverse before step- ping on the gas, the omitted step is a lapse. Failing to take into account the effect of a bridge closing and getting caught in traffic is a planning mistake. Slips are execution failures that are caused by inattention and occur when a planned action is incorrectly executed. Lapses, which are also execution failures, occur when an action is forgotten (omitted) while executing a planned task or when an individual forgets their place in a planned task and ends up repeating an action. Mistakes are plan- ning failures and are generally a result of being in an unfamiliar situation. (3) Human Error Abstraction Method (HEA): Although the HET provides a con- crete list of the most commonly occurring human errors, EA is still a subjective process that different people might perceive in different ways. In order to reduce the subjectivity and complexity of EA we developed the HEA method which can be found in [2]. The HEA was created after performing pilot empirical evaluation of human error based inspections with different set of subjects [1, 5]. After the studies, the subjects provided feedback that EA can be improved by focusing the inspector’s attention on various RE activities (elicitation, analysis, specification, and management). Hence, HEA is developed to guide the selection of the appropriate RE activity and the situation where the human error might have occurred. We created HEA (Figure 2) to act as an intuitive frame-work to systematically guide inspectors during EA. Inspectors have to answer a set of questions (decision points) to trace a fault to an underlying human error. Fig.2. Human Error Abstraction (HEA) Method The HEA method (that asked specific questions) has been converted into a decision tree style framework that can better guide inspectors during the error-discovery (in con- sultation with the Cognitive Psychology expert, Dr. Bradshaw). This decision tree (Fig. 2) uses the skill-rule-knowledge framework developed by Rasmussen [8], wherein in- spectors are directed through decision points (based on cognitive failure patterns). Ma- jor decision points are discussed below: (i) Decision point D1 guides inspectors to distinguish between an error scenario as a planning scenario (i.e., Mistakes) or an execution scenario (i.e., Slips and Lapses). (ii) Decision points D2 helps inspectors to distinguish between inattention failures (i.e. slips) and memory failures (i.e., lapses). (iii) Decision points D3 and D4 helps identify the type of Mistake (i.e., rule-based vs. knowledge based mistake). It is hoped that this type of EA framework can help in- spectors navigate to correct human error classes. 3 Study Design The main goal of the live study is to evaluate the use of the HEAA tool in helping inspectors correctly abstract and classify underlying human errors responsible for the requirement faults. 3.1 Research Questions RQ 1: Are inspectors able to use the HEA method to accurately abstract and classify human errors that occurred during the requirements development process? RQ 2: Are the human error classes (Slips, Lapses and Mistakes) adequate and rele- vant to the requirements development process?? 3.2 Subjects and Artifacts (1) Subjects: The population of interest are subjects with familiarity with require- ments engineering activities and industry experience. We want to evaluate the useful- ness of the HEA method from practitioners and experts in academia. (2) Artifacts: An SRS document that specified requirements for a Parking Garage Control System (PGCS) will be used during the live study. PGCS SRS was developed by researchers at University of Maryland and was seeded with 35 realistic faults.. 3.3 Study Procedure During the study, subjects will be trained on how to use the HEA method to abstract errors from a small subset of PGCS faults (that will be provided to them), and then use the training to abstract errors from a larger subset of remaining faults. Experimental steps are described as follows: Step 1 – Training on Error abstraction (EA): During this 25-30-minute training session, subjects will be trained on human errors classes of HET, and how to use the HEAA tool to abstract errors from supplied faults. Next, subjects will be asked to use the training to trace errors from a subset of five PGCS faults followed by the discussion of their results. Step 2 –Error abstraction (EA) on Remaining PGCS faults: During the remainder of the live study, subjects will use the HEA template to abstract and classify human errors (into Slips, Lapses, and Mistakes) from the a second subset of 15 faults in PGCS SRS. Step 3 – Survey: Post study, we will collect feedback from subjects on HEAA and EA using a survey that can either be handed out to the subjects or emailed to them. The following documents will be provided during the study run – • PGCS SRS: Hard copy (i.e., printout) or a downloadable PDF file that will be made available on local server. • HEA decision tree: Hard copy or a downloadable PDF file that will be made available on local server. The HEA decision tree will provide a handy template to enter their error abstraction data. • Error Report Form: The error report form will contain 20 faults (randomly se- lected from a list of 35 seeded faults) in PGCS SRS. Subjects will be asked to abstract errors from the first 5 faults during the training followed by error ab- straction for the remainder of 15 faults post training. We can supply the error- report form as a hard copy and also make a PDF copy available for download. 3.4 Data collection and Analysis Fig. 3 provides an example of the information that subjects will be asked to report when abstracting a human error from a fault (one row for each fault-error mapping). To enable an objective error data analysis, when analyzing the error abstraction accuracy, we will compare the human error classification reported against the expected human error class. The expected human error class for each fault (for PGCS faults) has been agreed upon after discussion amongst authors and Psychology expert, Dr. Bradshaw. This will provide insight into whether subjects are able to use the HEA method to dis- tinguish between the 3 error mechanisms. Fig. 2. Sample Error Report Form Additionally, at each step, we will ask subjects to report the effort spent (amount of time taken to complete the task). We will also analyze the written accounts of human errors collected during the study to analyze whether the accounts of human errors are consistent across all subjects. 4 Potential Validity Threats The live study faces the following validity threats: (1) For some participants, this study might be the first introduction to Cognitive Psy- chology concepts (like slips, lapses, and mistakes) and it is possible that they may not properly understand the concepts within the stipulated time allocated for the study run. We intend to mitigate this threat by involving a Cognitive Psychologist (Dr. Bradshaw) when training the participants on human errors and how to abstract errors. (2) The heterogeneity of participant population (mix of researchers and practitioners) is also a potential threat to the generalizability of the results of the live study. The par- ticipants will be of different backgrounds and this may contribute to variability in measures. We will collect data regarding the affiliation of subjects (industry or aca- demia) in order to perform separate data analyses on the two expected subgroups. References 1. Anu, V., Walia, G.S., Hu, W., Carver, J.C. and Bradshaw, G. 2016. Effectiveness of Human Error Taxonomy during Requirements Inspection: An Empirical Investigation. Software En- gineering and Knowledge Engineering, SEKE2016. 2. Anu, V., Walia, G.S., Hu, W., Carver, J.C. and Bradshaw, G. 2016. The Human Error Ab- straction Assist (HEAA) tool. http://vaibhavanu.com/NDSU-CS-TP-2016-001.html 3. Boehm, B. and Basili, V.R. Software Defect Reduction Top 10. Computer. 34,1 (2001), 135–137. 4. Hsieh, H.F. and Shannon, S.E. 2005. Three Approaches to Qualitative Content Analysis. Qualitative Health Research. 15, 9 (2005), 1277–1288. 5. Hu, W., Carver, J.C., Anu, V., Walia, G.S. and Bradshaw, G. 2016. Detection of Require- ment Errors and Faults via a Human Error Taxonomy: A Feasibility Study. 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM’16. 6. Lanubile, F., Shull, F. and Basili, V.R. 1998. Experimenting with Error Abstraction in Re- quirements Documents. In Proc. of the 5th International symposium on software metrics. 7. Porter, A.A., Votta, L.G. and Basili, V.R. 1995. Comparing detection methods for software requirements inspections: a replicated experiment. IEEE Transactions on Software Engi- neering. 21, 6 (1995), 563–575. 8. Rasmussen, J. and Vicente, K.J. Coping with human errors through system design: implica- tions for ecological interface design. Jour. of Man-Machine Studies. 31, 5 (1989), 517–534. 9. Reason, J. 1990. Human error. Cambridge University Press. 10. Walia, G.S. and Carver, J.C. 2010. Evaluating the use of requirement error abstraction and classification method for preventing errors during artifact creation: A feasibility study. In Proc. of the 24th International Symposium on Software Reliability Engineering, ISSRE’10