{ Human error { A mental error, i.e. failings of the thought process during problem solving, while formulating the plan, or while executing the plan. { Example { The requirements author lacks domain knowledge. As a result she incorrectly believes that the stakeholders have told her all information relevant to that functionality. { Fault { The requirements speci cation omits the requirement.
The elicitation and documentation of software requirements is a human-based
activity. Therefore, it is not surprising that most failures can be traced back to
defects related to human factors [
To clarify the terms, we introduce some de nitions: Section 3.2 provides a more complete description of human errors in the introduction to the rst survey question.
In our previous work, we identi ed and extracted human errors that have been reported in the literature. Building upon a standard human error framework from psychological research, we constructed an initial Human Error Taxonomy (HET) to classify and describe these human errors. To better understand this preliminary taxonomy, we conducted a number of studies to test the feasibiity and usefulness of the HET in classroom settings (see Section 2 for details).
In constructing and testing the HET, we realized that many of the reports about human errors in software engineering lack important detail to fully understand the error. The next step in our work is to gather a more complete understanding of human errors in software engineering by gathering input from the larger requirements engineering community. We are especially interested in input from industrial practitioners. This information will allow us to update the HET to more accurately re ect the types of human errors occurring in practice. Copyright 2017 for this paper by its authors. Copying permitted for private and academic purposes. Therefore, we are conducting this on-line study at REFSQ'17 to gather this important information. 2
This study is one in a series of empirical studies our research group has
conducted about the use of Human Error information to improve the quality of
software requirements. The prior studies that our team has conducted on this
topic include:
{ An experimental study to evaluate the feasibility of using the HET for fault
and error detection. The result of this study showed that the use of the
HET allowed developers to identify errors in requirements documents and
nd additional faults during a reinspection guided by knowledge of the
existing errors. The participants had a positive impression of the HET [
The results of this study have been submitted to REFSQ'17. { An interview study conducted with local industrial practitioners that serves as the pilot for the study proposed here. We followed a very similar protocol as described above. We are still in the process of analyzing the data.
We conducted most of these previous studies in the context of a classroom environment. While those results are interesting, for the work to have real value for practice, we need to gather additional data from practitioners, which is the goal of the survey. 3
This section describes the research questions, survey design, participants, analysis plan, and potential threats to validity.
To address the goals of this study, we de ne two research questions: RQ1: What type of human errors and related faults do requirement engineers make on real projects? RQ2: What methods do practitioners use (or intend to use) to prevent requirement errors? 3.2
To gather the data to answer these research questions, we conduct an online survey during the REFSQ'17 conference. The survey contains three sections, described as follows.
Part 1: Human Errors The survey provides the respondents with an introduction to the concept of human error in software requirements. The survey contains the following background text regarding human errors:
Developing software requirements depends heavily on human thought, and so may be compromised by the failing of human cognition during the process of problem-solving, planning, or execution. These failings are referred to as human errors.
Central to the concept of human error is the construction of a plan, i.e. a series of actions, to achieve a desired goal. Three types of human errors can occur during this process: slips, lapses, and mistakes. A simple example will help to illustrate these types of human errors. Suppose that your goal is to drive to the store. This goal entails a series of steps including: starting the car, backing down the driveway to the street, navigating the route, and parking in the store's parking lot. A slip occurs when you perform the planned steps incorrectly or in the wrong sequence. In our example, inserting the incorrect key into the ignition to start the car is a slip. A lapse occurs when you forget the overall goal of the planned task in the middle of the sequence of steps or you omit a step that is part of a routine sequence. In our example, forgetting to to put the transmission into reverse before pressing the accelerator is a lapse. A mistake occurs when you design an incorrect plan to begin with (rather than making an error while executing a correct plan). In our example, failing to account for a closed bridge (i.e. an incorrect plan) resulting in tra c delays is a mistake.
In this survey, we are collecting information about human errors to help improve the accuracy and reliability of requirements engineering. We invite you to describe the human errors that occur during the requirements engineering process in your organization. Along with each error, it will be helpful if you can also describe the resulting fault(s) (e.g., an incomplete requirement or inaccurate requirement) that resulted from the error. Please note, that if con dentiality reasons prevent you from disclosing full details, we would be happy with an anonymized version of the error/fault.
The survey then asks the following question: 1. Please describe each requirement error that occurs during the requirements engineering process, along with an example of one or more resulting faults. Part 2: Error Prevention Techniques The survey provides the respondents with some examples of the types of Error Prevention techniques used in other organizations, including concepts like: changing task design, changing institutional processes, or instituting training. Then, we will ask the participants to complete the second portion of the online survey to describe the approaches they have taken (or plan to take) to prevent the errors described in Part 1 from occurring again. 1. What revisions to the requirements engineering process, if any, did you or your organization make to prevent these errors from happening again? If no revisions have been made, are there any that you would like to make? (Please indicate whether the revision described is one that you have already made or one that is planned) Part 3: Demographics These questions provide context for understanding and interpreting the results of the survey. The survey includes the following questions: 1. What is your highest degree (e.g. Ph.D, Master, or Bachelors)? 2. In which eld is your highest degree? 3. For each of the following, indicate the number of years of experience: Requirements elicitation, Requirements documentation, Requirements validation. 4. In which organization are you currently employed (company or university)? 5. What is the domain(s) of the software produced by your organization? 6. How long have you been at your current organization? 3.3
We anticipate two types of participants: { Industrial Practitioners - we seek requirements engineers from industry who have experience with the requirements engineering process. These participants will be able to provide real-world experience regarding the types of errors that occur commonly in practice in their environment. No additional experience or background is necessary. { Researchers - we also seek requirement engineering researchers who are familiar with the types of errors that they have observed through their research. As researchers who study requirements engineering in multiple settings, these participants will provide a di erent perspective to the practitioners, who will likely only be knowledgeable of their own environment. The goal of the survey is to better understand the types of requirement errors that occur in practice. Given that the most interesting data will be provided in open-ended questions, we will employ a qualitative analysis process. First, two researchers will analyze and code each error description. Then the two researchers will compare their analyses and resolve any discrepancies. We can then use the demographic data to analyze the results of the qualitative analysis.
In addition to performing the above analyses on the entire data set, we will also conduct it on subsets of the data, including: Industrial Practitioners, Researchers, and PhD students. The goal of this subgroup analysis will be to help determine whether these groups of participants view the problem from di erent perspectives and provide additional insights into these groups.
We will use the results of this analysis to evaluate the su ciency of the HET developed previously. We will determine whether the HET can properly describe all of the errors reported in this study. If there are de ciencies in the HET, we will update it as necessary. 3.5
This study faces the following potential threats to valdity.
{ Construct Validity - we are asking participants to recall the types of errors that have occurred in the past. It is possible that participants may not properly remember these errors. Therefore the data we gather about errors could be invalid. { External Validity - we are not sure whether the participants who attend the study will provide a good cross-section of industrial experience. Therefore, any conclusion we draw may not be fully representative of the population.