[Context/Motivation] For areas such as Government IT Procurement, the Software Requirements Speci cation (SRS) often forms the basis for a public procurement. In these cases, having domain knowledge is often mutually exclusive to knowing RE. Domain experts lacking the necessary RE experience face issues assessing the quality and correctness of the SRS. This especially forms a problem for situations where the SRS acts as the base for a proposal and resulting contract, which is why often third party RE experts are consulted for evaluating the SRS beforehand. These experts are highly motivated to improve their process and o er a more uniform and better service. [Question/problem] Is our developed checklist a valid instrument to support the RE practitioner in the SRS validation process? [Principal ideas/results] We propose to empirically evaluate the checklist in a live study. Participants of our Evaluation Study are asked to simulate the validation of a sample SRS, guided by our checklist. We will analyze the data from a post-use questionnaire at the end of the session using a mixed method approach to assess the quality and usability of our checklist and its expected impact on the validation process. This assessment will be part of the overall validation of our checklist. [Expected Contribution] We expect to gain knowledge regarding the quality of our instrument. Secondly, this live study contributes to the validation, and thus the realization of a practical tool to be used by RE practitioners worldwide.
1.1
For areas such as Government IT Procurement, the Software Requirements Speci cation (SRS) often forms the basis for a public procurement, e.g. through a request-for-proposals process. In many situations, having domain knowledge is mutually exclusive to knowing RE. This introduces two challenges, namely (1) the persons with domain knowledge lacking the tools to properly create and/or assess the SRS and (2) the RE experts potentially not having su cient domain knowledge to cover. The use of an instrument supporting the SRS validation process could be of help in both scenarios. Whenever a call for bid is involved, the requirements speci cation will form a basis for the contract. Not only does having a good SRS prevent ambiguity between the procurer and the supplier, it also acts as a safeguard against any legal problems further down the road. Because of this, validation is an integral part of the overall creation of an SRS. It is of importance that a representation of the stakeholders c.q. domain experts within the procurer's company understand and agree with their respective parts of the SRS and can reasonably assess its quality. Often, third party experts are contracted to validate the SRS before initiating the IT procurement process. These experts are highly motivated to improve their process and o er a more uniform and better service. 1.2
Our motivation stems from improving the validation process in order to
contribute to better software being developed. The goal of this live study is to
evaluate our instrument (the checklist) and its e ect on the validation process of
a given SRS. The live study will most notably help identify strengths and
weaknesses, identify possible missing elements and give insights to the applicability
of the checklist by a variety of users [
The project resulted as a collaboration of the rst author and a large consulting
company in the Netherlands acting as a third party expert in the validation of
SRS's in the Government IT Procurement area. This research project started
within the University of Twente's course on Advanced Requirements
Engineering taught by the second author. The instrument is now being developed as part
of the Master's Project for Computer Science at the University of Twente.
Following [
The main goal of this live study is to get a hands-on impression of the quality of the instrument by having it applied by a variety of participants on a given SRS. A side bene t of the live study is the announcement of our research to the scienti c community and to gain traction amongst it. Speci cally, we want to know what he strong and weak points of the checklist and whether there are any missing elements (checks). To this end, we ask the following research questions: RQ1 What is the quality of the checklist based on the selected criteria? RQ2 What is the e ect of the instrument on the validation process? 1.5
For this speci c live study, we consulted the empirical evaluation guidelines
provided by Wieringa[
The rst author, de Laat, is a Master's student in the Computer Science program of the University of Twente. He earned a bachelor's degree in Business & IT from the University of Twente. He has a broad professional experience in developing business solutions for Microsoft SharePoint and is currently a part-time, Product Designer and Ruby On Rails developer at Nedap Healthcare.
The second author, Daneva, has been using empirical evaluation techniques in her research since 2001. In 2012, she co-authored an online study that was featured in the REFSQ program, and was about evaluation of a checklist for reporting empirical research in RE. 2
The study participants (50-60 expected) will be drawn from the pool of attendees at REFSQ 2018. We require no particular participant pro le. For the research to produce informative results, we need at least 20 participants so that we have at least 5 participants working on each non-core segment of the checklist. All participants will attend jointly the introduction to our live study. After this, the participators will be divided into the available rooms to make sure everybody has a seating spot and su cient space. Participants will receive one of four sets of materials (see section 2) and asked to apply their respective set of checks during a 60 minute session. Afterwards, during a 10 minute period, participants are asked to ll in a questionnaire in order to provide perceptions, opinions and their personal observations from using the checklist. The nal 10 minutes are reserved for the conclusion, thanking and gathering of the materials. 2.1
We need the help of the local organization to book su cient rooms where the participants can work in isolation from each other. Participants will be provided with (1) A sample (short) SRS, (2) their selected set of 'core' and 'non-core' checks including relevant information from the reference manual, (3) an answersheet with check-boxes to mark the results of the application of the check, (4) the post-use questionnaire. Those that brought a mobile device will be requested to ll in the questionnaire online using SurveyMonkey to speed up analysis. The checks given to the participant are described in full, checks that are not will be referenced by name only to allow for the participant to be able to say something about the completeness. Everybody will be given the option to leave their contact details if they desire to be kept informed about the research. 2.2
During the live study, all participants will be asked to ll in a questionnaire consisting of 1. A set of background questions assessing: (a) the participant's experience with SRS's, (b) which version of the non-core sets the participant received and (c) an indication of how much time they spent on the core v.s. the non-core part. 2. A critical feedback survey consisting of: (a) a set of closed questions and (b) a set of open questions
In the closed questions section the participants are requested to rate aspects
of the checklist based on a 9-point Likert scale where 1 means "strongly disagree"
and 9 means "strongly agree". The rst set of closed questions are derived from
[
The second set of questions, put together by the authors, expands on this
list and asks the participants whether they nd the instrument: to help prevent
task saturation [
The set of open questions consists of four open questions[
Finally, the participant is given the option to write down any nal thoughts regarding the instrument or the session. On a separate document the participants will be able to leave their contact details in case they wish to receive further noti cations regarding the development of the instrument. 2.3
Data resulting from both the paper and electronic questionnaires will be
combined and analyzed using o -line tooling on a secure computer. The recording
units (a single word or phrase describing a strength or weakness) resulting from
the open questions are grouped into categories identical to those of the closed
questions. The categories of the rst set of closed questions will attribute to
answering RQ1 and the second set to answering RQ2. Diverging stacked bar
charts and a Grouped Bar Chart will help visualize the quantitative responses
as described by Robbins, N. B., & Heiberger, R. M. [
Internal threats for validity include the possibility that some participants have experience in using checklists. Also, it may happen that some participants spend more time on one section than the other (for instance, the core v.s. non-core parts). This all may a ect our results. However, we plan to mitigate these threats by collecting information on the prior checklist-related experience of the participants and on how much time they spent on core v.s. non-core checks. If we get less than 20 participants we would only be able to assess the checklist based on it's 'core' contents. An external threat to validity is that the provided sample SRS might not be representable to those used in a real scenario, a ecting for instance the applicability of the instrument. We attempt to mitigate this by selecting a reasonably standard sample SRS. Outside the scope of this live study, we plan to mitigate this in future test sessions by having multiple experts test a diverse set of requirements speci cations. 2.5
We will work with the local organization so that our invitation to the study is received by each REFSQ attendee upon registration. During the conference we aim to share our ndings but might not have su cient time to analyze all results, meaning we would have to limit our analysis to data derived from the online responses. Apart from contributing to our research, participants will have the option to voice their opinions and/or concerns regarding the development of the checklist. Participants who chose to leave their contact information will be kept informed about the progress of the instrument and will be sent a copy of the resultant paper(s). After the conference the results of the live study will be part of a Technical Report issued by the University of Twente. We are planning of writing a journal paper in which we will motivate our checklist proposal and resent its empirical evaluation. This live study results will be also included in the journal submission. 2.6
The lling in of the questionnaire will be done anonymously and participants will be asked to give consent on the analysis, manipulation and publication of the data they provide during the live study. Apart from assessing the participants' experience with requirements speci cations, no personally identi able information will be asked of the participants during the test session. Optional participants' contact details will be collected on a separate document to guarantee the results of the questionnaire cannot be traced back to the participant.