[Context and motivation] Although traceability plays an important role in system development projects, it is often neglected or poorly performed. Stakeholders involved in system development projects have different use of traces, and therefore the required traceability and its implementation depends upon the stakeholders involved in the projects. [Problem/question] Traditional ways of implementing traceability using traceability matrices or office tools such Excel sheets are not effective, especially while developing large and complex systems. There exists several requirements management tools, but they have to be extended or tailored to achieve the traceability needed of the stakeholders, requiring human effort and resources. Little is known or has been reported on traceability usage, practises and challenges in the development of systems with safety and security implications. Such a knowledge and understanding is needed to develop cost-effective traceability solutions as well as to highlight the importance of traceability [Principal ideas] This paper presents a proposal for performing a survey - online questionnaire - with the participants of REFSQ'14 to collect and report their experiences on traceability. [Contribution] Since, REFSQ'14 participants have a diverse background, i.e. experiences with different roles and responsibilities in projects from industry, the survey will be a unique opportunity to collect and report data on traceability covering several stakeholder viewpoints.
1.1
Traceability is vital and should be implemented during development projects, especially for the development of systems with safety and other quality (e.g. security) implications where standards or regulatory bodies require implementation of traceability. However, when it comes to industrial practices, traceability is often neglected or not implemented in a satisfactory manner. In our experiences from both safety and non-safety industries, traceability matrices and Office tools (e.g. Excel sheets) are still widely used for implementing traceability. Such approaches to traceability practices are not viable, especially not for projects on developing large and complex systems.
There are several challenges related to implementing traceability including lack of
tools that are suitable for project needs and stakeholder needs, and additional cost and
effort needed to maintain traceability [
“Currently, there is poor understanding of what people need traceability for and how people actually use traceability over time. Further, traceability will not be created or maintained effectively if the required tasks to do so are themselves not understood and supported. Currently, there is poor understanding of what individuals and teams need to do to create and maintain traces. This distinction between satisfying the requirements of those stakeholders who establish traceability and those stakeholders who use traceability lies at the heart of many traceability problems, for these roles are not necessarily overlapping.”
Based on our experiences, this problem is also evident in development projects of
safety systems or systems with security implications. There is a lack of traceability
tools that can be used to address the needs of the stakeholders involved in such
development projects [
The planned survey is a part of a PhD project on improving traceability practices
during the development of safety critical systems. We have performed a narrative
literature study and proposed solutions – mostly focusing on the air traffic
management domain [
Goal, Research Questions and Research Design The primary goal of the survey is to investigate the purpose and uses of traceability during system development projects, identify the state of practice, and understand the challenges on implementing traceability during development projects.
The survey is mainly intended for the REFSQ participants who have experiences in projects on development of systems with safety and security implications. Since REFSQ attracts both practitioners and researchers in the area of requirements engineering and other system engineering activities, the variation of the profile of the participants – their roles and tasks in projects - will be large. Therefore, the survey will gather information related to several stakeholders and their traceability needs. 2.2
The survey addresses the following research questions: 1. What is the purpose of traceability in projects developing safety critical systems and systems with security implications? This question looks into the purposes – tasks, activities etc. – for which stakeholders use traces. 2. What are the practises of performing traceability? This question looks into identifying the techniques and tools being used by the stakeholders and their organisations for implementing traceability in their projects. 3. What are the challenges of implementing traceability? This question looks into the challenges the stakeholders face while implementing traceability. 2.3
We follow a non-experimental strategy, using the survey as a method to gather data from a large population. The purpose of the survey is both exploratory and descriptive on traceability uses and challenges.
The survey will be open to the participants of the REFSQ’14 conference. We aim for the participants who have experiences in creating and using traceability information during activities of a development project. Experiences in development of safety critical systems or systems with security implications will be beneficial. We consider participants with experiences in RE tasks, system designing, testing, V&V, quality assessment, qualification, project management.
We expect the REFSQ organisers to inform the conference participants about the survey. If required, we will provide organisers with information (slides etc.) on the importance of the topic covered by the survey, which could be used by the organisers to encourage participation.
If allowed, we will have a poster promoting the survey to the conference participants. We plan to offer four Amazon gift cards, each worth 50 euros, which can be used by four participants (selected by lottery) to purchase books of his or her choice.
Filling in the online questionnaire will be treated as participant’s consent to participate. Participants will be made of this before filling up the questionnaire.
The data collected through the survey will be kept confidential and will be stored securely in our company’s internal server, and will be deleted after completion of the survey related activities. The result of the survey will be anonymised. Implementation 1. The survey is available for the participants at the URL
http:// refsqsurvey.limequery.com/index.php/726296/lang-en 2. The questionnaire has been organized into the following parts: (a) Introduction: consent of participation, confidentiality and guidelines for completing the questionnaire. (b) Questions on the participants’ work experiences – their roles and responsibilities in relevant projects. (c) Questions on whether they have created or used traceability during developments, and if so, for which development activities (e.g. change impact analysis, V&V, and quality assessments) they have used traces. (d) Questions on what type of traceability techniques and tools the participants have used. (e) Questions on challenges the participants have experienced, e.g. lack of understanding, tools, and cost, while implementing traceability in the projects. 3. The questions has multiple choices, rating scales, and open-ended formats. 4. It is expected that the participant will be able to complete the survey within 15 minutes. 5. The survey was pre-tested with two colleagues with relevant work experiences. 6. The survey will be conducted during REFSQ’14.
Depending upon the format of the question, we will use analysis methods such as statistical tests for the multiple choice and rating scales and affinity diagrams and content analysis for the open ended questions.
A secure and reliable internet connection during the conference should be available for the participants. 2.4
A report with the data collected from the questionnaire will be sent to all the participants one month after the conference. The results of the survey will be made available by the end of May, 2014 on the REFSQ’14 homepage, and a description of the results might be published in the workshop proceedings of REFSQ’15. 2.5
Post REFSQ, using our national and international industrial networks, the questionnaire would be forwarded to stakeholders who might not be covered by REFSQ, e.g. regulators. We have planned a case study involving interviews with experts (e.g. developers, safety and security analysts) from a safety critical domain, which will concretise and detail some of the findings from the survey. In this way, survey will contribute to our on-going research on improving traceability approaches to safety critical systems development. 2.6
Low response and completion rate: Low response rate and low questionnaire completion rate are typical concerns of surveys. Some of the participants might require more than 15 minutes to completing the survey. Moreover, the sampling frame (i.e. participants from REFSQ) might not represent all types of stakeholders involved in development projects. We will mitigate these by promoting the survey during the conference and by keeping the survey open for participation also after the conference. We will also use other venues and our industrial networks.
Construct validity: To mitigate the threats related to construct validity, the questions will be carefully drafted based on our experiences as well as our theoretical knowledge. The survey was pre-tested for completeness and ambiguity. In order to avoid participants’ misunderstanding the questions we will supply a glossary of terms. External validity: Apart from performing the survey with REFSQ participants, the survey will be performed with other types of participants, especially with the contacts in our national and international industrial networks. This will give us a better base for generalizing the results of the survey at REFSQ.
Internal validity: no cause – effect relationships are implied Reliability and consistency: this has two components – will the participants give their true opinion and will another survey give the same result? The answer to this question will, at least partly, depend on how the questions in the survey are interpreted by the respondents. When you ask for somebody’s opinion based on practical experiences there are at least four possible bases for the answers. They can be based on the respondent’s general experience, the worst case, the most successful case or the latest case. In addition, we know that the respondents’ answers will vary with the degree of concreteness of the questions. Concrete questions will bring out the respondents’ opinion while general questions will get an answer that is considered “correct” in the respondent’s environment. Using ratings scale (Cronbach’s alpha estimate) and specific questions referring to their opinions or experiences we will address some of the reliability issues. 2.7
Preparation of questionnaire and publishing online Promotion and recruitment Perform survey Data analysis Preparation of report Dissemination – to participants, REFSQ Post REFSQ activities
This work is a part of PhD projects carried out in collaboration between the Institute for Energy Technology/Halden Reactor Project and Norwegian University of Science and Technology. We thank our colleagues, Christian Raspotnig and André Hauge, at the Institute for Energy Technology for pre-testing the questionnaire.