=Paper=
{{Paper
|id=Vol-1138/et3
|storemode=property
|title=Survey Protocol: Traceability during Development of Systems with Safety and Security Implications - Importance, Tools, and Challenges
|pdfUrl=https://ceur-ws.org/Vol-1138/et3.pdf
|volume=Vol-1138
}}
==Survey Protocol: Traceability during Development of Systems with Safety and Security Implications - Importance, Tools, and Challenges
==
https://ceur-ws.org/Vol-1138/et3.pdf
Survey Protocol: Traceability during Development of
Systems with Safety and Security Implications -
Importance, Tools, and Challenges
Vikash Katta*†, Tor Stålhane†
*
Institute for Energy Technology, Halden, Norway
†
Norwegian University of Science and Technology, Trondheim, Norway
vikash.katta@hrp.no, stalhane@idi.ntnu.no
Abstract. [Context and motivation] Although traceability plays an im-
portant role in system development projects, it is often neglected or poorly per-
formed. Stakeholders involved in system development projects have different
use of traces, and therefore the required traceability and its implementation de-
pends upon the stakeholders involved in the projects. [Problem/question] Tra-
ditional ways of implementing traceability using traceability matrices or office
tools such Excel sheets are not effective, especially while developing large and
complex systems. There exists several requirements management tools, but they
have to be extended or tailored to achieve the traceability needed of the stake-
holders, requiring human effort and resources. Little is known or has been re-
ported on traceability usage, practises and challenges in the development of sys-
tems with safety and security implications. Such a knowledge and understand-
ing is needed to develop cost-effective traceability solutions as well as to high-
light the importance of traceability [Principal ideas] This paper presents a pro-
posal for performing a survey – online questionnaire – with the participants of
REFSQ’14 to collect and report their experiences on traceability. [Contribu-
tion] Since, REFSQ’14 participants have a diverse background, i.e. experiences
with different roles and responsibilities in projects from industry, the survey
will be a unique opportunity to collect and report data on traceability covering
several stakeholder viewpoints.
1 Introduction
The aim of the survey is to investigate the purpose and uses of traceability during
system development projects, identify the state of practice, and understand the chal-
lenges when implementing traceability during development projects. It would be ben-
eficial if the survey participants have experiences in development of systems with
safety and quality consequences (for e.g. loss of life or mission or equipment, eco-
nomic implications).
1.1 Problem definition
Traceability is vital and should be implemented during development projects, espe-
cially for the development of systems with safety and other quality (e.g. security)
implications where standards or regulatory bodies require implementation of tracea-
bility. However, when it comes to industrial practices, traceability is often neglected
or not implemented in a satisfactory manner. In our experiences from both safety and
non-safety industries, traceability matrices and Office tools (e.g. Excel sheets) are still
widely used for implementing traceability. Such approaches to traceability practices
are not viable, especially not for projects on developing large and complex systems.
There are several challenges related to implementing traceability including lack of
tools that are suitable for project needs and stakeholder needs, and additional cost and
effort needed to maintain traceability [1,2,3]. Stakeholders involved in system devel-
opment projects have different traceability needs (i.e. different use of traces), and the
required traceability and its implementation therefore depend on the stakeholders
involved in the projects. For example, in projects on development of safety systems, a
regulator will use traces to verify compliance of requirements from regula-
tions/standards during system development, whereas a system designer will use traces
to verify whether the system design caters to all the safety requirements. As stated in
[1]:
“Currently, there is poor understanding of what people need traceability for and how people ac-
tually use traceability over time. Further, traceability will not be created or maintained effectively if
the required tasks to do so are themselves not understood and supported. Currently, there is poor un-
derstanding of what individuals and teams need to do to create and maintain traces. This distinction
between satisfying the requirements of those stakeholders who establish traceability and those stake-
holders who use traceability lies at the heart of many traceability problems, for these roles are not
necessarily overlapping.”
Based on our experiences, this problem is also evident in development projects of
safety systems or systems with security implications. There is a lack of traceability
tools that can be used to address the needs of the stakeholders involved in such devel-
opment projects [4,5,6]. In this regard, the RE community need to report on the re-
search and experiences on the importance as well as the challenges of traceability.
1.2 Context of the survey
The planned survey is a part of a PhD project on improving traceability practices
during the development of safety critical systems. We have performed a narrative
literature study and proposed solutions – mostly focusing on the air traffic manage-
ment domain [4,5,6]. So far, our proposed solutions aim to provide traceability sup-
port for certain tasks performed by the stakeholders, especially the safety analyst and
the safety case author, participating in the development of safety critical systems.
However, we need to broaden the scope of the profile of stakeholders and generalize
the traceability needs and challenges of the stakeholders involved in other roles, activ-
ities, projects and domains.
2 Goal, Research Questions and Research Design
2.1 Goal
The primary goal of the survey is to investigate the purpose and uses of traceability
during system development projects, identify the state of practice, and understand the
challenges on implementing traceability during development projects.
The survey is mainly intended for the REFSQ participants who have experiences in
projects on development of systems with safety and security implications. Since
REFSQ attracts both practitioners and researchers in the area of requirements engi-
neering and other system engineering activities, the variation of the profile of the
participants – their roles and tasks in projects - will be large. Therefore, the survey
will gather information related to several stakeholders and their traceability needs.
2.2 Research questions
The survey addresses the following research questions:
1. What is the purpose of traceability in projects developing safety critical systems
and systems with security implications? This question looks into the purposes –
tasks, activities etc. – for which stakeholders use traces.
2. What are the practises of performing traceability? This question looks into identi-
fying the techniques and tools being used by the stakeholders and their organisa-
tions for implementing traceability in their projects.
3. What are the challenges of implementing traceability? This question looks into the
challenges the stakeholders face while implementing traceability.
2.3 Research design and method
We follow a non-experimental strategy, using the survey as a method to gather data
from a large population. The purpose of the survey is both exploratory and descriptive
on traceability uses and challenges.
Subjects
The survey will be open to the participants of the REFSQ’14 conference. We aim
for the participants who have experiences in creating and using traceability infor-
mation during activities of a development project. Experiences in development of
safety critical systems or systems with security implications will be beneficial. We
consider participants with experiences in RE tasks, system designing, testing, V&V,
quality assessment, qualification, project management.
Recruitment and Incentives
We expect the REFSQ organisers to inform the conference participants about the
survey. If required, we will provide organisers with information (slides etc.) on the
importance of the topic covered by the survey, which could be used by the organisers
to encourage participation.
If allowed, we will have a poster promoting the survey to the conference partici-
pants. We plan to offer four Amazon gift cards, each worth 50 euros, which can be
used by four participants (selected by lottery) to purchase books of his or her choice.
Consent of participation
Filling in the online questionnaire will be treated as participant’s consent to partic-
ipate. Participants will be made of this before filling up the questionnaire.
Confidentiality
The data collected through the survey will be kept confidential and will be stored
securely in our company’s internal server, and will be deleted after completion of the
survey related activities. The result of the survey will be anonymised.
Implementation
1. The survey is available for the participants at the URL
http:// refsqsurvey.limequery.com/index.php/726296/lang-en
2. The questionnaire has been organized into the following parts:
(a) Introduction: consent of participation, confidentiality and guidelines for com-
pleting the questionnaire.
(b) Questions on the participants’ work experiences – their roles and responsibili-
ties in relevant projects.
(c) Questions on whether they have created or used traceability during develop-
ments, and if so, for which development activities (e.g. change impact analysis,
V&V, and quality assessments) they have used traces.
(d) Questions on what type of traceability techniques and tools the participants
have used.
(e) Questions on challenges the participants have experienced, e.g. lack of under-
standing, tools, and cost, while implementing traceability in the projects.
3. The questions has multiple choices, rating scales, and open-ended formats.
4. It is expected that the participant will be able to complete the survey within 15
minutes.
5. The survey was pre-tested with two colleagues with relevant work experiences.
6. The survey will be conducted during REFSQ’14.
Data analysis
Depending upon the format of the question, we will use analysis methods such as
statistical tests for the multiple choice and rating scales and affinity diagrams and
content analysis for the open ended questions.
Equipment
A secure and reliable internet connection during the conference should be available
for the participants.
2.4 Dissemination
A report with the data collected from the questionnaire will be sent to all the partic-
ipants one month after the conference. The results of the survey will be made availa-
ble by the end of May, 2014 on the REFSQ’14 homepage, and a description of the
results might be published in the workshop proceedings of REFSQ’15.
2.5 Post REFSQ’14
Post REFSQ, using our national and international industrial networks, the ques-
tionnaire would be forwarded to stakeholders who might not be covered by REFSQ,
e.g. regulators. We have planned a case study involving interviews with experts (e.g.
developers, safety and security analysts) from a safety critical domain, which will
concretise and detail some of the findings from the survey. In this way, survey will
contribute to our on-going research on improving traceability approaches to safety
critical systems development.
2.6 Threats to validity and reliability
Low response and completion rate: Low response rate and low questionnaire comple-
tion rate are typical concerns of surveys. Some of the participants might require more
than 15 minutes to completing the survey. Moreover, the sampling frame (i.e. partici-
pants from REFSQ) might not represent all types of stakeholders involved in devel-
opment projects. We will mitigate these by promoting the survey during the confer-
ence and by keeping the survey open for participation also after the conference. We
will also use other venues and our industrial networks.
Construct validity: To mitigate the threats related to construct validity, the questions
will be carefully drafted based on our experiences as well as our theoretical
knowledge. The survey was pre-tested for completeness and ambiguity. In order to
avoid participants’ misunderstanding the questions we will supply a glossary of terms.
External validity: Apart from performing the survey with REFSQ participants, the
survey will be performed with other types of participants, especially with the contacts
in our national and international industrial networks. This will give us a better base
for generalizing the results of the survey at REFSQ.
Internal validity: no cause – effect relationships are implied
Reliability and consistency: this has two components – will the participants give their
true opinion and will another survey give the same result? The answer to this question
will, at least partly, depend on how the questions in the survey are interpreted by the
respondents. When you ask for somebody’s opinion based on practical experiences
there are at least four possible bases for the answers. They can be based on the re-
spondent’s general experience, the worst case, the most successful case or the latest
case. In addition, we know that the respondents’ answers will vary with the degree of
concreteness of the questions. Concrete questions will bring out the respondents’
opinion while general questions will get an answer that is considered “correct” in the
respondent’s environment. Using ratings scale (Cronbach’s alpha estimate) and spe-
cific questions referring to their opinions or experiences we will address some of the
reliability issues.
2.7 Timetable
A preliminary timetable of the survey is provided below.
Table 1. Survey activities and completion dates
Activities Completion date
Preparation of questionnaire and publishing online April 07, 2014
Promotion and recruitment April 07-10, 2014
Perform survey April 07-10, 2014
Data analysis April 25, 2014
Preparation of report May 02, 2014
Dissemination – to participants, REFSQ May 07, 2014
Post REFSQ activities No concrete dates
Acknowledgements
This work is a part of PhD projects carried out in collaboration between the Insti-
tute for Energy Technology/Halden Reactor Project and Norwegian University of
Science and Technology. We thank our colleagues, Christian Raspotnig and André
Hauge, at the Institute for Energy Technology for pre-testing the questionnaire.
References
1. Gotel, O., et al.: The Grand Challenge of Traceability (v1.0). Software and Systems Trace-
ability, pp. 343-409, Springer (2012)
2. Winkler, S., Pilgrim, J. V.: A Survey of Traceability in Requirements Engineering and
Model-Driven Development. Software and Systems Modeling, vol. 9(4), pp. 529-565,
Springer Berlin: (2010)
3. Knethen A. V., Paech, B.: A Survey on Tracing Approaches in Practice and Research. Re-
search Report, IESE-Report, 095.01/E, Fraunhofer IESE, Kaiserslautern (2002)
4. Katta, V., Stålhane, T.: Traceability of Safety Systems: Approach, Meta-Model and Tool
Support. Technical report HWR-1053, OECD Halden Reactor Project (2013), Available
upon request.
5. Katta, V., Raspotnig, C., Karpati, P., Stålhane, T.: Requirements Management in a Com-
bined Process for Safety and Security Assessments. In: Proceedings 2013 International
Conference on Availability, Reliability and Security (2013)
6. Katta, V., Raspotnig, C., Stålhane, T.: Presenting A Traceability Based Approach For
Safety Argumentation. In: European Safety and Reliability conference ( 2013)