=Paper=
{{Paper
|id=Vol-1342/resacs5
|storemode=property
|title=Context Uncertainty in Requirements Engineering Definition of a Search Strategy for a Systematic Review and Preliminary Result
|pdfUrl=https://ceur-ws.org/Vol-1342/04-resacs.pdf
|volume=Vol-1342
|dblpUrl=https://dblp.org/rec/conf/refsq/BringsSS15
}}
==Context Uncertainty in Requirements Engineering Definition of a Search Strategy for a Systematic Review and Preliminary Result==
https://ceur-ws.org/Vol-1342/04-resacs.pdf
Context Uncertainty in Requirements Engineering:
Definition of a Search Strategy for a Systematic Review
and Preliminary Results
Jennifer Brings, Andrea Salmon, Selda Saritas
paluno – The Ruhr Institute for Software Technology
University of Duisburg-Essen, Essen, Germany
{jennifer.brings,andrea.salmon,selda.saritas}@paluno.uni-due.de
Abstract. [Context and motivation] Cyber-physical systems (CPS) and self-
adaptive systems (SAS) strongly rely on the context they are operating in and
need to adapt their behavior at run-time based on contextual information.
Therefore, it is challenging to completely predict the context of such systems
for their entire operating time already at design time. [Question/problem]
Since several approaches dealing with uncertainty have been proposed for
different research and problem domains in recent years, some might provide
valuable insights for the engineering of CPS or SAS in uncertain contexts.
However, there is no study so far that provides an overview of them. [Principle
ideas/results] Thus, we aim at conducting a systematic literature analysis to
create a research landscape of approaches coping with context uncertainty.
[Contribution] We manually searched one journal and the proceedings of two
conferences in the requirements engineering field to determine and evaluate the
adequateness of search strings to be used in an automated search. In doing so,
we can furthermore present preliminary findings from the manual search for
uncertainty in the requirements engineering field.
Keywords: Uncertainty, Context, Requirements Engineering, Cyber-Physical
Systems, Self-Adaptive Systems, Systematic Review, Search Strategy, System-
atic Literature Search.
1 Introduction
During requirements engineering it is important to consider the context, into which
the system under development will be deployed [1]. Hence, the context of a system
should be documented explicitly at design time (e.g., [2]). This, however, can be chal-
lenging for CPS and SAS, whose contexts might not be completely predictable at
design time, as is, for example, the case for long-living systems such as embedded
systems in aircraft, which will have to interact with newer versions of other systems
in the future [3]. Therefore, the engineering process must cope with a high degree of
uncertainty.
Copyright © 2015 by the authors. Copying permitted for private and academic purposes.
This volume is published and copyrighted by its editors.
171
In the past couple of years, several approaches for dealing with uncertainty have
been proposed for different research and problem domains. As these approaches
might provide valuable insights for the engineering of CPS or SAS in uncertain
contexts, it is important to provide an overview of existing research.
In this paper we present the development of a search strategy for a systematic re-
view on uncertainty. We particular focus on the development of a search string for an
automated search and present preliminary results from the field of requirements engi-
neering. The remainder of the paper is structured as follows: Section 2 briefly dis-
cusses related work regarding the definition of systematic search strategies. In Section
3 the process of our systematic search is introduced. Section 4 presents our prelimi-
nary findings, and Section 5 concludes the paper and gives an outline of future work.
2 Related Work
Systematic reviews, such as mapping studies [4] and systematic literature reviews
[5], have proven useful to review the state of the art with respect to a certain topic and
to derive research landscapes, since they are less biased and result in more reliable
findings than ad-hoc reviews [6]. One example for a systematic review is provided by
Yang et al. in [7]. The authors present a systematic literature review of requirements
modeling and analysis for SAS, where they also categorize papers with respect to
uncertainty. While the approach of Yang et al. can be seen as a basis for our study, we
aim at focusing on context uncertainty in a more fine-grained sense. In addition, the
final study shall not only be restricted to the field of requirements engineering, as
techniques from other fields might be transferable to requirements engineering as
well.
One important aspect of systematic reviews is the definition of a search strategy
[8], i.e. defining which approach will be used to find the relevant literature. There are
three main approaches for finding relevant papers: automated search, which uses pre-
defined search strings on selected search engines, manual search of selected proceed-
ings and journals, and snowball search, where the references of relevant papers are
searched [8]. These approaches are often combined to reduce the number of missed
papers. To conduct an automated search it is necessary to define a search string that
will be applied to the selected search engines. This search string should filter out as
many irrelevant papers as possible without filtering out any relevant papers.
Zhang et al. [8] propose the quasi-gold standard, a set of all relevant papers for a
limited time span and a limited number of publication venues, which have previously
been identified by manual search. The quality of a search string can, thus, be assessed
by using it for an automated search that is limited to the same time span and publica-
tion venues, and calculating the sensitivity (i.e. the ratio between the relevant papers
found and the total number of relevant papers found) and the precision (i.e. the ratio
between the number of relevant papers found and the total number of papers found).
172
3 Systematic Literature Search Strategy
Our systematic search process can be divided into four steps. First, we manually
searched the proceedings of the two main requirements engineering conferences: the
International Requirements Engineering Conference (RE) and the International
Working Conference on Requirements Engineering: Foundation for Software Quality
(REFSQ) and the Requirements Engineering Journal (REJ) from 2010 to 2014 for
articles on uncertainty. Not only did we search for papers that deal with context un-
certainty, we moreover searched for papers that deal with uncertainty in general to
also find approaches that could be adaptable to deal with context uncertainty. The
manual search found the 23 papers presented in Table 1.
Table 1. Selected Papers
Venue Selected Papers
REFSQ [9], [10], [11], [12], [13], [14], [15], [16]
RE [17], [18], [19], [20], [21], [22], [23], [24], [25], [26]
REJ [27], [28], [29], [30], [31]
In the second step, we derived search terms for an automated search by analyzing
title, abstract, and keywords of the selected papers to find the most frequently used
words (excluding common words such as ‘a’ or ‘is’ and unspecific terms such as sys-
tem or approach). Table 2 shows the 10 most frequently used words and their fre-
quency.
Table 2. Most Frequent Words in Title, Abstract, and Keywords
Word Frequency
requirement(s) 133
uncertainty 59
adaptive 47
model(s) 46
self 25
engineering 22
result(s) 20
goal 18
monitoring 18
analysis 17
Search strings were then constructed in the third step, starting with the second most
frequently used word ‘uncertainty’ and then using the Boolean “OR” to add further
terms in order of their frequency until we reached acceptable sensitivity. Subsequent-
ly, further terms where added using the Boolean “AND” in an attempt to increase the
precision without sacrificing too much sensitivity (see Table 3). The most frequent
term ‘requirement(s)’ was not considered for two reasons. First, because the manual
173
search was limited to requirements engineering publications; thus, making it obvious
that the term ‘requirement(s)’ will appear in almost every paper, and second, because
we do not want to limit future searches to the requirements engineering field, as oth-
ers might provide approaches that could be transferable to requirements engineering.
In the fourth and final step the derived search strings were evaluated with respect
to their adequateness for an automated search. As Table 3 shows, the search for the
term ‘uncertainty’ resulted in a sensitivity of 78.26% and a precision of 27.27%. Ac-
cording to Zhang et al. [8], an acceptable sensitivity is reached at 72%, and an appro-
priate precision is reached at 15%. Even though acceptable, this search string still
missed five papers. Hence, we tried to increase the sensitivity by adding the second
search term ‘adaptive’. This search found all papers in the quasi-gold standard, in-
creasing the sensitivity to 100%, while lowering the precision to 24.21%. The search
term ‘adaptive’ alone had a sensitivity of only 60.87%, thus making it unsuitable. The
sensitivity values for the second search string “uncertainty OR adaptive” are more
satisfactory than the values of the first one but the precision is lower, which makes the
search more time-consuming. In an effort to increase the precision we tried adding,
for example, the fourth most frequently used word to the search string using the Bool-
ean “AND”. While this did increase the precision slightly it also reduced the sensitivi-
ty by 25%, which means the search missed a quarter of the relevant papers. We there-
fore deem the search string “uncertainty OR adaptive” the most appropriate.
Table 3. Search String Evaluation
Search String Sensitivity Precision
uncertainty 78.26% 27.27%
uncertainty OR adaptive 100% 24.21%
adaptive 60.87% 29.17%
(uncertainty OR adaptive) AND model 78.26% 21.95%
Fig. 1 further illustrates the ratios between the relevant papers missed (light gray),
the relevant papers found (shaded), and the irrelevant papers found (dark gray) by
automated search for the four search strings and also the ratios between the quasi-gold
standard (all papers found by manual search) and the (relevant and irrelevant) results
from the automated search.
Results by automated search
uncertainty 5 18 48
Quasi-Gold Standard
uncertainty OR adaptive 23 72
adaptive 9 14 34
(uncertainty OR adaptive) AND model 5 18 64
Fig. 1. Relationship between Quasi-Gold Standard and Automated Search Results
174
4 Preliminary Results
This section gives insight into preliminary results gained from the papers published
at RE, REFSQ, and in the REJ between 2010 and 2014. Fig. 2 depicts the annual
publication volume for the years 2010 to 2014, which shows some fluctuations but no
clear trend for those years.
8
6
RE
REFSQ
4
RE Journal
2 Total
0
2010 2011 2012 2013 2014
Fig. 2. Annual Publication Volume
We furthermore categorized the papers by the type of their main contribution, dis-
tinguishing between problem statement, solution proposal, evaluation, and tool. The
findings show that most papers dealing with uncertainty propose solutions for current
challenges (19 out of 23) or described a problem (4 out of 23). Yet, there were no
papers whose main contribution was clearly an evaluation of a solution or a tool
presentation, indicating that the problem of uncertainty is well understood and being
solved but also showing a lack of evaluations regarding the feasibility and usefulness
of the proposed solutions.
Additionally, the papers were categorized based on the life cycle phase (run-time
or design-time) and the type of uncertainty the approaches deal with (uncertain con-
text or uncertainties within one context). Note, that some papers were sorted into more
than one category for each facet, while others, which did not provide the necessary
information, were sorted into not specified categories. As Fig. 3 shows most ap-
proaches deal with uncertainties within the context at design-time, very few with un-
certain contexts at run-time, and none with uncertain contexts at design-time. Overall,
more of the approaches are concerned with the run-time phase as opposed to the de-
sign-time phase.
175
Life Cycle
Phase
Not Specified 1
Design-Time 9 1
Run-Time 4 7 2
Type of
Uncertainty
Uncertain Uncertainty Not Specified
Context Within One
Context
Fig. 3. Mapping Results
A frequency analysis of the keywords chosen by the papers’ authors highlighted
the importance of uncertainty for self-adaptive systems in particular (see Fig. 4 for all
keywords that were used in more than one paper). The frequency analysis also re-
vealed a relatively high number of papers about goal models.
0 1 2 3 4 5
Self-Adaptive Systems
Requirements Engineering
Uncertainty
Requirements
Goal Models
Requirements Monitoring
Self-Adaptation
Adaptive Systems
Fig. 4. Most Frequent Keywords
5 Conclusion and Future Work
In this paper, we presented preliminary results from a literature review on uncer-
tainty in requirements engineering. The literature search was conducted manually for
a limited time span and limited publication venues. Based on titles, abstracts, and
keywords we derived search terms and evaluated their performance. In the future we
are planning to use this for a more extended systematic review on uncertainty.
176
Acknowledgements. This research was funded by the German Federal Ministry of
Education and Research (grant no. 01IS12005C).
References
1. Gause, D.C.: Why Context Matters-And What Can We Do about It? IEEE Softw.
22, 13–15 (2005)
2. Daun, M., Tenbergen, B., Weyer, T.: Requirements Viewpoint. In: Pohl, K.,
Hönninger, H., Achatz, R., Broy, M. (eds.) Model-Based Engineering of Embed-
ded Systems, The SPES 2020 Methodology, pp. 51–68. Springer (2012)
3. Daun, M., Tenbergen, B., Brings, J., Weyer, T.: Documenting Assumptions about
the Operational Context of Long-Living Collaborative Embedded Systems. In:
2nd Collaborative Workshop on Evolution and Maintenance of Long-Living
Software Systems (EMLS 2015). in press (2015)
4. Petersen, K., Feldt, R., Mujtaba, S., Mattsson, M.: Systematic Mapping Studies in
Software Engineering. In: Proc. of the Int. Conf. on Evaluation and Assessment in
Software Engineering, pp. 68–77 (2008)
5. Kitchenham, B., Charters, S.: Guidelines for performing Systematic Literature
Reviews in Software Engineering (2007)
6. Kitchenham, B., Brereton, P.: A systematic review of systematic review process
research in software engineering. Information and Software Technology 55,
2049–2075 (2013)
7. Yang, Z., Li, Z., Jin, Z., Chen, Y.: A Systematic Literature Review of Require-
ments Modeling and Analysis for Self-adaptive Systems. In: Proc. of REFSQ, pp.
55–71 (2014)
8. Zhang, H., Babar, M.A., Tell, P.: Identifying relevant studies in software engi-
neering. Information and Software Technology 53, 625–637 (2011)
9. Welsh, K., Sawyer, P.: Understanding the Scope of Uncertainty in Dynamically
Adaptive Systems. In: Proc. of REFSQ, pp. 2–16 (2010)
10. Oriol, M., Qureshi, N.A., Franch, X., Perini, A., Marco, J.: Requirements Moni-
toring for Adaptive Service-Based Applications. In: Proc. of REFSQ, pp. 280–
287 (2012)
11. Qureshi, N.A., Jureta, I.J., Perini, A.: Towards a Requirements Modeling Lan-
guage for Self-Adaptive Systems. In: Proc. of REFSQ, pp. 263–279 (2012)
12. Bencomo, N., Belaggoun, A.: Supporting Decision-Making for Self-Adaptive
Systems: From Goal Models to Dynamic Decision Networks. In: Proc. of
REFSQ, pp. 221–236 (2013)
13. Almaliki, M., Faniyi, F., Bahsoon, R., Phalp, K., Ali, R.: Requirements-Driven
Social Adaptation: Expert Survey. In: Proc. of REFSQ, pp. 72–87 (2014)
14. Kamsties, E., Kneer, F., Voelter, M., Igel, B., Kolb, B.: Feedback-Aware Re-
quirements Documents for Smart Devices. In: Proc. of REFSQ, pp. 119–134
(2014)
177
15. Ojameruaye, B., Bahsoon, R.: Systematic Elaboration of Compliance Require-
ments Using Compliance Debt and Portfolio Theory. In: Proc. of REFSQ, pp.
152–167 (2014)
16. Vierhauser, M., Rabiser, R., Grünbacher, P.: A Requirements Monitoring Infra-
structure for Very-Large-Scale Software Systems. In: Proc. of REFSQ, pp. 88–94
(2014)
17. Al-Emran, A., Pfahl, D., Ruhe, G.: Decision Support for Product Release Plan-
ning Based on Robustness Analysis. In: Proc. of RE, pp. 157–166 (2010)
18. Yang, H., De Roeck, A., Gervasi, V., Willis, A., Nuseibeh, B.: Speculative Re-
quirements: Automatic Detection of Uncertainty in Natural Language Require-
ments. In: Proc. of RE, pp. 11–20 (2012)
19. Welsh, K., Bencomo, N.: Run-Time Model Evaluation for Requirements Model-
Driven Self-Adaptation. In: Proc. of RE, pp. 329–330 (2012)
20. Welsh, K., Sawyer, P., Bencomo, N.: Run-Time Resolution of Uncertainty. In:
Proc. of RE, pp. 355–356 (2011)
21. Tran, L.M.S., Massacci, F.: An Approach for Decision Support on the Uncertain-
ty in Feature Model Evolution. In: Proc. of RE, pp. 93–102 (2014)
22. Nolan, A.J., Abrahao, S., Clements, P., Pickard, A.: Managing requirements un-
certainty in engine control systems development. In: Proc. of RE, pp. 259–264
(2011)
23. Salay, R., Chechik, M., Horkoff, J.: Managing requirements uncertainty with
partial models. In: Proc. of RE, pp. 1–10 (2012)
24. Sawyer, P., Bencomo, N., Whittle, J., Letier, E., Finkelstein, A.: Requirements-
Aware Systems: A Research Agenda for RE for Self-adaptive Systems. In: Proc.
of RE, pp. 95–103 (2010)
25. Horkoff, J., Salay, R., Chechik, M., Di Sandro, A.: Supporting early decision-
making in the presence of uncertainty. In: Proc. of RE, pp. 33–42 (2014)
26. Qureshi, N.A., Perini, A.: Requirements Engineering for Adaptive Service Based
Applications. In: Proc. of RE, pp. 108–111 (2010)
27. Ali, R., Dalpiaz, F., Giorgini, P.: A goal-based framework for contextual re-
quirements modeling and analysis. Requirements Eng 15, 439–458 (2010)
28. Dalpiaz, F., Giorgini, P., Mylopoulos, J.: Adaptive socio-technical systems: a
requirements-based approach. Requirements Eng 18, 1–24 (2013)
29. Katina, P.F., Keating, C.B., Jaradat, R.M.: System requirements engineering in
complex situations. Requirements Eng 19, 45–62 (2014)
30. Salay, R., Chechik, M., Horkoff, J., Di Sandro, A.: Managing requirements uncer-
tainty with partial models. Requirements Eng 18, 107–128 (2013)
31. Whittle, J., Sawyer, P., Bencomo, N., Cheng, Betty H. C., Bruel, J.-M.: RELAX:
a language to address uncertainty in self-adaptive systems requirement. Require-
ments Eng 15, 177–196 (2010)
178