Context Uncertainty in Requirements Engineering: Definition of a Search Strategy for a Systematic Review and Preliminary Results

Abstract

[Context and motivation] Cyber-physical systems (CPS) and selfadaptive systems (SAS) strongly rely on the context they are operating in and need to adapt their behavior at run-time based on contextual information. Therefore, it is challenging to completely predict the context of such systems for their entire operating time already at design time. [Question/problem] Since several approaches dealing with uncertainty have been proposed for different research and problem domains in recent years, some might provide valuable insights for the engineering of CPS or SAS in uncertain contexts. However, there is no study so far that provides an overview of them. [Principle ideas/results] Thus, we aim at conducting a systematic literature analysis to create a research landscape of approaches coping with context uncertainty. [Contribution] We manually searched one journal and the proceedings of two conferences in the requirements engineering field to determine and evaluate the adequateness of search strings to be used in an automated search. In doing so, we can furthermore present preliminary findings from the manual search for uncertainty in the requirements engineering field.

Keywords

• Uncertainty
• Context
• Requirements Engineering
• Cyber-Physical Systems
• Self-Adaptive Systems
• Systematic Review
• Search Strategy
• Systematic Literature Search

-

During requirements engineering it is important to consider the context, into which the system under development will be deployed [ 1 ]. Hence, the context of a system should be documented explicitly at design time (e.g., [ 2 ]). This, however, can be challenging for CPS and SAS, whose contexts might not be completely predictable at design time, as is, for example, the case for long-living systems such as embedded systems in aircraft, which will have to interact with newer versions of other systems in the future [ 3 ]. Therefore, the engineering process must cope with a high degree of uncertainty.

In the past couple of years, several approaches for dealing with uncertainty have been proposed for different research and problem domains. As these approaches might provide valuable insights for the engineering of CPS or SAS in uncertain contexts, it is important to provide an overview of existing research.

In this paper we present the development of a search strategy for a systematic review on uncertainty. We particular focus on the development of a search string for an automated search and present preliminary results from the field of requirements engineering. The remainder of the paper is structured as follows: Section 2 briefly discusses related work regarding the definition of systematic search strategies. In Section 3 the process of our systematic search is introduced. Section 4 presents our preliminary findings, and Section 5 concludes the paper and gives an outline of future work. 2

Related Work

Systematic reviews, such as mapping studies [ 4 ] and systematic literature reviews [ 5 ], have proven useful to review the state of the art with respect to a certain topic and to derive research landscapes, since they are less biased and result in more reliable findings than ad-hoc reviews [ 6 ]. One example for a systematic review is provided by Yang et al. in [ 7 ]. The authors present a systematic literature review of requirements modeling and analysis for SAS, where they also categorize papers with respect to uncertainty. While the approach of Yang et al. can be seen as a basis for our study, we aim at focusing on context uncertainty in a more fine-grained sense. In addition, the final study shall not only be restricted to the field of requirements engineering, as techniques from other fields might be transferable to requirements engineering as well.

One important aspect of systematic reviews is the definition of a search strategy [ 8 ], i.e. defining which approach will be used to find the relevant literature. There are three main approaches for finding relevant papers: automated search, which uses predefined search strings on selected search engines, manual search of selected proceedings and journals, and snowball search, where the references of relevant papers are searched [ 8 ]. These approaches are often combined to reduce the number of missed papers. To conduct an automated search it is necessary to define a search string that will be applied to the selected search engines. This search string should filter out as many irrelevant papers as possible without filtering out any relevant papers.

Zhang et al. [ 8 ] propose the quasi-gold standard, a set of all relevant papers for a limited time span and a limited number of publication venues, which have previously been identified by manual search. The quality of a search string can, thus, be assessed by using it for an automated search that is limited to the same time span and publication venues, and calculating the sensitivity (i.e. the ratio between the relevant papers found and the total number of relevant papers found) and the precision (i.e. the ratio between the number of relevant papers found and the total number of papers found).

Systematic Literature Search Strategy

Our systematic search process can be divided into four steps. First, we manually searched the proceedings of the two main requirements engineering conferences: the International Requirements Engineering Conference (RE) and the International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ) and the Requirements Engineering Journal (REJ) from 2010 to 2014 for articles on uncertainty. Not only did we search for papers that deal with context uncertainty, we moreover searched for papers that deal with uncertainty in general to also find approaches that could be adaptable to deal with context uncertainty. The manual search found the 23 papers presented in Table 1.

In the second step, we derived search terms for an automated search by analyzing title, abstract, and keywords of the selected papers to find the most frequently used words (excluding common words such as ‘a’ or ‘is’ and unspecific terms such as system or approach). Table 2 shows the 10 most frequently used words and their frequency.

Search strings were then constructed in the third step, starting with the second most frequently used word ‘uncertainty’ and then using the Boolean “OR” to add further terms in order of their frequency until we reached acceptable sensitivity. Subsequently, further terms where added using the Boolean “AND” in an attempt to increase the precision without sacrificing too much sensitivity (see Table 3). The most frequent term ‘requirement(s)’ was not considered for two reasons. First, because the manual search was limited to requirements engineering publications; thus, making it obvious that the term ‘requirement(s)’ will appear in almost every paper, and second, because we do not want to limit future searches to the requirements engineering field, as others might provide approaches that could be transferable to requirements engineering.

In the fourth and final step the derived search strings were evaluated with respect to their adequateness for an automated search. As Table 3 shows, the search for the term ‘uncertainty’ resulted in a sensitivity of 78.26% and a precision of 27.27%. According to Zhang et al. [ 8 ], an acceptable sensitivity is reached at 72%, and an appropriate precision is reached at 15%. Even though acceptable, this search string still missed five papers. Hence, we tried to increase the sensitivity by adding the second search term ‘adaptive’. This search found all papers in the quasi-gold standard, increasing the sensitivity to 100%, while lowering the precision to 24.21%. The search term ‘adaptive’ alone had a sensitivity of only 60.87%, thus making it unsuitable. The sensitivity values for the second search string “uncertainty OR adaptive” are more satisfactory than the values of the first one but the precision is lower, which makes the search more time-consuming. In an effort to increase the precision we tried adding, for example, the fourth most frequently used word to the search string using the Boolean “AND”. While this did increase the precision slightly it also reduced the sensitivity by 25%, which means the search missed a quarter of the relevant papers. We therefore deem the search string “uncertainty OR adaptive” the most appropriate. REFSQ RE Journal Total

Preliminary Results

This section gives insight into preliminary results gained from the papers published at RE, REFSQ, and in the REJ between 2010 and 2014. Fig. 2 depicts the annual publication volume for the years 2010 to 2014, which shows some fluctuations but no clear trend for those years.

2010 2011 2012 2013

2014

We furthermore categorized the papers by the type of their main contribution, distinguishing between problem statement, solution proposal, evaluation, and tool. The findings show that most papers dealing with uncertainty propose solutions for current challenges (19 out of 23) or described a problem (4 out of 23). Yet, there were no papers whose main contribution was clearly an evaluation of a solution or a tool presentation, indicating that the problem of uncertainty is well understood and being solved but also showing a lack of evaluations regarding the feasibility and usefulness of the proposed solutions.

Additionally, the papers were categorized based on the life cycle phase (run-time or design-time) and the type of uncertainty the approaches deal with (uncertain context or uncertainties within one context). Note, that some papers were sorted into more than one category for each facet, while others, which did not provide the necessary information, were sorted into not specified categories. As Fig. 3 shows most approaches deal with uncertainties within the context at design-time, very few with uncertain contexts at run-time, and none with uncertain contexts at design-time. Overall, more of the approaches are concerned with the run-time phase as opposed to the design-time phase. Life Cycle

Phase Not Specified Design-Time

Run-Time

Self-Adaptive Systems Requirements Engineering

Uncertainty Requirements

Goal Models Requirements Monitoring

Self-Adaptation Adaptive Systems

4 Uncertain Context 9 7 Uncertainty Within One Context 1 1 2 Not Specified

Type of

Uncertainty

A frequency analysis of the keywords chosen by the papers’ authors highlighted the importance of uncertainty for self-adaptive systems in particular (see Fig. 4 for all keywords that were used in more than one paper). The frequency analysis also revealed a relatively high number of papers about goal models.

0 1 2 3 4 5

In this paper, we presented preliminary results from a literature review on uncertainty in requirements engineering. The literature search was conducted manually for a limited time span and limited publication venues. Based on titles, abstracts, and keywords we derived search terms and evaluated their performance. In the future we are planning to use this for a more extended systematic review on uncertainty. Acknowledgements. This research was funded by the German Federal Ministry of Education and Research (grant no. 01IS12005C).

References

1. Gause , D.C. : Why Context Matters-And What Can We Do about It? IEEE Softw. 22 , 13 - 15 ( 2005 )

2. Daun , M. , Tenbergen , B. , Weyer , T. : Requirements Viewpoint . In: Pohl, K. , Hönninger , H. , Achatz , R. , Broy , M. (eds.) Model-Based Engineering of Embedded Systems , The SPES 2020 Methodology , pp. 51 - 68 . Springer ( 2012 )

3. Daun , M. , Tenbergen , B. , Brings , J. , Weyer , T. : Documenting Assumptions about the Operational Context of Long-Living Collaborative Embedded Systems . In: 2nd Collaborative Workshop on Evolution and Maintenance of Long-Living Software Systems (EMLS 2015 ). in press ( 2015 )

4. Petersen , K. , Feldt , R. , Mujtaba , S. , Mattsson , M. : Systematic Mapping Studies in Software Engineering . In: Proc. of the Int. Conf. on Evaluation and Assessment in Software Engineering , pp. 68 - 77 ( 2008 )

5. Kitchenham , B. , Charters , S. : Guidelines for performing Systematic Literature Reviews in Software Engineering ( 2007 )

6. Kitchenham , B. , Brereton , P.: A systematic review of systematic review process research in software engineering . Information and Software Technology 55 , 2049 - 2075 ( 2013 )

7. Yang , Z. , Li , Z. , Jin , Z. , Chen , Y. : A Systematic Literature Review of Requirements Modeling and Analysis for Self-adaptive Systems . In: Proc. of REFSQ , pp. 55 - 71 ( 2014 )

8. Zhang, H., Babar , M.A. , Tell , P. : Identifying relevant studies in software engineering . Information and Software Technology 53 , 625 - 637 ( 2011 )

9. Welsh , K. , Sawyer , P. : Understanding the Scope of Uncertainty in Dynamically Adaptive Systems . In: Proc. of REFSQ , pp. 2 - 16 ( 2010 )

10. Oriol , M. , Qureshi , N.A. , Franch , X. , Perini , A. , Marco , J.: Requirements Monitoring for Adaptive Service-Based Applications . In: Proc. of REFSQ , pp. 280 - 287 ( 2012 )

11. Qureshi , N.A. , Jureta , I.J. , Perini , A. : Towards a Requirements Modeling Language for Self-Adaptive Systems . In: Proc. of REFSQ , pp. 263 - 279 ( 2012 )

12. Bencomo , N. , Belaggoun , A. : Supporting Decision-Making for Self-Adaptive Systems: From Goal Models to Dynamic Decision Networks . In: Proc. of REFSQ , pp. 221 - 236 ( 2013 )

13. Almaliki , M. , Faniyi , F. , Bahsoon , R. , Phalp , K. , Ali , R.: Requirements-Driven Social Adaptation: Expert Survey . In: Proc. of REFSQ , pp. 72 - 87 ( 2014 )

14. Kamsties , E. , Kneer , F. , Voelter , M. , Igel , B. , Kolb , B. : Feedback-Aware Requirements Documents for Smart Devices . In: Proc. of REFSQ , pp. 119 - 134 ( 2014 )

15. Ojameruaye , B. , Bahsoon , R.: Systematic Elaboration of Compliance Requirements Using Compliance Debt and Portfolio Theory . In: Proc. of REFSQ , pp. 152 - 167 ( 2014 )

16. Vierhauser , M. , Rabiser , R. , Grünbacher , P. : A Requirements Monitoring Infrastructure for Very-Large-Scale Software Systems . In: Proc. of REFSQ , pp. 88 - 94 ( 2014 )

17. Al-Emran , A. , Pfahl , D. , Ruhe , G. : Decision Support for Product Release Planning Based on Robustness Analysis . In: Proc. of RE , pp. 157 - 166 ( 2010 )

18. Yang , H. , De Roeck , A. , Gervasi , V. , Willis , A. , Nuseibeh , B. : Speculative Requirements: Automatic Detection of Uncertainty in Natural Language Requirements . In: Proc. of RE , pp. 11 - 20 ( 2012 )

19. Welsh , K. , Bencomo , N.: Run-Time Model Evaluation for Requirements ModelDriven Self-Adaptation . In: Proc. of RE , pp. 329 - 330 ( 2012 )

20. Welsh , K. , Sawyer , P. , Bencomo , N.: Run-Time Resolution of Uncertainty . In: Proc. of RE , pp. 355 - 356 ( 2011 )

21. Tran , L.M.S. , Massacci , F. : An Approach for Decision Support on the Uncertainty in Feature Model Evolution . In: Proc. of RE , pp. 93 - 102 ( 2014 )

22. Nolan , A.J. , Abrahao , S. , Clements , P. , Pickard , A. : Managing requirements uncertainty in engine control systems development . In: Proc. of RE , pp. 259 - 264 ( 2011 )

23. Salay , R. , Chechik , M. , Horkoff , J.: Managing requirements uncertainty with partial models . In: Proc. of RE , pp. 1 - 10 ( 2012 )

24. Sawyer , P. , Bencomo , N. , Whittle , J. , Letier , E. , Finkelstein , A. : RequirementsAware Systems: A Research Agenda for RE for Self-adaptive Systems . In: Proc. of RE , pp. 95 - 103 ( 2010 )

25. Horkoff , J. , Salay , R. , Chechik , M. , Di Sandro , A. : Supporting early decisionmaking in the presence of uncertainty . In: Proc. of RE , pp. 33 - 42 ( 2014 )

26. Qureshi , N.A. , Perini , A. : Requirements Engineering for Adaptive Service Based Applications . In: Proc. of RE , pp. 108 - 111 ( 2010 )

27. Ali , R. , Dalpiaz , F. , Giorgini , P. : A goal-based framework for contextual requirements modeling and analysis . Requirements Eng 15 , 439 - 458 ( 2010 )

28. Dalpiaz , F. , Giorgini , P. , Mylopoulos , J. : Adaptive socio-technical systems: a requirements-based approach . Requirements Eng 18 , 1 - 24 ( 2013 )

29. Katina , P.F. , Keating , C.B. , Jaradat , R.M.: System requirements engineering in complex situations . Requirements Eng 19 , 45 - 62 ( 2014 )

30. Salay , R. , Chechik , M. , Horkoff , J. , Di Sandro , A. : Managing requirements uncertainty with partial models . Requirements Eng 18 , 107 - 128 ( 2013 )

31. Whittle , J. , Sawyer , P. , Bencomo , N., Cheng, Betty H. C., Bruel , J.-M.: RELAX: a language to address uncertainty in self-adaptive systems requirement . Requirements Eng 15 , 177 - 196 ( 2010 )