An Approach: SysML-based Automated Completeness Evaluation of the System Requirements Specification Jovita Bankauskaite Aurelijus Morkevicius Department of Information Systems Department of Information Systems Kaunas University of Technology Kaunas University of Technology Kaunas, Lithuania Kaunas, Lithuania jovita.bankauskaite@ktu.lt aurelijus.morkevicius@ktu.lt Abstract — Model-Based Systems Engineering (MBSE) is requirements. Poor requirements management is one of the systems engineering methodology that emphasizes the application biggest reasons why 47% of project fail to meet goals [2]. of strict visual modeling principles. Models are created to deal with complexity, they allow to understand an area of interest or In order to avoid issues detection and correction in later concern and provide unambiguous communication amongst stages of development, it is important to identify the issues of interested sides. MBSE improves the quality of models of the incompleteness in the early stages of requirements specification. system by providing the ability to evaluate it for completeness, The mistakes due to incompleteness, inconsistency, and correctness and consistency. MBSE is enabled by Systems ambiguity introduced at the stage of requirements engineering Modeling Language (SysML) that supports the analysis, are difficult and more expensive to correct than those introduced specification, design, verification, validation of complex systems in later stages of system development [3]. The uncertainty of the and is used for modeling system requirements, behavior, SRS completeness causes the risk of the requirements change structure, and parametrics. SysML is not a methodology, nor a during the development process [4]. Completeness and method. In this case, it is necessary to choose a specific method in correctness (C&C) analysis of requirements specification aims combination with system modeling language to comprehensively to eliminate occurred issues. and accurately evaluate the completeness of system requirements specification (SRS). This opens up discussions of how to apply In this paper, we focus on a subset of the C&C task – SysML provided infrastructure to evaluate the system completeness analysis only. We understand the completeness of requirements specification throughout the entire specifying the SRS as atomic requirements coverage by atomic model process of SRS and achieve a high-quality of the SRS. In this elements. The question is how to utilize SysML provided paper, a new approach of how requirements specification, infrastructure to successfully achieve a high quality of the expressed with sufficient precision in SysML can be used for requirements specification: what method to use in combination automated completeness evaluation. with SysML. Keywords—SysML, MBSE Grid, Completeness Metrics, System In this paper, we propose a new approach of how Requirements Specification, Requirements Engineering, MBSE requirements specification that is expressed in SysML in combination with MBSE Grid method can be used for I. INTRODUCTION automated completeness evaluation of the system requirements One of the main objectives of the Model Based Systems specification. Engineering is the improved quality of early identification of The MBSE Grid method guides how to specify principal requirements issues. In order to achieve this goal, the areas of the system model and how to manage different layers of organization needs to implement appropriate practices for abstraction [5]. The MBSE Grid is organized in a matrix view. modeling qualitatively. Nowadays, MBSE is enabled by Rows represent two main viewpoints: one to define the problem Systems Modeling Language. It is used for modeling complex in order to understand it, other to provide one or several systems such as submarines, trains, aircraft, spacecraft, etc. alternative solutions to solve it. Columns represent four main SysML is intended to create cohesive and consistent models of aspects of systems engineering (requirements, system structure, structure, behavior including their interconnections [1]. system behavior and parameters). Cells of the grid (Fig. 1) Requirements engineering widely recognized as a critical represent different views of model-based systems engineering phase in MBSE which consists of two main processes: [6]. Specified traceability among view specifications is a very specification and management. Requirements management is an important aspect of the MBSE Grid method. The method helps important part of the discipline that includes the planning, to organize and maintain the model. monitoring, analyzing, communicating, and managing of Copyright held by the author(s). 66 developed as an extension to the value-based approach supported by the TOSCA Testsuite™. In conclusion, all the analyzed methods to evaluate the completeness of system requirements specification encounter several common issues: (i) unsupported completeness evaluation of particular stage of SRS, (ii) unsupported completeness evaluation during entire specifying process of SRS, (iii) unclear traceability relationships between requirements and design elements, (iv) are applied to the small area of the domain or a specific tool. Overall, researches carried out in this area have very little proof that they been successfully applied in real-world industry. Fig. 1. MBSE Grid We are proposing a more generic approach, applicable to the majority of SysML modeling tools for different systems This research is carried out using MagicDraw toolset, which engineering domains. The proposed approach in combination supports SysML. It was chosen because of several published with MBSE Grid will evaluate the completeness of particular studies, e.g. [7], [8], [9], [10]. stage or entire specifying process of requirements specification. The rest of this paper is structured as follows: in section 2, This will ensure the high quality of each stage of SRS. the related works are analyzed; in section 3, the proposed III. AN APPROACH FOR COMPLETENESS EVALUATION OF approach for automated completeness evaluation of the SYSTEM REQUIREMENTS SPECIFICATIONS requirements specification is presented; in section 4, evaluation of the proposed approach is described; in section 5, the achieved This section describes the proposed approach in detail. results, conclusions, and future work directions are indicated. The approach consists of the following metric groups to II. RELATED WORKS evaluate the completeness of SRS that is defined in accordance with the principles of MBSE Grid method: A number of evaluation methods and techniques for requirements specification are currently used. Most of them are A. Requirements Refinement Metrics applied to the small area of the domain or a specific tool, e.g. B. Requirements Satisfaction Metrics [11], [12]. C. Requirements Derivation Metric Several authors proposed methodologies for evaluation of completeness [13], [14], [12], [15], [16], [17] use formal D. Requirements Verification Metric techniques, e.g. mapping of model elements between successive Completeness metrics are based on the determined levels of the refinement hierarchy in [13], service-based domain traceability relationship between requirements and other model requirements completeness in [12]. [14] describes an ontology- elements in the MBSE Grid. Only atomic model elements that based approach for completeness verification of a requirements are linked to the atomic requirements are included in the specification. [15] describes an approach to support the calculation of completeness metrics. The relation between quantitative assessment of goal-oriented and scenario-based atomic requirements and atomic model elements eliminates the requirements model completeness. [16] describes how the ambiguities that may occur having relations between higher compositional properties of the formalism can be used to level elements. perform completeness analysis. In [3] publication is proposed three metrics categories of the SRS Completeness: Formal Completeness, Semantic Completeness and Reference Completeness. Formal Completeness - counts the number of elements that are required by meta-classes and searches missing ones. Semantic Completeness –measures a missing semantic element count. Reference Completeness - evaluates the "trace" references leading to the "solutions” and all missing references that are required by the elements meta-classes. Use of traceability relationships to evaluate the completeness or coverage of the requirements specification has been defined in [18], [19], [20], [21]. An approach in [18] calculates the coverage of a requirement as the degree to which the source code of (e.g. relevant to execute) a requirement is covered by tests. Fig. 2. MBSE Grid Traceability [19] paper describes a model-based testing process directed by structural coverage and functional requirements. The approach In order to obtain the more precise evaluation results of in [20] describes measuring the requirements coverage requirements specification completeness, metrics are 67 categorized by three aspects of the system engineering: RRPhR – physical requirements refinement by structure elements Behavior, Structure, and Parameters. Each elements group of the metric system aspects coverages the specific requirement category: PhRSN – quantity of atomic physical requirements of stakeholder needs refined by atomic structure element.  functional requirements are covered by behavior elements. Atomic behavior element - call behavior PhR – quantity of atomic physical requirements of stakeholder action which has assigned behavior that does not have needs owned elements of call behavior actions. Call behavior  Interface Requirements Refinement by Proxy Ports action has to represent the behavior of the system; Elements metric  physical requirements are covered by structure This metric evaluates the refinement of interface elements. Atomic structure element - block which does requirements by proxy ports. This evaluation represents the not have owned Part Property or Part Property which completeness of Logical Subsystem Communication. Below has assigned Type that not have owned elements of Part is provided the metric formula. Property;  interface requirements are covered by Proxy Port; 𝑅𝑅𝐼𝑅 = 𝐼𝑅𝑆𝑁 × 100%   𝐼𝑅  performance requirements are covered by Value Property. RRIR – interface requirements refinement by proxy ports elements metric The proposed method concerns the completeness evaluation of the system requirements specifications. An approach is IRSN – quantity of atomic interface requirements of stakeholder implemented in the MagicDraw modeling tool. needs refined by atomic proxy ports. IR – quantity of atomic interface requirements of stakeholder needs The subsections below describe in detail each completeness metric of requirements specification.  Performance Requirements Refinement by Parameters A. Requirements Refinement Metrics Elements metric Metric group of requirements refinement evaluates the This metric evaluates the refinement of performance completeness of White Box stage of Problem layer in MBSE requirements by parameters. This evaluation represents the Grid. The metric group evaluates the refinement of stakeholder completeness of Measurements of Effectiveness. Below is needs by model elements which are specified at white box layer. provided the metric formula. Requirements refinement metric group consists of the 𝑃𝑅𝑆𝑁 𝑅𝑅𝑃𝑅 = × 100%   following metrics: 𝑃𝑅  Functional Requirements Refinement by Behavior RRPR – performance requirements refinement by parameters Elements Metric elements metric This metric evaluates the refinement of functional PRSN – quantity of atomic performance requirements of stakeholder requirements by behavior elements. This evaluation needs refined by parameters element. represents the completeness of Functional Analysis . Below PR – quantity of atomic performance requirements of stakeholder is provided the metric formula. needs 𝐹𝑅𝑆𝑁 B. Requirements Satisfaction Metrics 𝑅𝑅𝐹𝑅 = × 100%   𝐹𝑅 Metric group of requirements satisfaction evaluates the RRFR – functional requirements refinement by behavior elements completeness of Solution layer in MBSE Grid. The metric group metric evaluates the system requirements satisfaction by atomic model elements which are specified at solution layer. FRSN – quantity of atomic functional requirements of stakeholder needs refined by atomic behavior element. Requirements satisfaction metric group consists of the following metrics: FR – quantity of atomic functional requirements of stakeholder needs  Functional Requirements Satisfaction by Behavior Elements metric  Physical Requirements Refinement by Structure Elements metric This metric evaluates the satisfaction of functional requirements by behavior elements. This evaluation This metric evaluates the refinement of physical represents the completeness of Component Behavior. Below requirements by structure elements. This evaluation is provided the metric formula. represents the completeness of Logical Subsystem Communication. Below is provided the metric formula. 𝐹𝑅𝑠 𝑅𝑆𝐹𝑅 = × 100%   𝐹𝑅 𝑃ℎ𝑅𝑆𝑁 𝑅𝑅𝑃ℎ𝑅 = × 100%   𝑃ℎ𝑅 68 RSFR – functional requirements satisfaction by behavior elements represents the completeness of System Requirements. Below metric is provided the metric formula. FRS – quantity of atomic functional requirements of system 𝑆𝑅𝐷 satisfied by atomic behavior element. 𝑅𝐷𝑆𝑅 = × 100%   𝑆𝑅 FR – quantity of atomic functional requirements of the system RDSR – system requirements derivation from stakeholder needs  Physical Requirements Satisfaction by Structure metric Elements metric SRD – quantity of atomic system requirements derived from This metric evaluates the satisfaction of physical stakeholder needs. requirements by structure elements. This evaluation represents the completeness of Component Structure. Below SR – quantity of atomic system requirements. is provided the metric formula. D. System Requirements Verification metrics 𝑃ℎ𝑅𝑆 This metric evaluates the verification of system 𝑅𝑆𝑃ℎ𝑅 = × 100%   requirements by test cases. Below is the formula for 𝑃ℎ𝑅 evaluating the system requirements verification. RSPhR – physical requirements satisfaction by structure elements metric. 𝑆𝑅𝑉 𝑅𝑉𝑆𝑅 = × 100%   𝑆𝑅 PhRS – quantity of atomic structure requirements of system satisfied by atomic structure element. RVSR – system requirements verification by test cases metric PhR – quantity of atomic physical requirements of the system. SRV – quantity of atomic system requirements verified by test  Interface Requirements Satisfaction by Proxy Elements cases. metric SR – quantity of atomic system requirements. This metric evaluates the satisfaction of interface E. Satisfaction evaluation of System Requirements requirements by proxy ports. This evaluation represents the completeness of Component Structure. Below is provided This subsection describes in the detail the principles of the metric formula. the system requirements satisfaction evaluation by elements that are defined in Component Structure (S3) stage. 𝐼𝑅𝑆 𝑅𝑆𝐼𝑅 = × 100%   The figure below (Fig. 3) represents the system 𝐼𝑅 requirements satisfaction by structure and proxy ports RSIR - interface requirements satisfaction by proxy ports elements elements. metric IRS – quantity of atomic interface requirements of system satisfied by proxy ports element. IR – quantity of atomic interface requirements of the system.  Performance Requirements Satisfaction by Parameters Elements metric This metric evaluates the satisfaction of performance requirements by parameters. This evaluation represents the completeness of Component measurements of effectiveness. Below is provided the metric formula. 𝑃𝑅𝑆 𝑅𝑆𝑃𝑅 = × 100%   𝑃𝑅 RSPR – performance requirements satisfaction by parameter elements metric PRS – quantity of atomic performance requirements of system satisfied by parameter element. PR – quantity of atomic performance requirements of the system. Fig. 3. Satisfaction of system requirements C. System Requirements Derivation metric In the figure below (Fig. 4), is provided the Satisfy This metric evaluates the system requirements Requirement Matrix, which visualizes the satisfy dependencies derivation from stakeholder needs. This evaluation between system requirements and model element. 69 IRS = 1 IR = 2 1  𝑅𝑆𝐼𝑅 = × 100% = 50%   2 This indicates that 50% of the interface system requirements are satisfied by proxy ports that are defined at Component Structure (S2). IV. CASE STUDY This section describes the case study of the proposed approach. This is a case study of a commercial project to evaluate the completeness of the system requirements specification. The commercial project is based on SysML and is modeled in the MagicDraw toolset. The SRS has been modeled according to the principles of MBSE grid. The completeness of system requirements specification has been evaluated over the whole period of requirements Fig. 4. Satisfy Requirement Matrix specification process. In the beginning, the manager determined that the coverage of each completeness metric should be at least The evaluation process of the system requirements 90 % in order to move to the next stage of the specification. After satisfaction by structure elements and proxy ports: each metric calculation, the manager has been analyzed the metric data and made appropriate decision to ensure a high 1. The quantity of atomic physical requirements of the quality of the SRS. system is calculated. Fig. 5 provides the calculated table of Requirements 2. The quantity of atomic physical requirements of the Refinement in the MagicDraw tool. In order to effectively system that are satisfied by atomic structure elements analyze the metrics data, charts were generated based on is calculated. Only those physical requirements which calculated metrics data. The charts help the manager to quickly are satisfied by a block that does not have owned Part compare data and monitor the progress of SRS completeness. Property or are satisfied by a Part Property that has assigned a Type that does not have owned Part Property are included in the calculation. 3. The metric “Physical Requirements Satisfaction by Structure Elements” (6) is calculated using before calculated quantities at first and second step. 4. The quantity of atomic interface requirements of the Fig. 5. Metric Table of Requirements Refinement Evaluation system is calculated. Below is provided a detailed completeness analysis of each 5. The quantity of atomic interface requirements of a metric group. Calculated metric data is provided in the charts. system that are satisfied by a Proxy Port is calculated. 6. The metric “Interface Requirements Satisfaction by Proxy Elements” (7) is calculated using before calculated quantities at fourth and fifth step. Following is the evaluation result of systems requirements satisfaction according to Fig. 3 and Fig. 4. PhRS = 2 PhR = 2 2  𝑅𝑆𝑃ℎ𝑅 = × 100% = 100%   2 This indicates that 100% of the physical system requirements are satisfied by structure elements that are defined at Component Structure (S2). Fig. 6. Requirements Refinement Chart 70 Fig. 6 provides the requirements refinement analysis chart. calculation showed that only 68.12% of functional requirements Requirements refinement metrics have been calculated over the were satisfied by behavior elements. The decision was taken to entire period specifying the problem layer of requirements continue the specification of requirements satisfaction by specification. First of all, functional requirements of stakeholder behavior element. At third metric calculation the satisfaction by have been refined by behavior elements. Results of first metric behavior reached 91.30% and has been started another stage of calculation showed that only 65.52% of functional requirements the specification, the satisfaction of physical and interface were refined by behavior elements. The decision was taken to system requirements. The specification of physical and interface continue the specification of requirements refinement. At second requirement refinement was continued until the satisfaction metric calculation reaching the 91.38% of functional level reached over the 90%. When all metrics of satisfaction requirements refinement has been started another stage of the reached over 90%, it was decided that the satisfaction of system specification, refinement of physical and interface requirements. requirements is sufficient. The specification of physical and interface requirement refinement was continued until the refinement reached over the 90%. When all metrics of refinement reached over 90%, it was decided that the refinement of stakeholder requirements is sufficient. Fig. 9. Requirements Verification Chart Fig. 9 provides the analysis chart of system requirements verification by test cases. The first calculation of metric showed that only 43.09% of system requirements were verified. The Fig. 7. Requirements Derivation Chart decision was taken to continue the specification of requirements verification. At third metric calculation reaching the 98.38% of Fig. 7 provides the analysis chart of system requirements system requirements verification was decided that the derivation from Stakeholder Needs. Reaching 93.37% of verification level is sufficient. derivation at the third calculation of metric was decided that the system requirements derivation from stakeholder needs is V. CONCLUSION AND FUTURE WORKS sufficient. In this paper, we have analyzed the methods for completeness evaluation of system requirements specification. The analysis disclosed that there are many different methods for evaluating the completeness of the SRS, but none of them are appropriate for use in the evaluation of the SRS during the entire period of specifying process. Also, most of the methods cannot be used in combination with systems modeling techniques, such as SysML, in practice. We have identified the need for a more generic approach, applicable to the majority of SysML modeling tools for different systems engineering domains. In this paper, we have proposed a new approach of how requirements specification, expressed with sufficient precision in SysML, can be used for automated completeness evaluation. The approach is composed of metric groups that are defined on the basis of the principles of MBSE Grid method: Requirements Refinement Metrics, Requirements Satisfaction Metrics, Fig. 8. Requirements Satisfaction Chart Requirements Derivation Metric, Requirements Verification Metric. Fig. 8 provides the requirements satisfaction analysis chart. The proposed approach has been implemented in the Requirements satisfaction metrics have been calculated over the MagicDraw CASE tool and the case study of commercial project entire period specifying the solution layer of requirements based on the principles of SysML and MBSE Grid has been specification. First, functional requirements of the system have demonstrated. The analysis of case study disclosed that been satisfied by behavior elements. Results of first metric 71 computing a particular metric group determines the completion [11] L. Mangeruca, O. Ferrante and A. Ferrari, "Formalization and of a certain stage of SRS. Completeness calculation reduces the completeness of evolving requirements using Contracts," in 013 8th risk of issues detection and correction in the late stage of IEEE International Symposium on Industrial Embedded Systems (SIES), Porto, 2013. development and ensures a high quality of requirements specification. [12] W. Liu, Chengwan He and Kui Zhang, "Service-based domain requirements completeness analysis," in 2009 Asia-Pacific Conference Currently, the approach is oriented to automated on Computational Intelligence and Industrial Applications (PACIIA), Wuhan, 2009. completeness evaluation of system requirements specification. We plan to expand this approach in the near future, to more [13] N. Deb, N. Chaki and A. Ghose, "Using i* model towards ontology integration and completeness checking in enterprise systems requirement accurately evaluate the completeness and consistency of the hierarchy," in 2015 IEEE International Model-Driven Requirements system requirements specification. Engineering Workshop (MoDRE), Ottawa, 2015. [14] T. Avdeenko and N. Pustovalova, "The ontology-based approach to VI. REFERENCES support the completeness and consistency of the requirements [1] S. Friedenthal, A. Moore and R. Steiner, A practical guide to SysML, specification," in 2015 International Siberian Conference on Control San Francisco: Morgan Kaufmann, 2014. and Communications (SIBCON), Omsk, 2015. [2] P. M. Institute, "PMI's Pulse of the profession®: Requirements [15] C. Gralha, "Evaluation of Requirements Models," in 2016 IEEE 24th management-A core competency for project and program success," International Requirements Engineering Conference (RE), Beijing, Newtown Square, 2014. 2016. [3] Lian Yu, Shuang Su, Shan Luo, Yu Su, "Completeness and Consistency [16] M. P. E. Heimdahl, N. G. Leveson, "Completeness and Consistency Analysis on Requirements of Distributed," in 2nd IFIP/IEEE Analysis of State-Based Requirements," in 17th International International Symposium on Theoretical Aspects of Software Conference on Software Engineering, Seattle, 1995. Engineering, 2008. [17] D. Alrajeh, J. Kramer, A. van Lamsweerde, A. Russo and S. Uchitel, [4] E. Knauss and C. E. Boustani, "Assessing the quality of software "Generating obstacle conditions for requirements completeness," in 2012 requirements," in 2008 16th IEEE International Requirements, 34th International Conference on Software Engineering (ICSE), Zurich, Catalunya, 2008. 2012. [5] D. Mazeika, A. Morkevicius and A. Aleksandraviciene, "MBSE driven [18] R. Mordinyi, S. Biffl, "Exploring Traceability Links via Issues for approach for defining problem domain," in 1th System of Systems Detailed Requirements Coverage Reports," in 2017 IEEE 25th Engineering Conference, Kongsberg, 2016. International Requirements Engineering Conference Workshops (REW), Lisbon, 2017. [6] A. Morkevicius, A. Aleksandraviciene, D. Mazeika, L. Bisikirskiene, Z. Strolia, "MBSE Grid: A Simplified SysML-Based Approach for [19] Y. Sun, G. Memmi and S. Vignes, "Model-Based Testing Directed by Modeling Complex Systems," in 27th Annual INCOSE International Structural Coverage and Functional Requirements," in 2016 IEEE Symposium, 2017. International Conference on Software Quality, Reliability and Security Companion (QRS-C), Vienna, 2016. [7] R. Cloutier, M. Bone, "Compilation of SysML RFI- Final Report," 20 02 2010. [Online]. Available: [20] R. Ramler, T. Kopetzky and W. Platz, "Value-Based Coverage https://www.nomagic.com/mbse/images/whitepapers/ Measurement in Requirements-Based Testing: Lessons Learned from an omg_rfi_final_report_02_20_2010-1.pdf. Approach Implemented in the TOSCA Testsuite," in 2012 38th Euromicro Conference on Software Engineering and Advanced [8] S. C. Spangelo, "Applying Model Based Systems Engineering (MBSE) Applications, Cesme, 2012. to a Standard CubeSat," in Aerospace Conference IEEE, 2012. [21] P. Rempel and P. Mäder, "Preventing Defects: The Impact of [9] C. Delp, D. Lam, E. Fosse, Cin-Young Lee., "Model based document and Requirements Traceability Completeness on Software Quality," IEEE report generation for systems engineering," in Aerospace Conference, Transactions on Software Engineering, vol. 43, pp. 777-797, 2017. 2013. [10] P. Godart, J. Gross, R. Mukherjee, W. Ubellacker, "Generating real-time robotics control software from SysML," in IEEE Aerospace Conference, 2017. 72