This paper presents the tool ScopeMaster ® developed by Albion Technology Ltd. and discuss its benefits in terms of better COSMIC measurements, defect detection and improved requirement quality. ScopeMaster ® is a COSMIC Functional Size Measurement tool that takes free form textual software requirements as input.
The tool, basically:
• detects the probable data movements and so measures them using the COSMIC FSM • points out potential defects within the requirements • generate project estimates based on COSMIC size Automating functional size measurement with COSMIC is one of the top priorities in research in COSMIC community today. It is widely accepted that automating COSMIC measurements is crucial for its acceptance in industry.
There exist numerous proposals for tools that support COSMIC measurement (automated and non-automated) and new ones are emerging every day [1][2][3][4][5][6][7][8]. These tools can be grouped based on the main functionality they provide as [1,9]:
• Data collection and calculation: These tools make it easier to record and manage the measurement data. Measurement is performed manually, and the tools enable entering measurement data in an orderly fashion and keep the meta data about the measurements. • Expert systems for measuring (Measurement Facilitation): These tools enable measurement to be performed within the tool. They enable entering and managing pre-set constructs for COSMIC measurements such as objects within a system, standard users, standard functionality. They may also provide basic checks for measurement rules to improve measurement quality. Measurement is still performed manually but with guidance and facilitation of expert systems for measuring. There exist a wide scope of both academic and industrial research potential on functional size measurement automation. Most of the existing research and tools utilize formalized input for measurement such as structured requirements, conceptual models, UML models, source code or similar constructs.
To the best of our knowledge there is only limited academic research on automating COSMIC measurement using textual requirements in natural language as input [10] [11] and there exist no commercial tool that implements that.
The paper is structured as follows, Section 2 summarizes how applying COSMIC measurement to requirements inherently improves their quality. Section 3 presents ScopeMaster ® including its features, generated reports and its current limitations. Section 4 describes how it helps improving requirement quality through pointing out defects and helping analysts fix those defects. Section 5 presents a summary and lists the road map for intended additional features for the tool.
Requirement Quality and COSMIC Quality of software requirements hugely impact the quality of any software product as well as any measurement result based on them [12]. Between 16% and 20% of defects in software are requirement defects [13]. Moreover, any defect in requirements cost much higher to fix compared to defects injected in further phases of SDLC. The rework cost to fix a defect injected in requirements increase exponentially as it propagates through lifecycle phases. As illustrated in Fig. 1, rework cost for defects found later in the SDLC are significantly more expensive than those found in earlier phases.
Pressman [14] states the of the cost of fixing a defect in requirements within requirements phase is 1 unit, cost to fix it in coding, test and deployment phases will be multiples of 1, 6.5, 15 and 80. Based on these, it is imperative to detect and fix a defect in requirements as early as possible to minimize rework costs. For this, organizations typically perform reviews on their requirements documents. Reviews are typically performed to make sure that requirements have some generic and domain specific quality attributes. IEEE Std 830-1998 -IEEE Recommended Practice for Software Requirements Specifications [15] defines a set of quality attributes for requirements such as being:
• Clear (Unambiguous)
• Current However, free form requirements expressed in natural language are prone to being vague, long and incomplete.
Like other recognized Functional Size Measurement (FSM) methods [16][17][18][19]] COSMIC [20] requires the functional requirements to be defined in a certain level of detail and quality. In order for the measurement rules to be properly applicable, requirements should possess certain qualities.
Users, objects and functions should be clearly identifiable. Any functional process within a requirement should include clear definitions for triggering events, users, inputs, outputs and functional steps.
Studies showed that whenever a requirement cannot be properly measured, that also points out an inadequacy in terms of basic requirement qualities [12][21] [22][23]. This renders COSMIC a valuable tool to verify requirement quality and detect defects extremely early in the software development life cycle (SDLC).
There are many studies that demonstrate how an FSM, and in particular COSMIC, can improve software specification quality through pointing out defects in requirements:
In [24], Talib et.al. illustrates COSMIC FSM can be used to asses the quality of specifications of a real-time software system. They state that COSMIC measurement especially helps in detecting ambiguous requirements and requirements whose hardware/software allocation is not clear.
In [22] Trudel and Abran demonstrate that using COSMIC measurement during inspections lead an increase of 16% to 32% in the number of identified critical functional defects. Moreover, the study also showed that inspectors spent only 54% of the planned effort for inspections when they utilized COSMIC measurement during inspections.
COSMIC Guideline for Measurement Accuracy [25] also proposes an approach to assess the quality of requirements in terms of COSMIC measurement principles. The assessment is performed in terms of:
• The presence or absence of a data model.
• The presence of absence of information to identify the data movements (entry, read, write, exit). • The presence (or absence) of documentation enabling identification of each functional process.
ScopeMaster® is a web-based tool which users login using their credentials and can create many measurement projects.
Once the user creates a measurement project in the tool, ScopeMaster ® enables adding requirements one at a time or importing them in bulk via a CSV file as well as through integration with the JIRA[26] tool.
As the user adds user software requirements or user stories, ScopeMaster ® performs several successive steps of analysis individually and collectively on the requirements in order to detect possible Objects of Interest, potential users, potential data movements and potential defects.
The details of how ScopeMaster ® performs these techniques are proprietary and subject to a pending patent application, however the results are fully transparent. The underlying steps include natural language processing and several modules of pattern matching.
The tool focuses on detecting a "Subject verb object" structure to identify necessary measurement elements (functional users, objects of interest, data movements) from requirements.
A user story such as "As a user I want to display orders" has "user" as the subject, "display" as the verb and "orders" as the object. In this structure, the subject is a candidate for the Functional User, the object is a candidate for an Object of Interest and verb corresponds to one of Create, Read, Update, Delete, List (CRUDL) set of functions.
Below, we present features of the tool through the example of C-REG COSMIC Case study [27].
Each requirement consists of four fields: Title and Body, which are analyzed for possible interpretation, ID and Notes, which are searchable but not analyzed for functional interpretation. The Body field is the main focus of the requirement. The tool encourages this field to be a succinct but complete statement of the overall purpose of the requirement or user story. The scenarios, conditions and success criteria should all be put into the notes field.
ScopeMaster® believes that the most important characteristic to get right first, is what is the requirement's main purpose. The main purpose of a requirement can be different functionalities such as:
• "Update requirements"
• "Maintain calendar entries"
• "Display orders", "Delete invoice",
• "Search for companies"
• "Book a room" Information such as the triggering event conditions, the scenarios in which the requirement applies and the outcomes to be tested are all deemed peripheral to the primary purpose of the requirement and should be put into the notes field.
As soon as the requirement is added, the tool measures the size the requirement and displays the Functional Process, Objects of Interests, CRUDL operation type and Data Movements. Tool also assesses the readability of the requirement and color codes the requirement to indicate any problems in its readability.
ScopeMaster® generates a number of reports for the project based on the analysis of requirements. Primary analysis outcomes include:
Reporting of ambiguous requirements. ScopeMaster reports possible ambiguities in the requirement texts which may lead to a reader interpreting the requirement differently from the author's intent. Tool highlights words causing ambiguity in a sentence and advise the author to revise them (see Fig. 5.) and also generates reports for requirements that are not concise and that require multiple verbs for a single operation (see Fig. 6. Ambiguous or verbose requirement warning Secondary analysis steps derive:
IFPUG sizing estimates. ScopeMaster ® also generates size estimates and reports for IFPUG [16] measurement. However, the estimates and reports for IFPUG are less accurate than those for COSMIC due to the sensitivity of detecting actual ILFs. Any wrongfully identified ILF in IFPUG measurement has a much bigger impact on accuracy then any wrongfully identified Object of Interest in COSMIC measurement.
The tool generates indicative estimates and gives a range for development cost, duration and resources. It also gives an estimate for scope creep and expected number of defects in all phases of SDLC.
Most of these estimations are based on public benchmark data.
Classes and Methods. The tool generates a list for potential classes and method for implementation, based on the objects and operations identified from the requirements.
ScopeMaster® interprets the English used to write the requirements or user stories. Currently this interpretation is not 100% accurate, but algorithms are being continuously improved, through formal verification and through machine learning. It is expected to reach a consistent accuracy in excess of 85%. Currently results are in the 70-95% accuracy range. Dealing with the nuances of the natural language renders %100 accurate interpretation almost impossible. However, the clearer the requirements the more accurate the counts will be. Moreover, any measurement with the tool is repeatable, that is, given the same set of requirements it always produces the same measurement results.
Currently the tool can only analyze the main function within a requirement. It does no text analysis of the success criteria or triggering events mentioned in the requirement text. This sometimes result in mixed results if there are too much side information given within the requirement text.
Similarly, interpretation accuracy decreases when there are multiple sophisticated condition based requirements within a text. It is suggested that such requirements are broken down into smaller requirements.
Ontology used is not customizable at the moment. For example the list of recognized verbs is fixed and same for each project. There might a need for nuances or combinations of verbs for different development contexts. It is planned to make ontology customizable in the future.
We believe COSMIC size estimation performed by ScopeMaster® provides a very effective starting point to fine tune the measurement manually. As ScopeMaster® displays the interpreted data movements of every FUR, the user has full transparent visibility as to the makeup of the count and can adjust the wording of the requirement appropriately.
In its current state, ScopeMaster ® size estimates are observed to be typically within 20-30% of manual count equivalents. This was also verified with the case study explained above. Given that manual COSMIC estimates also tend to have a level of discrepancy among different measurers in manual measurement [28], an automated estimate in these ranges pose a fairly good basis especially when the automated results are reviewed and fixed manually.
ScopeMaster ® also helps making requirements "measurable" by pointing out possible problems regarding the requirements in the measurement process. It allows requirement texts to be improved by giving constant feedback about their measurability and ambiguity. Requirements can be tweaked and updated until the desired structure is attained for the COSMIC measurement. This, in turn, results in improving the quality attributes of requirements mentioned in section 2:
• By pointing out too long or too short requirements, it enables requirements to be defined in appropriate length for measurement. It prevents many Functional Processes to be combined under a single Functional User Requirement or User Story. This enable the requirements to be more concise. • By pointing out inconsistencies between the number of data movements detected in a requirement and the length of its text, it points out probable non-functional or unnecessary information cramped in a functional user requirement. This enables the requirements to be clear and atomic. • By pointing out similar objects and users among requirements, it highlights naming inconsistencies so that they can be corrected. This enables the requirements to be consistent with each other. • By pointing out potentially missing operations (eg. CRUDL) it helps requirements to be complete. • By pointing out multiple operations of same type (eg. CRUDL) on a given object of interest, it enable detection of any inconsistencies among requirements and enables requirements to be consistent and correct. • There is a limited list of verbs that are recognized by the tool. Authors are required to use verbs from that list for their requirements to be measurable. This forces requirements to be written in a more formalized fashion and in turn becoming more concise and clear.
ScopeMaster ® also decreases the time and effort needed for both COSMIC measurement and finding and fixing requirement defects.
Typically a COSMIC certified measurer can measure 125-500 CFPs per day, however, using ScopeMaster® this productivity goes up to 500-2500 CFPs a day. The tool performs the measurement analysis in a matter of minutes, these daily CFP numbers are for cases when the measurement results that are analyzed, fixed and verified by the measurer. Moreover, when the tool's measurement results are used as a basis for measurement, it standardizes measurement throughout an organization. This will also reduce discrepancy among measurement results from different measurers and increase repeatability of results.
Manual requirements quality inspections typically require half an hour in a 4 people meeting to groom and find 1 defect and requires half a day of work for 1 person to fix it. This totals 5-6 hours of effort for each defect found and fixed manually. The ScopeMaster ® authors claim that a single person can find and fix a defect in as little as 15 minutes. The results of our case study described below also proved this improvement.
This improvement is due to the fact that the when the author of a requirement is going through and fixing the initial COSMIC estimations of ScopeMaster ® , he is also fixing most of the defects in the requirements.
In order to demonstrate how ScopeMaster® can benefit practitioners, we conducted a small case study. This study was conducted as a proof of concept rather than an academic study.
A senior software project assurance expert participated in the study and was asked to analyze a software specification for a real life game software using ScopeMaster ® without any prior experience with the tool.
Specifications consisted of 80 user stories at start and increased to 90 user stories by the time the study finished.
When the requirements were run, ScopeMaster ® counted 400 CFPs and discovered over 200 potential defects in less than 60 seconds.
The participant was able to go through all 90 requirements and fixed 150 requirements defects within 16 hours. This corresponded to finding and fixing defects at a rate of 1 every 10 minutes.
Participant commented that while he was working through requirements, it was possible to review measurement results, see defects, fix defects and add missing requirements all at the same time.
In this paper we presented some of the features of the automatic COSMIC size measurement tool ScopeMaster ® . To our knowledge this tool is the first commercial tool that performs COSMIC measurement on free form textual requirements.
We believe it has a great potential in not only automating COSMIC size measurement from requirements in natural language but also improving the quality of requirements. It allows users to immediately see if their requirements are measurable and enables them to revise and update them on the fly by proving constant feedback.
As shown by many studies, the process of making requirements measurable, on its own, improves quality of requirements. ScopeMaster ® improves requirement quality through COSMIC measurement principles and on top of this, it explicitly points out defects in clarity, completeness, concision and consistency.
Using the tool to facilitate COSMIC measurement improves the accuracy of measurement results compared to purely manual measurement. It significantly reduces measurement effort. It also enables requirement defects to be detected and fixed with much less effort compared to manual inspections.
We believe, ScopeMaster ® would particularly benefit big organizations, consisting of many teams that similar software with similar requirements, by streamlining their measurement and requirement verification activities.
On the other hand, the tool lets practitioners to utilize COSMIC principles and measurement results without any prior knowledge about the method. This means, whenever ScopeMaster ® is deployed in an organization for quality improvement and defect detection purposes, it will be indirectly introducing COSMIC Functional Size Measurement and its benefits to the organization as well. This would be particularly beneficial for the COSMIC community to widen the methods adaption as industry always demands tools to mitigate rework cost and improve software quality however, it is not always easy to get organizations appreciate the value added by FSM methods.
The tool is under constant development and improvement. In the near future these features are expected to be added:
• Managing measurement elements • Custom ontology: Making ontology used for requirements such as making the verb list customizable for organizations or defining a pre-determined set of objects to use for entire organization. This will enable organizations better tune the interpretation of their requirements and also will let them introduce and enforce usage of their own requirement standards and wording guidelines. • Requirements suggestions engine (re-use, security): Certain frequent and typical requirements will be presented in the form of functional patterns. Certain generic functions such as security functions will be suggested based on type of identified objects. This will further prevent facilitate requirement development and prevent missing requirements. • More Machine Learning and better language interpretation. • Multilanguage support: It is expected to include other language than English for requirement interpretation. • Local Benchmarking: Currently certain reports utilize generic ratios and productivity values from public benchmarking data sets. It is planned to enable organizations utilize their internal benchmarking data sets and standard values. • Test suggestions: The tool will suggest a list of potential unit tests for any given method on an object. cases for requirements.