This paper presents a new COSMIC functional measurement automation tool ScopeMaster®, which automatically generates a size estimation from textual requirements. The tool points out problems in requirements in clarity, completeness, consistency and concision. This facilitates both COSMIC measurement and quality inspection processes. Utilizing ScopeMaster guides users to create higher quality requirements. It improves measurement accuracy, increases the number of defects found in inspections and drastically reduces the effort for both activities. Enabling higher number of requirement defects to be found early in the SDLC has a huge effect on overall software quality and rework costs. ScopeMaster also provides detailed reports on project estimates and measurement details.
• Data collection and calculation: These tools make it easier to record and manage the measurement data. Measurement is performed manually, and the tools enable entering measurement data in an orderly fashion and keep the meta data about the measurements. • Expert systems for measuring (Measurement Facilitation): These tools enable measurement to be performed within the tool. They enable entering and managing pre-set constructs for COSMIC measurements such as objects within a system, standard users, standard functionality. They may also provide basic checks for measurement rules to improve measurement quality. Measurement is still performed manually but with guidance and facilitation of expert systems for measuring. • Automated Measurement: ─ Based on Structured Input: These tools perform COSMIC measurement autonomously and automatically utilizing structured inputs such as: o Structured/Formalized Functional Requirements o UML Diagrams o Software Design Models o User Interfaces o Source Code ─ Based on unstructured input: These tools perform measurement on free form textual requirements.
There exist a wide scope of both academic and industrial research potential on functional size measurement automation. Most of the existing research and tools utilize formalized input for measurement such as structured requirements, conceptual models, UML models, source code or similar constructs.
To the best of our knowledge there is only limited academic research on automating
COSMIC measurement using textual requirements in natural language as input [
The paper is structured as follows, Section 2 summarizes how applying COSMIC measurement to requirements inherently improves their quality. Section 3 presents ScopeMaster® including its features, generated reports and its current limitations. Section 4 describes how it helps improving requirement quality through pointing out defects and helping analysts fix those defects. Section 5 presents a summary and lists the road map for intended additional features for the tool. 2
Quality of software requirements hugely impact the quality of any software product
as well as any measurement result based on them [
Pressman [
Like other recognized Functional Size Measurement (FSM) methods [
Users, objects and functions should be clearly identifiable. Any functional process within a requirement should include clear definitions for triggering events, users, inputs, outputs and functional steps.
Studies showed that whenever a requirement cannot be properly measured, that also
points out an inadequacy in terms of basic requirement qualities [
There are many studies that demonstrate how an FSM, and in particular COSMIC, can improve software specification quality through pointing out defects in requirements:
In [
In [
COSMIC Guideline for Measurement Accuracy [
ScopeMaster® is a web-based tool which users login using their credentials and can create many measurement projects.
Once the user creates a measurement project in the tool, ScopeMaster® enables
adding requirements one at a time or importing them in bulk via a CSV file as well as
through integration with the JIRA[
As the user adds user software requirements or user stories, ScopeMaster® performs several successive steps of analysis individually and collectively on the requirements in order to detect possible Objects of Interest, potential users, potential data movements and potential defects.
The details of how ScopeMaster® performs these techniques are proprietary and subject to a pending patent application, however the results are fully transparent. The underlying steps include natural language processing and several modules of pattern matching.
The tool focuses on detecting a “Subject verb object” structure to identify necessary measurement elements (functional users, objects of interest, data movements) from requirements.
A user story such as “As a user I want to display orders” has “user” as the subject, “display” as the verb and “orders” as the object. In this structure, the subject is a candidate for the Functional User, the object is a candidate for an Object of Interest and verb corresponds to one of Create, Read, Update, Delete, List (CRUDL) set of functions.
Below, we present features of the tool through the example of C-REG COSMIC
Case study[
Each requirement consists of four fields: Title and Body, which are analyzed for possible interpretation, ID and Notes, which are searchable but not analyzed for functional interpretation. The Body field is the main focus of the requirement. The tool encourages this field to be a succinct but complete statement of the overall purpose of the requirement or user story. The scenarios, conditions and success criteria should all be put into the notes field.
ScopeMaster® believes that the most important characteristic to get right first, is what is the requirement’s main purpose. The main purpose of a requirement can be different functionalities such as: • “Update requirements” • “Maintain calendar entries” • “Display orders”, “Delete invoice”, • “Search for companies” • “Book a room”
Information such as the triggering event conditions, the scenarios in which the requirement applies and the outcomes to be tested are all deemed peripheral to the primary purpose of the requirement and should be put into the notes field.
As soon as the requirement is added, the tool measures the size the requirement and displays the Functional Process, Objects of Interests, CRUDL operation type and Data Movements. Tool also assesses the readability of the requirement and color codes the requirement to indicate any problems in its readability.
Requirements are displayed as a list with their corresponding estimated sizes next to them. The color of the size information indicates whether the requirement is potentially too big or involves different functional changes to multiple objects of interest. ScopeMaster® generates a number of reports for the project based on the analysis of requirements. Primary analysis outcomes include: Reporting of ambiguous requirements. ScopeMaster reports possible ambiguities in the requirement texts which may lead to a reader interpreting the requirement differently from the author's intent. Tool highlights words causing ambiguity in a sentence and advise the author to revise them (see Fig. 5.) and also generates reports for requirements that are not concise and that require multiple verbs for a single operation (see Fig. 6. Ambiguous or verbose requirement warning
gested users and suggested objects in the project with their number of occurrences in the requirements. Report potential missing and duplicate requirements. The tool shows which data movements correspond to CRUDL operations on any given object of interest and reports on potential missing and duplicate requirements. It assumes whenever an object is maintained in the system through one or more of CRUDL operations, it indicates other activities should be defined in a requirement as well. It also detects if there are multiple requirements depicting one of these operations on an object. This may point out to inconsistencies among requirements and flags those requirements for further inspection.
Secondary analysis steps derive:
IFPUG sizing estimates. ScopeMaster® also generates size estimates and reports for
IFPUG [
estimates and gives a range for development cost, duration and resources. It also gives an estimate for scope creep and expected number of defects in all phases of SDLC. Most of these estimations are based on public benchmark data.
Classes and Methods. The tool generates a list for potential classes and method for implementation, based on the objects and operations identified from the requirements. 3.2
ScopeMaster® interprets the English used to write the requirements or user stories.
Currently this interpretation is not 100% accurate, but algorithms are being continuously improved, through formal verification and through machine learning. It is expected to reach a consistent accuracy in excess of 85%. Currently results are in the 7095% accuracy range. Dealing with the nuances of the natural language renders %100 accurate interpretation almost impossible. However, the clearer the requirements the more accurate the counts will be. Moreover, any measurement with the tool is repeatable, that is, given the same set of requirements it always produces the same measurement results.
Currently the tool can only analyze the main function within a requirement. It does no text analysis of the success criteria or triggering events mentioned in the requirement text. This sometimes result in mixed results if there are too much side information given within the requirement text.
Similarly, interpretation accuracy decreases when there are multiple sophisticated condition based requirements within a text. It is suggested that such requirements are broken down into smaller requirements.
Ontology used is not customizable at the moment. For example the list of recognized verbs is fixed and same for each project. There might a need for nuances or combinations of verbs for different development contexts. It is planned to make ontology customizable in the future. 4
We believe COSMIC size estimation performed by ScopeMaster® provides a very effective starting point to fine tune the measurement manually. As ScopeMaster® displays the interpreted data movements of every FUR, the user has full transparent visibility as to the makeup of the count and can adjust the wording of the requirement appropriately.
In its current state, ScopeMaster® size estimates are observed to be typically within
20-30% of manual count equivalents. This was also verified with the case study
explained above. Given that manual COSMIC estimates also tend to have a level of
discrepancy among different measurers in manual measurement [
ScopeMaster® also helps making requirements “measurable” by pointing out possible problems regarding the requirements in the measurement process. It allows requirement texts to be improved by giving constant feedback about their measurability and ambiguity. Requirements can be tweaked and updated until the desired structure is attained for the COSMIC measurement. This, in turn, results in improving the quality attributes of requirements mentioned in section 2: • By pointing out too long or too short requirements, it enables requirements to be defined in appropriate length for measurement. It prevents many Functional Processes to be combined under a single Functional User Requirement or User Story.
This enable the requirements to be more concise. • By pointing out inconsistencies between the number of data movements detected in a requirement and the length of its text, it points out probable non-functional or unnecessary information cramped in a functional user requirement. This enables the requirements to be clear and atomic. • By pointing out similar objects and users among requirements, it highlights naming inconsistencies so that they can be corrected. This enables the requirements to be consistent with each other. • By pointing out potentially missing operations (eg. CRUDL) it helps requirements to be complete. • By pointing out multiple operations of same type (eg. CRUDL) on a given object of interest, it enable detection of any inconsistencies among requirements and enables requirements to be consistent and correct. • There is a limited list of verbs that are recognized by the tool. Authors are required to use verbs from that list for their requirements to be measurable. This forces requirements to be written in a more formalized fashion and in turn becoming more concise and clear.
ScopeMaster® also decreases the time and effort needed for both COSMIC measurement and finding and fixing requirement defects.
Typically a COSMIC certified measurer can measure 125-500 CFPs per day, however, using ScopeMaster® this productivity goes up to 500-2500 CFPs a day. The tool performs the measurement analysis in a matter of minutes, these daily CFP numbers are for cases when the measurement results that are analyzed, fixed and verified by the measurer.
Moreover, when the tool’s measurement results are used as a basis for measurement, it standardizes measurement throughout an organization. This will also reduce discrepancy among measurement results from different measurers and increase repeatability of results.
Manual requirements quality inspections typically require half an hour in a 4 people meeting to groom and find 1 defect and requires half a day of work for 1 person to fix it. This totals 5-6 hours of effort for each defect found and fixed manually. The ScopeMaster® authors claim that a single person can find and fix a defect in as little as 15 minutes. The results of our case study described below also proved this improvement.
This improvement is due to the fact that the when the author of a requirement is going through and fixing the initial COSMIC estimations of ScopeMaster®, he is also fixing most of the defects in the requirements. 4.1
In order to demonstrate how ScopeMaster® can benefit practitioners, we conducted a small case study. This study was conducted as a proof of concept rather than an academic study.
A senior software project assurance expert participated in the study and was asked to analyze a software specification for a real life game software using ScopeMaster® without any prior experience with the tool.
Specifications consisted of 80 user stories at start and increased to 90 user stories by the time the study finished.
When the requirements were run, ScopeMaster® counted 400 CFPs and discovered over 200 potential defects in less than 60 seconds.
The participant was able to go through all 90 requirements and fixed 150 requirements defects within 16 hours. This corresponded to finding and fixing defects at a rate of 1 every 10 minutes.
Participant commented that while he was working through requirements, it was possible to review measurement results, see defects, fix defects and add missing requirements all at the same time. 5
In this paper we presented some of the features of the automatic COSMIC size measurement tool ScopeMaster®. To our knowledge this tool is the first commercial tool that performs COSMIC measurement on free form textual requirements.
We believe it has a great potential in not only automating COSMIC size measurement from requirements in natural language but also improving the quality of requirements. It allows users to immediately see if their requirements are measurable and enables them to revise and update them on the fly by proving constant feedback.
As shown by many studies, the process of making requirements measurable, on its own, improves quality of requirements. ScopeMaster® improves requirement quality through COSMIC measurement principles and on top of this, it explicitly points out defects in clarity, completeness, concision and consistency.
Using the tool to facilitate COSMIC measurement improves the accuracy of measurement results compared to purely manual measurement. It significantly reduces measurement effort. It also enables requirement defects to be detected and fixed with much less effort compared to manual inspections.
We believe, ScopeMaster® would particularly benefit big organizations, consisting of many teams that similar software with similar requirements, by streamlining their measurement and requirement verification activities.
On the other hand, the tool lets practitioners to utilize COSMIC principles and measurement results without any prior knowledge about the method. This means, whenever ScopeMaster® is deployed in an organization for quality improvement and defect detection purposes, it will be indirectly introducing COSMIC Functional Size Measurement and its benefits to the organization as well. This would be particularly beneficial for the COSMIC community to widen the methods adaption as industry always demands tools to mitigate rework cost and improve software quality however, it is not always easy to get organizations appreciate the value added by FSM methods.
The tool is under constant development and improvement. In the near future these features are expected to be added: • Managing measurement elements • Custom ontology: Making ontology used for requirements such as making the verb list customizable for organizations or defining a pre-determined set of objects to use for entire organization. This will enable organizations better tune the interpretation of their requirements and also will let them introduce and enforce usage of their own requirement standards and wording guidelines. • Requirements suggestions engine (re-use, security): Certain frequent and typical requirements will be presented in the form of functional patterns. Certain generic functions such as security functions will be suggested based on type of identified objects. This will further prevent facilitate requirement development and prevent missing requirements. • More Machine Learning and better language interpretation. • Multilanguage support: It is expected to include other language than English for requirement interpretation. • Local Benchmarking: Currently certain reports utilize generic ratios and productivity values from public benchmarking data sets. It is planned to enable organizations utilize their internal benchmarking data sets and standard values. • Test suggestions: The tool will suggest a list of potential unit tests for any given method on an object. cases for requirements.