Our department has the role of promoting reform in the SI business of the NEC Group and improving the quality and productivity of products and services. To this end, we are working to manage technology and distribute know-how across organizations in relation to software and system production. This includes the drafting and implementing of policies for software and system engineering such as development methodologies. As quality requirements for IT systems become increasingly complex and diversified, the need is felt for a mechanism that can logically and objectively explain quality. One means of perceiving quality is to apply the ISO/IEC 25000 (SQuaRE) series of international standards, but at present, they are not being sufficiently used on-site in IT system construction projects. Going forward, it will be necessary to perceive quality not only from the viewpoint of process quality but also in terms of product quality based on customer concerns. In this paper, we report on the results of our activities in testing the effectiveness of applying the ISO/IEC 25000 (SQuaRE) series to IT system construction projects and in devising ways of applying them.
In recent years, IT systems have become such an indispensable part of people’s lives that failures in those systems have turned into major social problems. This state of affairs has forced the providers of IT systems and services to be held accountable and explain why those failures occurred.
At the same time, the role of IT systems in the corporate world is shifting from being just a tool to being a form of management itself, and as a result, the requirements placed on IT systems are becoming increasingly complex and diversified. Customers are becoming increasingly aware of quality, and a trend is emerging in which customers themselves are performing objective evaluations of IT system quality such as by using outside process management companies for development processes and quality management.
In Japan, the Information-technology Promotion
Agency (IPA) has made recommendations on the need
for suppliers of software products to explain software
quality to users premised on the ISO/IEC 25000
(SQuaRE) series of international standards (referred
to below as the “SQuaRE series”) [
Additionally, on examining the situation at System Integration (SI) project sites involved in the construction of IT systems, it can be seen that factors like sudden changes in IT technologies and short delivery times are making it difficult to properly apply quality management techniques that use “statistical bug prediction and management techniques” specific to type of development project, type of language, and type of organization taking the number of bugs to be a prime indicator. Techniques that place importance on process quality aim to achieve an indicator value in terms of the number of bugs, but this results in a situation in which quality is managed only from the viewpoint of software developers.
At present, quality awareness is rising among customers, so product quality that presumes process quality management is being reconsidered and the need is being felt for quality management techniques that focus on quality of interest to customers.
Quality of interest to customers is not limited to the
presence or absence of bugs but covers a wide range of
issues such as effectiveness in business tasks, ease of
system use, and reliability. Definitions of these various
aspects of quality have been made such as the Kano
model [
The SQuaRE series, however, are international standards for quality requirements of software products and for evaluating those requirements targeting software overall, and as such, suffer from the following problems.
• The SQuaRE series cover a wide range of standards and target a variety of software products, so the descriptions in those standards tend to be generic in nature. As a result, project personnel engaged in IT system construction encounter expressions that they are not familiar with and find difficult to understand. • Among Quality, Cost, and Delivery (QCD) in SI projects, cost and delivery are highly prioritized, and there are many cases in which they are decided on in advance, which makes it difficult to secure the cost and time to study and deal with the quality characteristics/subcharacteristics of all the quality models in the SQuaRE series. • Applying the general-purpose SQuaRE series of standards to SI projects requires tailoring them to the actual circumstances surrounding a particular project along with a thorough knowledge of SQuaRE (we ourselves spent about one year in achieving an understanding of quality models in the SQuaRE series and holding discussions on their application to SI projects).
to SI On applying the SQuaRE series to SI project sites, we were aware of the above problems, but we began with “definitions of quality of interest to customers” in IT system construction using the Quality Model Division (ISO/IEC 2501x) of the SQuaRE series.
To begin with, we set up a study team to learn
about the basics of the SQuaRE series using Japan’s JIS
standards [
Additionally, for the quality characteristics /subcharacteristics of the ISO/IEC 25010 (product quality) and ISO/IEC 25012 (data quality) quality models, we discussed our understanding of each of those characteristics/subcharacteristics (57 quality characteristics in all) and examples of interpreting them for application to SI projects.
These discussions were held over a period of about one year and included an exchange of opinions with a certain outside vendor that had experience in dealing with the SQuaRE series. Through these activities, we discovered the following policies in using the SQuaRE series of standards. 1. The objective is not strict classification or exhaustive use of quality characteristics /subcharacteristics To begin with, the quality models of the SQuaRE series cover a very wide range of quality characteristics and the need may be felt for using all quality characteristics /subcharacteristics in an exhaustive manner. However, it is often the case in SI projects that resources that can be allocated (cost) and delivery date are highly prioritized in addition to target quality. Consequently, it is realistic to just use these quality characteristics /subcharacteristics as a viewpoint in setting priorities for quality of interest to customers while keeping a QCD balance in mind (they can guide the thinking of personnel and be used as a reference for prioritizing quality requirements).
Next, the objective of many SI projects is not to obtain ISO third-party certification, so strict classification of quality characteristics /subcharacteristics is not recommended (increases costs). 2. Systematic use of the SQuaRE series from the upstream in consensus building with customers The SQuaRE series can be used to form an agreement on vendor quality requirements with customers and stakeholders in the project proposal and planning stage.
Quality management based on the management of indicator values such as number of bugs and review time are processquality centric with respect to work results from reviews, tests, etc. In contrast, by focusing on product quality of interest to customers, we believe that applying the SQuaRE series to a project in a balanced manner from the viewpoint of product quality and managing quality from multiple perspectives in the upstream—where many waterfall projects incorporate quality—can provide an IT system of even higher quality. 3.
Use in analysis when quality problems occur (not recommended) We believe that the quality model framework of the SQuaRE series can also be used for projects in progress or existing deliverables from the viewpoint of checking for excesses or deficiencies from a quality perspective along the way. From the beginning, however, it has been important to execute a project by turning quality requirements into specifications in consultation with customers and stakeholders from the project proposal/planning stage and reaching a consensus on assigning priorities and making measurements and evaluations. In this context, we do not recommend using the SQuaRE series when quality problems occur.
Using the relationships between quality
characteristics
Relationships exist between ISO quality
models and quality characteristics
/subcharacteristics as reported in IPA’s
“Software Quality Guide for a Connected
World” [
We present the following guidelines to effectively use these relationships between quality characteristics in SI projects (described in detail later). ⚫ ⚫ ⚫
investment of an SI project Negative effect: Results from implementing risk countermeasures in the operation of an SI project Quality derivation: Used in deriving related quality requirements (makes work more efficient and improves the accuracy of quality targets)
We investigated how to efficiently convey the results of our study team activities to personnel at SI project sites and prepared a guide based on the following policies. ⚫ ⚫ ⚫
Provide an SI-oriented explanation based on
SQuaRE definitions of quality characteristics
/subcharacteristics (original text)
Illustrate quality characteristics
/subcharacteristics in line with situations in
IT system construction in a way that SI
project personnel can easily visualize those
characteristics/subcharacteristics
Illustrate relationships between quality
characteristics (positive effects, negative
effects, quality derivation) and explain how
to use them in an SI project
Specifically, we created a guide that, assuming SI
projects, includes definitions of quality characteristics
/subcharacteristics in the quality models of the
SQuaRE series and explanations of those definitions
from IPA’s “Software Quality Guide for a Connected
World” [
Along with the above, we have added illustrations
of quality subcharacteristics in NEC SI projects to help
personnel in SI projects understand product quality
models (Figure 2: Example of a quality model guide for
SI project personnel (illustration of quality
characteristics)).
This guide also presents samples of the
relationships between quality characteristics
/subcharacteristics of different ISO quality models as
reported by IPA’s “Software Quality Guide for a
Connected World” [
First, recognizing that the following types of relationships exist between quality characteristics of different quality models, we present an example (Figure 3: Example of relationships between quality characteristics of different quality models).
Quality characteristics that achieve quality during use can be used to derive quality characteristics in the product quality model. These relationships can be used as reference when incorporating quality from the user’s point of view in specific quality characteristics in the product quality model.
In the case that quality in the product quality model is studied first, these relationships can be used as reference to study target quality during use in a retroactive manner. 3. Clarification of division between functions and data The division between quality achieved with functions (that include data) and quality achieved by data only can be clarified. These relationships can be used as reference when, for functions, an awareness of data characteristics is needed, and for data, when identifying characteristics to be linked with a function.
Next, there are positive effects and negative effects between quality characteristics /subcharacteristics. Given certain quality characteristics/subcharacteristics having a relationship, a positive effect occurs when improving the quality of one characteristic/subcharacteristic has a good effect on the quality of the other. In contrast, a negative effect occurs when improving the quality of one characteristic/subcharacteristic has a bad effect on the quality of the other.
For example, improving response time and raising performance so that screen transitions become smoother and user operability improves is considered to be a positive effect. On the other hand, introducing two-factor authentication to improve security increases the number of screens to be checked and the operations needed to reach the function one wants to use. This reduces user satisfaction and is therefore a negative effect.
We created a matrix showing these relationships and presented those relationships together with grounds explaining them (Figure 4: Relationship matrix between quality characteristics (Compatibility), Figure 5: Grounds explaining relationships between quality characteristics (Interoperability)).
We therefore believe that focusing on these relationships between quality characteristics and making designers aware of them can produce specifications for quality requirements that achieve a balance in a more efficient way.
However, the interpretation of quality characteristics and the relationships among them may change according to the nature of the target project, so tailoring them to the project is possible.
We ourselves conducted proof of concept (PoC) trials for several projects to gain insights from a quality perspective in executing the project and to check the effectiveness of risk management. In these trials, we used the viewpoints expressed by quality models in the SQuaRE series of standards in actual SI projects and classified and organized quality requirements (explicit/implicit needs).
Our purpose in conducting these PoC trials was to test the following hypotheses. 1. Visualization of quality requirements using quality models Can quality requirements that must be satisfied by a project be further clarified by classifying quality in terms of quality characteristics/subcharacteristics? ⚫ ⚫
recognized compared with that before quality classification? Can excesses or deficiencies in quality requirements be noticed? 2. Checking effectiveness in an SI project Could measures be taken to avoid project risk from quality problems extracted from the relationships among quality characteristics?
The projects targeted for PoC trials are summarized below. The time period of each trial was about a month and a half to two months according to each project’s schedule. ⚫ ⚫ ⚫ ⚫
Software as a Service (SaaS) development project PoC execution phase: planning phase IT system construction project including construction of machine facilities, etc.
PoC execution phase: basic design phase Government-related IT system construction project
PoC execution phase: basic design phase
On performing a PoC trial, we ourselves as persons knowledgeable about the SQuaRE series classified and analyzed quality requirements based on materials provided from each project. 1. Visualization of quality status by quality classification Using project documents including the proposal and definitions of requirements, we classified quality based on quality models of the SQuaRE series and visualized the quality status of the target project.
We clarified the quality problems (including speculations) and countermeasures that should be considered in the target project from quality status classified as described above and from relationships between quality characteristics (based on ISO definitions, research results, and SI considerations).
Based on the results of analysis, we proposed the following countermeasures (including speculations) such as risk hedges in executing a project.
Proposed that the presence of excesses or deficiencies in terms of completeness be checked from a quality perspective (prevent omission of requirements). “Deficiencies” from a quality perspective may indicate implicit needs, so we proposed that deficient points be checked to see whether they are indeed implicit needs. Proposed that “outside the scope” from a quality perspective may be excluded from the project (eliminate waste).
Proposed that quality goals be explained to stakeholders including customers using a visualized quality perspective (objective explanation of quality).
Quality status report (Figure 6: Sample of project quality report) This deliverable reported on the classification and analysis of quality requirements, proposals for risk countermeasures to be taken by the project from the perspective of quality characteristics, etc. Specifically, we focused on groups of related quality characteristics based on project characteristics from system requirements and analyzed why those trends occurred (Figure 6: Sample of project quality report (example of visualizing trends in quality characteristics) and Figure 7: Sample of project quality report (example of analyzing trends in quality characteristics)).
Next, among various quality characteristics (explicit/implicit), we visualized quality characteristics that have little mention in system requirements and proposed measures for future projects from the relationships between quality characteristics, etc. (Figure 8: Sample of project quality report (analysis of trends in quality characteristics: quality characteristics of concern) and Figure 9: Sample of project quality report (example of quality characteristics that a project should consider)).
Quality perspective checklists by type of design specification This deliverable for the basic design phase includes design perspectives and review points that take into account quality characteristics.
By proposing not only general measures for quality characteristics but also measures that consider project characteristics, we could increase motivation for risk avoidance in the project.
Since an understanding of the quality models and quality characteristics /subcharacteristics of the SQuaRE series would normally be assumed to complete quality perspective checklists for different deliverables, only check items with abstract expressions were created for personnel in PoC projects, so effects could not be confirmed. However, if a checklist is created that assumes no knowledge at all of the SQuaRE series, a huge amount of checklists including a variety of explanations would be created and there would be doubts as to whether such an amount of checklists could be realistically checked.
Consequently, to enable the use of checklists from a quality perspective, we determined that it would be best if SI project personnel were to first obtain an understanding of the quality models and quality characteristics/subcharacteristics of the SQuaRE series and to then use those checklists at the beginning of each phase with the aim of checking off specific design points.
Through activities that lasted for a year and a half, we found the following effects to be true by applying the SQuaRE series of standards to SI projects involved in the construction of IT systems.
⚫ ⚫
The visualization of quality requirements (classifying and presenting quality status) is effective in further clarifying quality status.
The application of SQuaRE series is effective in improving return on investment and avoiding risk in achieving project QCD.
In the above way, we confirmed that explaining IT system quality in SI projects is effective, but as described in policies on using the SQuaRE series in SI projects, building a consensus with the customer from the project proposal and planning stage is important and that the SQuaRE series should be used as a manual or bible for project quality during project execution and up to project completion.
This can be taken to mean “quality strategy,” and going forward, we plan to use not only the Quality Model Division (ISO/IEC 2501x) of the SQuaRE series but also the Quality Measurement Division (ISO/IEC 2502x), Quality Requirements Division (ISO/IEC 2503x), Quality Management Division (ISO/IEC 2500x), and Quality Evaluation Division (ISO/IEC 2504x). Furthermore, in addition to SI projects using a waterfall type of development, we plan to apply the SQuaRE series to DevOps (SI projects of the type in which factors such as agile development, sudden changes in IT technologies, and short delivery times make it difficult to properly apply “statistical bug prediction and management techniques” specific to type of development project, type of language, and type of organization taking the number of bugs to be a major indicator). 7. References