Quality requirements is a difficult concept in software projects, and testing software qualities is a well-known challenge. Without proper management of quality requirements, there is an increased risk that the software product under development will not meet the expectations of its future users. In this paper, we share experiences from testing quality requirements when developing a large system-of-systems in the public sector in Sweden. We complement the experience reporting by analyzing documents from the case under study. As a final step, we match the identified challenges with solution proposals from the literature. We report five main challenges covering inadequate requirements engineering and disconnected test managers. Finally, we match the challenges to solutions proposed in the scientific literature, including integrated requirements engineering, the twin peaks model, virtual plumblines, the QUPER model, and architecturally significant requirements. Our experiences are valuable to other large development projects struggling with testing of quality requirements. Furthermore, the report could be used by as input to process improvement activities in the case under study.
projects. In a large study on challenges related to alignment of Requirements
Engineering and Testing (RET), Bjarnason et al. report that three out of six studied
companies struggle with verifying QRs [
In our previous work on reported general challenges of RET alignment in the
public sector [
Our main contribution is to report five major challenges in verifying QRs in the
case under study, namely: 1) changing RE documentation, 2) test managers’ need of
domain understanding, QRs are neither 3) quantified nor 4) prioritized frequently
enough, and 5) simulation of complex operational states in a test environment. We
also propose solutions based on the research literature for the five challenges,
including improved commination between requirements engineers, architects and test
managers (i.e., extending the Cleland-Huang et al.’s “twin peaks” model [
The paper is structured as follows: Section 2 introduces previous research on QRs, and particularly related work on testing QRs. Section 3 describes the case company and the relevant development processes applied in the project under study. Section 4 presents the research method and the limitations of our work. Section 5 reports our results and our corresponding interpretations. Finally, Section 6 concludes our paper by summarizing the main takeaways. 2
While management of QRs is known to play a crucial role in large software
engineering projects, the RE research community has not agreed on a single definition [
Several researchers highlight the importance of QRs during software product
development. Simply implementing all functional requirements is not enough to ensure a
successful product; Disregarding QRs might lead to a product that is too difficult to
use or too expensive to maintain [
Software quality models have been developed to support identification of QRs and
to help establish control criteria for quality assurance. Two of the most established
models are ISO/IEC 25010 [
Previous work highlights QRs as among the most challenging aspects to manage
during software projects. Moreover, Chung et al. report that development
organizations often do not properly acknowledge the significance of QRs [
A number of studies point out more specific challenges of QRs. Ernst and
Mylopoulus report that software quality aspects are often first considered at late stages of
the development lifecycle, typically assessed by a set of measures applied to the final
product [
Functional suitability Performance efficiency Compatibility Usability
ISO/IEC 25010
Completeness, Correctness, Appropriateness Time-behaviour, Resource utilization, Capacity Co-existence, Interoperability Recognizability, Learnability, Aesthetics etc.
Maturity, Availability, Fault tolerance, Recoverability Confidentiality, Integrity, Accountability etc.
+
Capability, Reusability, Security Aesthetics, consistency, responsiveness etc.
Predictability etc.
Speed, Efficiency, Throughput, Scalability etc.
Testability, Flexibility, Installability etc.
ity, Testability etc.
Adaptability, Installability, Replaceability
Table 1. Side by side comparison of ISO/IEC 25010 and FURPS+. 2.2
Even in organizations that acknowledge the value of QRs, testing them is often a
problem. Berntsson Svensson et al. report in an interview study with 11 companies
that practitioners frequently fail to specify QRs in quantifiable formats that can be
verified by a testing organization [
In a multi-unit case study by Bjarnason et al., testing QRs is highlighted as one of
16 challenges in RET alignment [
Several researchers have presented approaches to support testing QRs. The
QUPER model helps development organizations set appropriate quality targets in a
market-driven context, and Berntsson Svensson and Regnell proposed adding also test
results to the model to support verification of QRs [
We study a large software project at a governmental agency in Sweden. The first
author has extensive experience as a consultant at the agency, as also reported in our
publication [
The new platform will combine several existing systems, integrating all steps of
the application process into a system-of-systems [
Figure 1 shows an overview of the integration platform under development. Instead of having all 12 partaking systems communicate directly with each other (a) in Figure 1, an integration platform approach is used (b), which is detailed in (c). Each of the 12 systems, as well as each integration between a system and the integration platform, are specified in a separate requirements document.
Two of the most important quality attributes for the new SoS are interoperability and performance. The SoS comprises multiple dependencies, and most external interfaces will be connected through a common integration platform (cf. b) in Figure 1). Reliable integration with services that provide data is essential, thus providing high quality APIs is an explicit requirement on the SoS. Also, as the number of future users is high, approximately 500 simultaneous users during the peak periods, the network performance is critical. Additionally, the data validations can sometimes require extensive resources, which can affect the performance of the IS, e.g., response times.
We define two key stakeholders of the SoS. End users are domain experts working at either GOV or county administrative boards across Sweden. They have detailed knowledge about the application process and assist clients. Clients (i.e., individuals and enterprises active in the sector) use the SoS to apply for subsidies. Furthermore, we define the business as the organization at GOV that will use the SoS to fulfil their operational needs. The IT department of GOV employs business analysts, i.e., requirements engineers responsible for elicitation and specification of the requirements, as well as software developers and system architects.
Analogous to other governmental agencies in Sweden, new development as well as operations at GOV is governed by framework agreements with subcontractors. Several subcontractors develop a large fraction of the SoS together, and also perform RE and testing activities. The development organization employs approximately 100 development engineers in total. Development of the new SoS is organized as 12 different projects, reflecting the number of systems to be integrated, employing roughly 10-20 engineers per project.
The development process at GOV is based on the Rational Unified Process
complemented by agile practices expressed as six goals: 1) End users and the business
work closely together, 2) High quality through continuous integration and automated
testing, 3) Incremental development enables frequent acceptance testing, 4)
Continuous delivery of system documentation, 5) Development teams are cross-functional
with integrated competences, and 6) Retrospective meetings at every sprint planning.
GOV uses the FURPS+ model to elicit and specify QR [
Planning within the development project at GOV is performed on three different
levels. Between 6-12 months, strategic planning, which involves resource allocation
and specification of dates for integration, referred to as milestones. Operational
planning (planning which covers more than 3 sprints). Finally, sprint planning (2 weeks)
deals with detailed planning of what is to be delivered in the sprint. Further details on
the development practices at GOV are reported in our previous paper [
This paper primarily documents experiences from the first author. However, we complement the experiences with an analysis of documents at GOV. 4.1
All experiences reported in this paper belong to the first author, and do not necessarily reflect the views of neither Capgemini nor GOV. Larsson is a software engineering consultant specializing in testing, especially test processes and test management. He has worked in software development projects in the public sector for more than a decade in Sweden and Denmark, with extensive experience of public sector development of large information systems. Furthermore, he has consulted in RE, mainly requirements elicitation and analysis. Note however that the experiences shared in this paper mainly reflect the perspective of a test manager. 4.2
The archival analysis of this work is dominated by qualitative analysis. The second
and third authors independently analyzed project documentation at GOV to find
support for the experiences reported by the first author. The analysis of GOV’s archival
data, also referred to as content analysis, was conducted on a subset of the available
project documentation. We studied all available documents describing the
development process at GOV, and selected one of the twelve systems in the SoS for in-depth
analysis of all parts and requirements. This is complemented by a light-weight
analysis of three additional systems where the requirements are sampled. The emphasis on
what is in the project documents rather than how they were created is inspired with
the concept of artifact-based RE as described by Méndez Fernández and
Penzenstadler [
The primary purpose of this paper is to present the first author’s personal experiences. Consequently, our paper is subject to the general limitations of experience reports; Generalization from our findings is uncertain. While we do not discuss extrapolating beyond the case under study, it is relevant to question how general the experiences of the first author are among his co-workers. Other engineers at GOV might highlight other challenges, and also prioritize them differently. However, we increase the validity of our findings by independent data source triangulation, i.e., the second and third authors performed document analysis to identify supporting evidence in GOV’s project documentation. 5
This section first presents findings from the archival analysis, and then reports the first author’s experiences from testing QRs at GOV. 5.1
The development project under study encompasses an extensive document structure. Documents at GOV are hierarchically organized in overall development process guidelines and project documentation for the individual systems. The general quality of the documents is high, and GOV adheres to good practices such as careful change management and consistent templates. All documentation is in Swedish.
Figure 2 shows an overview of the information flow related to QRs at GOV, from RE to the left, to testing at the right side. First, the business analysts create the Quality Requirement Baseline (QRB), an overview of the required system qualities, and the use cases based on 1) the European Union and Swedish legislation, 2) the overall business model at GOV, and 3) various process models describing high-level usage of the system under development. The system architects create the system’s architecture specification, complemented by a set of Interface Descriptions (IDs) for the relevant inter-system collaboration. The output from the business analysts and system architects is used as input for the software developers (cf. the lower part of the figure, not in focus of this study). Apart from the developed software system (cf. the white box), the test manager uses the QRB and IDs as direct input for testing QRs. The use cases are also relevant; the use cases must be understood to setup a realistic test environment, i.e., most QRs cannot be tested in isolation, rather the test manager needs to establish representative test scenarios in which to verify the QRs.
As illustrated in Figure 2, two artifact types constitute the key information flow of QRs from RE to testing. First, the Quality Requirements Baseline (QRB) is a single document that encompasses the overall qualities of the integrated SoS, i.e., the QRB is the quality centerpiece from an RE perspective. It is structured according to FURPS+, with sections for each corresponding category (see Section 2.1). Examples of system qualities specified in the QRB include: 1) acceptable downtime, 2) recoverability and startup performance, 3) response times, and 4) the number of simultaneous users and actions. Second, the Interface Descriptions (ID) refer to a set of documents specifying how the individual systems shall interact within the SoS. Each interface description specifies the behavior of both the producing and consuming end, as well as the transmitted messages. Examples of QRs for the producers include: 1) size and complexity of the messages, and 2) security. Examples of QRs for the consumers include: computational performance, e.g., 1) number of messages processed per second, and 2) time behavior, e.g., time to validate received data.
As a majority of GOV’s QRs are specified in the QRB and the IDs, we restrict the
discussion in the paper to these two artifact types. This focus mirrors the approach
taken by test managers when planning verification of GOV’s quality goals. There are
about the same number of FRs as there are QRs in the QRB and IDs (note that the ‘F’
in FURPS+ represents “Functionality”). The FRs are typically high-level textual
requirements in natural language, e.g. “The system should handle payments of
environmental support”. The QRs are written in a similar manner, e.g., “Response time
should not exceed 1 second when searching and registering information on a
particular issue”. While all FURPS+ categories are represented in the QRB usability and
implementation/architectural requirements are the biggest (i.e., ‘U’ and ‘+’ in
FURPS+). The level of QR quantification varies depending on type, as reported also
in previous work [
The QRB, the central quality document, provides an overall structure to work with QRs at GOV. The categorization of QRs into FURPS+ helps the test managers to identify the key QRs of each category (e.g., usability and reliability) and to plan the verification activities accordingly. Still, testing QRs is far from trivial at GOV.
The IDs, specifying integration within the SoS, are the fundamental artifacts to verify QRs related to the data flows. Early specification of data flows enables early planning of QR verification, but requires cooperation between the system architects and test managers. While some of the IDs reached a stable state early on in the project, others are stabilized later in the process. Hence, as the requirements change, the test plan must be updated to reflect the changes.
Typically, the IDs contain a subset of QR from the QRB, but elaborated for the specific interface. Not all QR from the QRB are applicable though, e.g., usability (as the IDs do not describe any user interfaces). On the other hand, other QRs become more important, e.g., efficiency (as the IDs are specifying integration into a SoS, covering network throughput, response times etc.).
Challenge 1: The RE documents evolve while testing is planned and ongoing. Challenge 2: Test managers need to understand the business.
Challenge 3: QRs are not quantified.
Challenge 4: QRs are not prioritized.
Challenge 5: Hard to simulate all operational states.
Table 2. Main challenges (Ch 1-5) in testing quality requirements at GOV.
Table 2 summarizes our experienced challenges of testing QRs. First, the QRB and
the ID are repeatedly updated during development, thus the test managers do not
know when the artifact is stable. As a consequence, early test plans might target the
wrong quality goals, or miss out on verifying entire quality aspects. This confirms
previous research, stating that QRs are often discovered late during development [
Second, to plan the verification, the test managers must have a broad
understanding of the business. The QRs can typically not be interpreted in isolation. Rather, the
test manager must understand the domain and possess insights in the data flow
through the SoS. This is further complicated by the fact that the QRs are not specified
in their context. Rather, they are written as part of the QRB or the ID and in a
structure related to the type of requirement rather than their relation to the other
requirements and parts of the system. Consequently, the testers must identify and understand
the context themselves. In addition, the importance of domain knowledge to software
testing is well-known in the literature [
Third, QRs are often not quantified, thus it is difficult to generate appropriate test
data. Vague QRs is a frequently reported issue in RE research [
Fourth, QRs are not prioritized. As a set of QRs typically involves
interdependencies [
Fifth, it is hard to generate test data that reflects the various operational states of the future SoS, as required for e.g. load and performance testing. The QRs expressed in the IDs put constraints that require the test organization to simulate various states using stubs and mock objects; different QRs need different operational states to be verified, and verification of certain QRs might require considerable provocation of the SoS. Some states are difficult to create in a lab environment, e.g. user interaction that returns a specific expected result within a given timeframe, simultaneously as real time data is exchanged over some interfaces with other systems. Another example is verification of complex business logic that is run concurrently in parallel activities. The validation of transmitted messages within the SoS requires heavy read and write to databases, which affects response times of systems retrieving data from the same databases. Currently, the QRs do not specify which use cases can run in parallel or what the quality level should be under such constrained conditions, making both design of test cases and evaluation of test results challenging. 5.3
GOV is already addressing the five reported challenges to some extent. One of the
main initiatives is to get different experts and roles to work closer together. Inspired
by integrated RE [
Transferring knowledge of various operational scenarios at GOV to the
development project (including the testing activities (Ch 2)) should be facilitated further.
While Bjarnason et al. state the importance of communication between requirements
engineers and testers [
One solution proposal from the literature is the QUPER model, an approach to
support setting quality targets [
A common trade-off in software engineering is having complete information
available when you start an activity versus being responsive to a changing
environment. GOV faces the same challenge, as explained in Ch 1: “The RE documents
evolve while testing is planned and ongoing” and Ch 4 “QRs are not prioritized”.
However, not all QRs have an equal impact on the system [
Verification of some QRs at GOV requires monitoring various aspects of the SoS
over time. However, interpreting the results of monitoring is still not a simple task.
Performance and load testing of an interface not only require monitoring of both the
producing and consuming sider; As the business flows gets more complex, the
number of interfaces grows, thus monitoring of other applications must also be set up
(related to Ch 5). For example, monitoring of the database server has proven to be
fundamental. Especially in the iterative integration of more systems into the
integration platform, values need to be monitored and regression tests run not once but
several times as the system changes. Another possibility to verify QRs over time at GOV
could be to implement “virtual plumblines” as suggested by Cleland-Huang et al.
[
This paper reports the first author’s experiences from testing QRs at a large governmental SoS development project in Sweden. We highlight five challenges involved in testing QRs, and complement the experiences by supporting evidence from an analysis of project documentation. The five main challenges in testing QRs at GOV are: 1) RE documentation evolves during testing, 2) test managers need extensive understanding of the business, 3) QRs are not quantified, 4) QRs are not prioritized, and 5) simulating all operational states is difficult, challenging testing prior to deployment.
To help GOV tackle these challenges, we match them with solution proposals from the research literature. Evolving QRs is a well-known challenge in software engineering, and could be addressed by an integrated RE approach, i.e., plan for RE to continue throughout the entire development project (Ch 1). Strengthened communication between the stakeholders from the “twin peaks” (RE and architecture) and the test managers would support RET knowledge transfer and specification of testable QRs (Ch 2). Moreover, improved communication among roles, e.g. business, architects and test managers, could help identifying architecturally significant requirements, e.g., to prioritize QRs that have considerable impact on the test planning (Ch 4). The QUPER model could be a tool to stimulate discussion on quality levels and support quantification (Ch 3). GOV already monitors several quality measures, but the process could be further inspired by “virtual plumblines” to quickly detect quality deterioration (Ch 5).
Future work could systematically explore which of the solution proposals GOV considers most promising to support testing QRs, and also try to implement and evaluate the corresponding process improvements.