=Paper= {{Paper |id=Vol-1938/paper-ket |storemode=property |title=Identifying Up-to-the-Minute Topics in Software Process Improvement Research |pdfUrl=https://ceur-ws.org/Vol-1938/paper-ket.pdf |volume=Vol-1938 |authors=Harri Keto,Hannu Jaakkola |dblpUrl=https://dblp.org/rec/conf/sqamia/KetoJ17 }} ==Identifying Up-to-the-Minute Topics in Software Process Improvement Research== https://ceur-ws.org/Vol-1938/paper-ket.pdf

Identifying Up-to-the-Minute Topics in Software Process
Improvement Research
HARRI KETO AND HANNU JAAKKOLA, Tampere University of Technology

The development of standards and methods used in software process improvement has for many years been oriented to the
identification of best practices, but recent trends show an increasing interest in new research topics, especially in the category
of small-to-medium-sized enterprises (SME) and very small entities (VSE). What are these research areas, and what kind of
relationship do they have with software process improvement? The goal of this study is to aggregate topics that are up-to-the-
minute or very new openings in research from the viewpoint of SMEs and VSEs. Using a systematic mapping study and
confining it to the last three years, 26 studies were selected for deeper analysis. Proposals for further studies are made by
suggesting practically relevant research questions. Results indicate that software process improvement research remains a
robust discipline with plenty of possibilities. The challenges can be met by co-operating with researchers from other disciplines.
Categories and Subject Descriptors: D.2.9 [Software Engineering]: Management—Software Process Models;
General Terms: Mapping study; Software process improvement; very small entities; small-to-medium-sized enterprises
Additional Key Words and Phrases: SPI, VSE, SME

1. INTRODUCTION
As part of the larger quality management approach called continuous process improvement, software
process improvement (SPI) has become a serious engineering discipline for improving the quality of
software engineering and its products. According to [Sommerville 2011, p.706], software process
improvement means understanding existing processes and changing these processes to increase
product quality and/or reduce costs and development time. Today, there are many international SPI
standards and disciplined methods that are used in the field. For many years, the development of SPI
standards and methods has been oriented to the identification of best practices and their mapping to
form a coherent framework like CMMI, ISO/IEC 15504, or ISO/IEC 29110. However, a recent trend
shows increasing interest in new areas like customized SPI models and the combination of agile
methods and process improvement [Kuhrmann et al. 2016]. These do not contradict each other. On the
contrary, recent literature studies also show an increasing interest in the merging of different
approaches.
The goal of this study is to identify up-to-the-minute research topics and presumptions about the
overriding knowledge bases of SPI in the context of very small entities (VSEs) and small-to-medium-
sized enterprises (SMEs). The European Commission has defined the category of SME to consist of
micro enterprises (staff < 10), small enterprises (staff < 50), and medium-sized enterprises (staff <
250) [European Commission 2005]. ISO/IEC JTC1/SC7 Working Group 24 has defined the acronym
VSE to refer to an enterprise, organization, department, or project having up to 25 people [ISO 2016].
A systematic mapping study was chosen as the research method to be applied [Petersen et al.
2015] [Kitchenham et al. 2011]. We have limited the study to research papers that have been
published since 2015 in journals or conference proceedings. The emphasis is on issues, thus the
number of factors or studies is not our main interest. If the results are encouraging, it is possible to
enlarge this study to cover a longer period and spread the search for a more complete set of papers.
This study is limited to small-to-medium-sized enterprises (SMEs) and very small entities (VSEs).
There are two main reasons for this. Firstly, a considerable number of software organizations belong
to this category [Richardson and Cresse von Wangenheim 2007][Larrucea et al. 2016]. Secondly, many

Author's address: Harri Keto, Tampere University of Technology, PL 300, 28101 Pori, Finland; email: harri.keto@tut.fi; Hannu
Jaakkola, Tampere University of Technology, PL 300, 28101 Pori, Finland; email: hannu.jaakkola@tut.fi.

Copyright © by the paper’s authors. Copying permitted only for private and academic purposes.
In: Z. Budimac (ed.): Proceedings of the SQAMIA 2017: 6th Workshop of Software Quality, Analysis, Monitoring, Improvement,
and Applications, Belgrade, Serbia, 11-13.09.2017. Also published online by CEUR Workshop Proceedings (CEUR-WS.org,
ISSN 1613-0073)
7:2 • H. Keto and H. Jaakkola

SMEs and VSEs have implemented agile practices in their development process to a greater or lesser
extent. The influence of the agile paradigm poses challenges for SPI approaches. As some papers did
not explicitly define the acronyms SME and VSE, it was unrealistic to make reliable deductions about
the data between the SMEs and VSEs. The risk is that the categories do not separate the cases in a
controlled way.
This paper is structured as follows. In section two, we present related studies in brief, which have
been performed recently. In section three, we describe the research design and implementation. The
results are presented in section 4. In the last chapter, we complete the paper with a discussion and
proposals for further research.

2. RELATED STUDIES
In this section, we briefly present recent literature reviews closely related to our study.
In [Kuhrmann et al. 2016] a large systematic literature study is reported where 769 publications
were analyzed. The study covers the field of SPI over a period of more than 25 years, presents a big
picture of the state-of-the-art, and maps out directions for future research. There are no up-front
constraints concerning the organizations, SPI methodology, or software engineering approach. The
companies were scaled in three categories: VSE/SME, other company sizes, and global software
engineering organizations (GSE). According to the study, the academic community shows a growing
research interest in SPI in the context of very small entities and small-to-medium-sized enterprises.
While standard assessment and maturity models like CMMI and ISO/IEC 15504 are considered to be
well researched, other main research trends focus on new/custom models, success factors, and agile
and lean development practices as part of SPI.
There are examples of literature studies that are confined to research on the combination of agile
and maturity models. From the viewpoint of CMMI, a systematic literature review in [Henriques and
Tanner 2017] brings out two major themes in this context: the coexistence of the agile with CMMI,
and the development of agile principle based maturity models. After excluding and reviewing, the
final working set was 39 articles. One of their suggestions on future research topics is how to attain
higher levels of CMMI maturity using only agile methods. In [Selleri Silva et al. 2015], a similar
research interest is reported.
In [da Silva and Carneiro 2016] the systematic literature study focuses on which difficulties are
faced by these companies regarding SPI. The study is limited to SMEs. After filtering the relevant
articles, 36 articles were left for further investigation. The study indicates that discussion and
research on the problems and success factors of SPI implementation in SMEs are relevant and
current.

3. RESEARCH DESIGN AND IMPLEMENTATION
In this section, we describe the research design and implementation. First, the research method is
presented and justified. Then, after describing the research questions, we explain the research steps
and make remarks about considerations and constraints concerning each step.

3.1 Research method
The research method is based on a systematic mapping study. According to [Petersen et al. 2015], a
systematic mapping study is designed to give an overview of the research area by classifying and
counting contributions in relation to the categories of the classification. In a nutshell, according to
[Kitchenham et al. 2011], the goal is the classification and thematic analysis of literature, the
research questions are generic, and only classification data is collected. Quality evaluation is not
essential and it is complicated by the inclusive nature of the search. The result is a categorized set of
papers related to a topic area. We make an exception in the method such that although we count the
Identifying Up-to-the-Minute Topics in Software Process Improvement Research • 7:3

papers per category, the number of papers is interpreted as being irrelevant. We justify this by the
fact that even an individual contribution can be relevant from the point of view of further study. We
restrict this study only to the past few years, so the identification of wide trends is not possible either.
As for trends, we refer to related studies in section 2.
We have applied the following five research steps:
1) Defining research questions
2) Search for studies
3) Study selection
4) Classification of papers
5) Analysis
The process has been very iterative. Re-reading was necessary because two of the four classes were
partly based on analysis. We used two general classification categories and two categories were
deduced by analyzing the papers. Analysis was performed by making cross-reference tables between
the categories.

3.2 Defining the research questions
The primary goal of this mapping study is to create a preliminary categorization of up-to-the-minute
SPI research topics in the context of SMEs and VSEs. The second goal is to identify the main
presumptions about the overriding knowledge bases of SPI. In other words, what are the trusted
premises of SPI? The main research question is expressed as openly as possible. Instead of referring to
predefined SPI themes, we try to identify relevant topics by recognizing the primary goals of the
studies. Then, within the material found, assumptions about the source of SPI knowledge and
proposals for further research can be identified. In Table I, we present three research questions and
their respective goals.

Table I. Research questions and respective goals
Research Question Goal
RQ1: What kinds of practical challenges of SPI can To illustrate the primary challenges which a software organization has
be identified from the viewpoint of a software to overcome. These challenges reflect potential current research topics.
organization?
RQ2: What presumptions about the sources of SPI To illustrate assumptions about the characteristics and bases of
knowledge can be identified? knowledge behind the SPI approaches.
RQ3: What kinds of SPI-related research subjects are To illustrate potential and current SPI research subjects.
suggested or what subjects can be implicitly derived?

RQ1 reflects the topics that are relevant from the viewpoint of software organizations to overcome
SPI challenges. By identifying the primary goals of SPI research, we can illustrate what the practical
challenges behind the research are. RQ2 concentrates on the knowledge sources of SPI, i.e., where the
overall understanding about the means, goals, and resources of SPI initiatives comes from. Practical
research subjects can be derived from practical challenges and therefore RQ1 is a forerunner of RQ3.

3.3 Search for studies
To retrieve relevant research papers, a search string was defined. The common term “software process
improvement” is a generally used expression. We expanded the possibilities by simplifying the string
to “process improvement”. “Software” can be left out by restricting the query to the field of science. If
there are extra papers on the result set, they can be excluded manually. The term “process
assessment” is used in papers which concentrate only on the assessment step. We used this as an
alternative to “improvement”. The other component of the query is the size of the organization. We
first assumed that if the acronyms VSE and SME are used in the abstract, they should also be
7:4 • H. Keto and H. Jaakkola

explained. The test query showed that this is not true in some cases. As a result, we also used the
acronyms in the query.
The search string was formed as follows: ("process improvement" OR "process assessment") AND
("small*” OR “VSE" OR “SME”). The searches performed on the databases of Scopus, IEEE Xplore,
and ACM on June 10 2017. The search targets were abstracts. The subject area was limited to
computer science and engineering. Only papers published in conference proceedings or journals were
retrieved. The year of publication was restricted to 2015 or later. After removing the duplicates, a
total number of 54 studies were found. Mendeley reference manager software was used to detect
duplicates.

3.4 Study selection
The selection of relevant studies was made by filtering the papers found by using inclusion and
exclusion criteria. We considered the following two inclusion criteria:
IC1: The study presents or discusses methods, approaches, standards, scenarios, research
challenges, or opportunities of software process improvement (SPI).
IC2: The study presents research in the context of very small entities (VSEs) or small-to-
medium-sized enterprises (SMEs).
For those publications that passed the inclusion criteria, five exclusion criteria were applied to reduce
the publications to only those that were directly aligned with the focus of the study. These criteria
were:
EC1: The paper is not directly related to software engineering.
EC2: The paper is not directly related to process improvement.
EC3: The paper is not a primary study, but uses other studies as a main source.
EC4: The paper is a previous version of a more complete study about the same research.
EC5: The paper is not written in English.
After using the inclusion and exclusion criteria, 26 articles from the 54 were left for deeper analysis
and classification (See the APPENDIX)

3.5 Classification of papers
We used three groups of qualities to classify the papers:
C1: Category of SPI approach.
C2. Primary goal of the study.
C3: Presumptions about the primary source of SPI knowledge.
C1 is a general category which has been used by other researchers. The remaining categories C2 and
C3 were drawn up by analyzing the content of the papers.
Pettersson et al. divide SPI approaches (C1) into two categories: prescriptive top-down and
inductive bottom-up approaches [Pettersson et al. 2008] [Gorschek 2006]. We refer to these categories
as the P category (prescriptive) and the I category (inductive). The P category frameworks are based on
recognized best practices, which have been documented as a standard model. Frameworks like CMMI,
ISO/IEC 15504, and ISO/IEC 29110 belong to this category. There is an assumption that the more an
organization’s processes matches the requirements of the model, the higher its level of maturity or
process capability. In the I category methods, the focus of the first step is to understand organizations’
current situation (processes) and to determine the causes of significant problems. The improvement
efforts are focused on the most critical issues from the organization point of view. Examples of the
inductive approach are the Experience Factory (EF), the Quality Improvement Paradigm (QIP) [Basili
1995], and the retrospective meeting of Scrum [Schwaber and Beedle 2001]. Using an inductive
approach, the basis for improvement comes from organizational experience and knowledge rather
than a normative reference model. Inductive methods are adapted to the current state of software
organizations [Pettersson et al. 2008]. The category of SPI approach was concluded by identifying the
primary referential background of the framework. There were 14 articles on the P category and 10
Identifying Up-to-the-Minute Topics in Software Process Improvement Research • 7:5

articles were classified into the I category. Furthermore, two articles were identified as having a
mixed approach (P/I).
The second category C2, the goal of each study reflects the research problems RQ1 and RQ3. We
interpreted the goal of each study from the viewpoint of the software organization. This way we
emphasize the practical value of the papers. This was carried out by first writing down the goal of the
research in one sentence. Then, by using an iterative reading of both the article and the goal, we
picked out the keywords which mostly reflected the practical interest of the software organization. We
used the following criterion for the keyword: it should be possible to form a question using the words:
“How to … SPI”. For example, with the keyword “enable” we can ask “How to enable SPI in an
organization?” Table II shows how the keywords are divided into P and I category approaches.

Table II. Challenges of SPI in VSEs and SMEs
SPI approach Primary goal / Challenge (How to… SPI?)
P category adopt, adopt/integrate, enable, integrate, self-assess, simplify/tailor, simplify/integrate
I category enable, identify, integrate, learn, perceive
P/I category adopt, model/tailor

After a careful revision, 10 keywords were identified: adopt, enable, identify, integrate, learn,
model, perceive, simplify, self-assess, and tailor. There were four articles where we had to use two
keywords instead of one to capture the main message. Two articles were classified as “simplify/tailor”
because their message was “How to simplify a standard model and tailor it for practical purposes”.
The other two-part keywords were “adopt/integrate,” “integrate/simplify,” and “model/tailor.”
The third category C3 is presumptions about the primary source of SPI knowledge. In other words,
the knowledge base that is trusted as an overriding source of SPI expertise. This was identified by
keywords that answer research question RQ2: What presumptions about the sources of SPI knowledge
can be identified? These “SPI drivers” were categorized into SPI standard, people, organization,
method (other than SPI standard), and multi-factor.
The overriding sources of SPI knowledge that were found are presented in Table III. The category
of SPI standard consists of industry standards, which are used together (multi-model) or are somehow
tailored or simplified to adapt to the context. The organization category refers to properties that refer
to organization-specific factors. The category of people refers to human properties. The category of
method consists of approaches other than standards. The research category collects success factors
that are identified and reported by researchers, mostly in case studies.

Table III. Primary source of SPI knowledge
Category Primary source of knowledge
SPI standard lightweight model, multi-model, standard, tailored model
Organization business objectives, information flows, organization environment, organizational characteristics,
organizational elements, organizational knowledge, practical needs, process modeling
People experience, human factors, interpersonal skills, beliefs, tacit knowledge, professional knowledge,
culture
Method agile approach, lean approach, total quality management (TQM)
Research success factors

The categories of the primary sources of SPI knowledge overlap. For example, success factors in
the research category include factors from the other groups. In that regard, the categories are not
theoretically valid; they are merely the subjects of overlapping groups.
7:6 • H. Keto and H. Jaakkola

4. RESULTS
The categories and data presented in the previous chapter are analyzed by making cross-reference
tables between the categories. The results are collected here under the research questions.

RQ1: What kinds of practical challenges of SPI can be identified from the viewpoint of a
software organization?
In Table II, the challenges are presented by grouping them according to the SPI approach. In the P
category, there are challenges about how to enable or adopt standard-based SPI in the organization,
and how to integrate different approaches. There are also the challenges of simplifying and tailoring a
standard-based approach. In addition, self-assessment is mentioned. Also, in the I category
approaches, the integration of different SPI methods is involved. Another similarity is the enabling
factors of an SPI initiative. In addition, there are challenges about the identification of the subject to
improve and challenges about learning during the SPI initiative.

RQ2: What presumptions about the sources of SPI knowledge can be identified?
We defined the primary source of SPI knowledge as referring to the knowledge bases that are trusted
as an overriding source of SPI expertise. In Table III, these trusted premises of SPI are categorized. In
a standard-based approach, a lightweight or tailored model, in addition to the standard itself, is seen
as the main driver of SPI. In a multi-model approach, two or more standards are used to form a
comprehensive and coherent framework. Each standard has its own purpose in the framework. In the
organization category, eight organizational properties or components of an operations model are
identified. The people category consists of seven human-centric properties. These human issues
together with the organization category refer to the fact that the methods of anthropocentric sciences
and organization study could offer information and additional possibilities for SPI research. The
method category refers to approaches other than standard SPI methodologies. The three identified
approaches are well known and accepted quality initiatives. The agile and lean are modern
approaches that concentrate on the customer-oriented production philosophy. TQM is a management
approach that forms an overall framework for quality measurement and improvement.

RQ3: What kinds of SPI-related research subjects are suggested or what subjects can be
implicitly derived?
Research question RQ3 is an offshoot of RQ1. The following research question proposals are derived
from the practical challenges facing software organizations. The list is organized by the category of
approach (C1).
- Proposed research questions of P category approaches:
o How to adopt a prescriptive SPI method?
o How to adopt a prescriptive SPI method and integrate it with other methods used?
o How to enable a standard-based SPI initiative?
o How to integrate a prescriptive SPI method with other methods used?
o How to carry out self-assessment?
o How to simplify a prescriptive method and tailor it for a project’s needs?
o How to simplify a prescriptive method and integrate it with other methods used?
- Proposed research questions of I category approaches:
o How to enable an inductive SPI initiative?
o How to identify the targets of improvement in inductive SPI?
o How to integrate an inductive method with other methods used?
o How to enable learning in an inductive SPI initiative?
o How to perceive progress in inductive SPI?
- Proposed research questions of P/I category approaches:
o How to adopt an SPI approach in a multi-model SPI initiative?
Identifying Up-to-the-Minute Topics in Software Process Improvement Research • 7:7

o How to model a process and tailor methods in a multi-model SPI initiative?
Challenges such as self-assessment, simplification, integration, and tailoring are interesting
research subjects from the viewpoint of VSEs and SMEs. Most prescriptive approaches presume an
outside evaluator or at least special expertise, which means extra resource costs. Self-assessment with
a simplified and tailored model is one way to challenge the thresholds of SPI. The tailoring of models
for specific needs and the integration of approaches brings flexibility, but at the same time there is a
risk that special expertise will be needed. Two challenging subjects stand out in the I category: the
identification of improvement targets and perceiving the progress of SPI. This might indicate that
inductive methods lack methodological support for these steps of the SPI process. However, since our
sample consists of just a few papers, reliable deductions cannot be made. SPI is not the playground of
a single method or approach. A lot of interest has also been focused on multi-model approaches and
integration.

5. CONCLUSION
Using a systematic mapping study and confining it to the last three years, 26 studies were selected for
deeper analysis. After classifying and analyzing the results set, proposals for further studies were
made by suggesting practically relevant research questions. Results indicate that research of software
process improvement is still a robust discipline with plenty of possibilities. By integrating the results
of RQ3 with the knowledge sources listed in RQ2, even more variety in research possibilities can be
obtained.
This study gives a good starting point for further study. Firstly, the timeline of the search should
be extended. The chronological scale of searched papers should be at least five years. Secondly, the
size of the target organizations should be narrowed down more. Our study covers data concerning
both SME and VSE software organizations. Other studies state that there are significant differences
between these categories [Richardson and Cresse von Wangenheim 2007] [Sánchez-Gordón and
O’Connor 2016]. Finally, our results show clearly that software process improvement is not just a
pocket of software engineering. The challenges can be met by co-operating with other disciplines. For
example, a comparative study of prescriptive and inductive approaches with other fields of
engineering could be fruitful.

REFERENCES

Victor R. Basili. 1995. The Experience Factory and Its Relationship to Other Quality Approaches. In Advances in Computers.
65–82. DOI:https://doi.org/10.1016/S0065-2458(08)60231-4
European Commission. 2005. The New SME Definition - User guide and model declaration,
Tony Gorschek. 2006. Requirements Engineering Supporting Technical Product Management. Blekinge Institute of Technology.
V. Henriques and M. Tanner. 2017. A systematic literature review of agile and maturity model research. Interdiscip. J.
Information, Knowledge, Manag. 12 (2017), 53–73.
ISO. 2016. ISO/IEC TR 29110-1 Systems and software engineering - Lifecycle profiles for Very Small Entities (VSEs) -- Part 1:
Overview., Geneva.
Barbara Kitchenham, David Budgen, and O. Pearl Brereton. 2011. Using mapping studies as the basis for further research - A
participant-observer case study. Inf. Softw. Technol. 53, 6 (2011), 638–651.
DOI:https://doi.org/10.1016/j.infsof.2010.12.011
Marco Kuhrmann, Philipp Diebold, and Jürgen Münch. 2016. Software process improvement: a systematic mapping study on
the state of the art. PeerJ Comput. Sci. 2 (May 2016), e62. DOI:https://doi.org/10.7717/peerj-cs.62
Xabier Larrucea, Rory V O’Connor, Ricardo Colomo-Palacios, and Claude Y. Laporte. 2016. Software Process Improvement in
Very Small Organizations. IEEE Softw. 33, 2 (March 2016), 85–89. DOI:https://doi.org/10.1109/MS.2016.42
Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in
software engineering: An update. Inf. Softw. Technol. 64 (2015), 1–18. DOI:https://doi.org/10.1016/j.infsof.2015.03.007
F. Pettersson, M. Ivarsson, T. Gorschek, and P. Öhman. 2008. A practitioner's guide to lightweight software process assessment
and improvement planning. J. Syst. Softw. 81, 6 (June 2008), 972–995. DOI:https://doi.org/10.1016/j.jss.2007.08.032
Ita Richardson and Christiane Cresse von Wangenheim. 2007. Why Are Small Software Organizations Different? IEEE Softw. ,
January/February (2007), 18–22.
7:8 • H. Keto and H. Jaakkola

Mary-Luz Sánchez-Gordón and Rory V. O’Connor. 2016. Understanding the gap between software process practices and actual
practice in very small companies. Softw. Qual. J. 24, 3 (September 2016), 549–570. DOI:https://doi.org/10.1007/s11219-
015-9282-6
Ken Schwaber and Mike Beedle. 2001. Agile software development with Scrum, Prentice Hall.
Fernando Selleri Silva et al. 2015. Using CMMI together with agile software development: A systematic review. Inf. Softw.
Technol. 58 (February 2015), 20–43. DOI:https://doi.org/10.1016/j.infsof.2014.09.012
Gledston Carneiro da Silva and Glauco de Figueiredo Carneiro. 2016. Software Process Improvement in Small and Medium
Enterprises: A Systematic Literature Review. In Information Technology: New Generations. Advances in Intelligent
Systems and Computing. Springer, 603–613. DOI:https://doi.org/10.1007/978-3-319-32467-8_53
Ian Sommerville. 2011. Software Engineering 9th ed., Boston: Pearson Education Inc.

APPENDIX

Access to the list of papers included in the analysis: https://tutfi-my.sharepoint.com/personal/harri_keto_tut_fi/_layouts/15/
guestaccess.aspx?docid=08278b0dc0a394e109103bf8b388fbdfb&authkey=ASlZjCCL9cDsFsjoBJ1dKYU