- As a lesson learned, the readiness review process of SCAMPI appraisal is very complicated and effort and cost consuming for novice organizations who need to know their status of CMMI practices classification. In SCAMPI appraisal, the readiness review process is a single process that runs from start to the end. During the readiness review process, nobody collected database to improve the process's performance. Our research towards the re-engineering of readiness review process aims to enhance the process performance and reduce effort/cost of the readiness review process implementation.
This paper presents our conceptual view of the R2P2 Lifecycle Model and a lesson learned from the preliminary development of the model. We are at the first stage of the R2P2 Lifecycle Model development by using 30 historical appraisals cases for root causes analyzing of the weakness of readiness review process. Therefore, we will use this lesson learned to enhance the R2P2 Lifecycle Model in next stage. Improvement,
Re-engineering process,
Process
Performance
INTRODUCTION
Since the needs of software solutions delivery are rapidly expanding in competition across countries nowadays, Capability Maturity Model Integration (CMMI) certificate is required by many software development organizations.
The Capability Maturity Model Integration (CMMI) [
claims that CMMI can be used to guide process improvement across a project, division, or an entire organization.
Standard CMMI Appraisal Method for Process
Improvement (SCAMPI) [
The readiness review process is typically used for CMMI practices classification either for self-appraisal by the internal team or for official appraisal (SCAMPI) by appraisal team. In each appraisal, stakeholders consisting of members from internal team and appraisal perform document reviews and conduct gap analysis.
Achievement factors of CMMI practices implementation are high-skill people, well-defined processes, supporting tools, a perfect work environment including process assets repository, and process improvement model. Typically, novice organizations rarely have strength in those required factors. Therefore, they spend a lot of money to hire consultants to settle and prepare the readiness review process for SCAMPI appraisal.
In additions, significant difficulties in SCAMPI readiness review process were from uncertainly defined documents verification criterion in SCAMPI manual on how to systematically characterize each direct artifact as “strength or weakness”. Consensus in the readiness review process was judged by participants. Those difficulties become our research motivation to reengineering the process by developing a so called “Readiness Review Process Performance Lifecycle Model” or in short “R2P2 Lifecycle Model”. The model is based on the lifecycle descriptive model for a gap analysis and readiness review procedure.
This paper presents our systematic re-engineering approach of R2P2 Lifecycle Model. The model is based on a capabilitybased reference model of CMMI. The R2P2 Lifecycle Model is an alternative solution to help organizations to perform the readiness review process and to measure the process performance as well as its efficiency by themselves. The results of R2P2 Lifecycle Model continuously provide predictability and reusability of the readiness review process performance to assess organizations. The presented model is evaluated using real assessment data from industry and the results show the potential of the method.
This paper is structured as follows: Section II presents related works. Section III describes the R2P2 Lifecycle Model. Section IV presents the development process of the R2P2 Lifecycle Model. Section V presents a lesson learned from the R2P2 Lifecycle Model development. Finally, Section VI concludes the summary and future works.
II.
RELATED WORKS
Process improvement frameworks such as IDEAL [
There are several existing assessment methods (e.g., [
In this paper, the R2P2 Lifecycle Model focuses on document review methodology and i-DocVer provides sample and checklist to map artifacts to their requirements. The input data is widely collected from various work environments such as software development methodology, different technologies, team size, document languages, organization structure, skill level, and types of work products. The total number of the organization is 49 appraisals in 6 countries.
Most of known solutions in practices of readiness review process and document verification are without structured model or easily implementable procedure to elicit and develop its performance criteria and measurement. Because of confidentiality agreement of appraisal, nobody collects and broadcasts the appraisals data for improvement of readiness review (RR) process in official appraisals to calibrate the RR process’s efficiency. The RR process composes of document review, team readiness, and logistics readiness. The R2P2
Lifecycle Model is a model description for readiness review process in CMMI appraisal. It is lifecycle model of quantitative management to improve process performance. The R2P2 lifecycle Model also contains precise descriptions of method’s contexts and best practices for documents verification guidelines that support better performance and more accurate result.
Several key challenges faced in the traditional readiness review process performance include:
1. RR process is very complicated. It is not defined with systematic methodology or easily implementable procedures for any novice organization to perform. Whenever stakeholders have conflict indiscretion, only consensus method is held on.
2. RR process is used only for one appraisal without reusable data from previous appraisals. Therefore, the process performance never manages continuously as an adaptive model to improve its performance and efficiency.
3. The number of GAP found and the number of information needed reflect effort/cost and scheduling of the appraisal planning. Number of GAP means the different number of founded finding e.g. “strength”, “weakness” between internal team and appraisal team.
The questions for how to solve and manage these problems are defined as following. What is the weakness of RR process? Then, how can we improve RR process performance and its quality? Finally, how can toolkit support to improve the RR process performance and its quality?
Therefore, we attempt to identify the root causes and the resolutions to improve the RR process performance thru the R2P2 Lifecycle Model with the following perspective: 1. The relevant critical factors are investigated by statistical process control methodology. Analysis results are used as inputs to solve problems to reduce GAP between stakeholders.
2. The R2P2 Lifecycle Model primarily relies on the principles and advantages of process performance management to elicit and establish these performance criteria and associated measures based on the efficiency of new readiness review methodology in either self-appraisal or official-appraisal.
3. The R2P2 Lifecycle Model aims to help novice stakeholders to simplify works either self-appraisal or CMMI appraisal by having a supported toolkit called “i-DocVer Toolkit” to assist them to classify a characteristic of direct artifacts effectively.
To specify study area to work on in this research, we focus on only CMMI Maturity Level 2 and 3 (18 PAs). The information of SCAMPI appraisals is during 2010-2017 (total of 49 appraisals) and all appraisal datasets are obtained by Appraisal Team Members (ATMs). This research emphasizes on the readiness review (RR) process and document verification methodology in the preparation phase of SCAMPI and self-appraisal for process performance and reduces GAP between stakeholders.
Relevant Stakeholders are categorized into 2 groups thus; C1: Self-appraisals: Software Engineering Process Group (SEPG) and Quality Assurance (QA) who investigate on CMMI classification activity as internal team.
C2: SCAMPI appraisals: Appraisal Team Member (ATM) who elect to perform cross-checking of all documents prepared during Software Development Life Cycle (SDLC). The core responsibility of an ATM also includes checking dependency on each other followed by crossing verification and practice classification. Other than this, ATM participates in the interview of function support members.
The re-engineering Readiness Review Process Performance Model (R2P2 Lifecycle Model) is a description of the relationships among the attributes of RR processes/work products which are developed from historical processperformance data. The R2P2 Lifecycle Model is probabilistic or statistical in nature. The R2P2 Lifecycle Model is used to predict results, along with the associated variation, to be achieved by following the Readiness Review (RR) process.
The R2P2 Lifecycle Model provides proactive management and insights to the operations of a readiness review (RR) process based on data and statistical analysis. Moreover, typically, the R2P2 Lifecycle Model is calibrated using collected process and product measures from the appraisals. Most of R2P2 Lifecycle Model include one or more controllable factors, which allow users to perform what-if analyses. An appraisal has a set of the R2P2 Lifecycle Model that covers its objectives for quality and process performance management. The R2P2 Lifecycle Model establishes the relationships between the organizational processes and the CMMI process areas as well as among the types of artifacts that can be produced in each process and the corresponding CMMI practices based on R2P2 Lifecycle Model (see Figure 1).
Fig.1. R2P2 Lifecycle Model
The R2P2 Lifecycle Model is purposely to establish as a lifecycle for process performance management. The R2P2 Lifecycle Model includes Initiating, Plan, Do, Check, and Act. The "Initiating" is to identify R2P2 Lifecycle Model's goal which integrates team, vision, and environment as process improvement via Appraisal Team Leader (ATL). The "Plan" is to establish Process Improvement (PI) master plan and appraisal plans. The "Do" is to establish a process improvement master plan and customer's key performance indexes which perform the improvement by Appraisal Team Member (ATM) and internal team. The "Check" is to establish a lesson learned, historical data and measurement information by track, predict and set the corrective actions. Finally, the "Act" aims to find the opportunities for continuous process improvement via ATL team.
IV. DEVELOPMENT PROCESS OF THE R2P2 LIFECYCLE MODEL
The R2P2 Lifecycle Model has main objectives and goals as mentioned following; 1.
Achieve appraisal’s (effectiveness).
goals across all appraisals 2. Increase efficiency by reducing the GAP statements from the Readiness Review (RR) process throughout the R2P2 Lifecycle Model life cycle.
Rapidly introduce new technology into the R2P2 Lifecycle Model and the readiness review (RR) process and achieve successful transitions. 4. Integrate Toolkit across traditional RR process boundaries to provide a composite set of capabilities to the end user.
Continuously improve the Readiness Review (RR) process performance. The development process and the expected outputs of the R2P2 Lifecycle Model is based on three stages as depicted in Figure 2 and described as followed.
The development process of R2P2 Lifecycle Model is conducted in 3 stages. The first stage targets and analyzes the root-causes to establish process performance baseline (PPB) for the founded information of the R2P2 Lifecycle Model based on 30 appraisals datasets. The second stage evaluates to reduce the gap results and enhance the R2P2 Lifecycle Model based on other 14 appraisals datasets. The third stage is to repeat the evaluations to enhance the better performance and to institutionalize the R2P2 Lifecycle Model. Furthermore, the concept of the R2P2 Lifecycle Model is a mechanism of the continuously adaptive model by following datasets of appraisal information plan to up-to-date and to achieve the R2P2 Lifecycle Model.
In the next figure (see Figure 3), there are three phases of SCAMPI defined in the SCAMPI Method Definition Document (MDD) that presents the activities and the outputs which are defined in MDD. Hence, the R2P2 Lifecycle Model establishes an additional phase, namely “Lesson learned repository”, which establishes both of process performance baseline (PPB) and process performance model (PPM) as an adaptive learning tool for the R2P2 readiness review model.
ATM: Appraisal Team Member ATL: Appraisal Team Leader PIID: Practices Implementation Indicator Description A: Approver R: Reviewer
D: Developer P: Participant
I: Instructor
Iv: Interviewer Ive: Interviewee
On the other angle, figure 3 presents the related stakeholders and their roles and responsibilities for internal team, appraisal team member (ATM), appraisal team leader (ATL). The roles are approver, reviewer, developer, interviewee, interviewer, instructor and participant.
Next, comparison of traditional readiness review (RR) process and re-engineering of the RR process in Appraisal (Selfassessment or Official appraisal) is shown in the table 1. In the table 1 above, present the outputs of re-engineering RR process based on R2P2 Lifecycle model from three phase of SCAMPI which include; 1.preparation and planning (No.1-6), 2.onsite activities (No.7), and 3.report (No.8-10). The centralizes appraisal data to process performance baseline (PPB) and process performance management are additional work products established by semi-automatic system via i-DocVer Toolkit. Figure 4 above, shows an example to present the relations between processes, the topics/content mapping with the artifacts and process areas which are the founded information to establish the semi-automatic tool namely, “The i-DocVer toolkit” is based on the R2P2 Lifecycle Model. The purpose is to map the CMMI practices and relative objective evidence and procedures. The relationship of analyzing data is verified and is consent by a group of experience appraisal team in Delphi method based on SCAMPI.
As mentioned above, the R2P2 Lifecycle Model institutionalizes a mechanism of a continuously adaptive model for updating the modern of the R2P2 Lifecycle Model which composes of three phases following the SCAMPI Method Definition Document (MDD) (See Figure 5).
Fig. 5: Research Methodology: R2P2 Lifecycle Model
The research methodology and the process of the R2P2 Lifecycle Model basically plan to evaluate and validate based on 49 appraisals from appraisal information of 6 countries. The three measurement analysis stages are defined as below: 1. The 1st stage, “understand past” dataset: 30 appraisals of the year 2010-2015 were used to analysis the root causes of the weakness of RR process. The root cause analysis result was used for illustrating the R2P2 Lifecycle Model. At this stage, we applied the Initiation phase, Plan phase that were defined in R2P2 Lifecycle Model description.
2. The 2nd stage, “control present” dataset: 14 appraisals of the year 2016 were used to assess the R2P2 Lifecycle Model. At this stage, we applied the Initiation phase, Plan phase, Do phase and Act phase that were defined in the R2P2 Lifecycle Model description.
3. The 3rd stage, “predict future” dataset: 5 appraisals of the year 2017 will be used to improve the R2P2 Lifecycle Model. At this stage, we applied the Initiation phase, Plan phase, Do phase and Act phase that are defined in the R2P2 Lifecycle Model description.
The target of each stage is a transition from the unpredictable result to improve the quality by reducing the gap and better performance as more predicted accurate and precise results.
The aims of this R2P2 Lifecycle Model are a challenge on how to improve readiness review (RR) process, efficiency attribute and together with continually improve the R2P2 Lifecycle Model for better performance of gap reduction.
To investigate problems of traditional RR process, we have defined two research questions and hypothesis as follows: RQ1: What is the weakness of the traditional RR process?
H1: There are weaknesses of traditional RR process, no required any improvement
RQ2: How to improve the RR process in reducing of GAP between stakeholders for 80 percentage?
H0: No reduction of GAP using the R2P2 Lifecycle
Model versus the traditional method H1: The reduction of GAP in 80 percentage would be possible by using the R2P2 Lifecycle Model
RQ3: How to continually improve R2P2 Lifecycle Model for better performance and quality of GAP reducing?
H0: R2P2 Lifecycle Model_v2 is not performing better than ever H1: R2P2 Lifecycle Model_v2 is more effective by reducing the Gap by more than 80 percentage
The relationship between RQs and potential causes of traditional RR process is presented in Table 2. The potential solutions and associated research questions (RQ) are detailed in Table 3. Steps to develop the re-engineering RR Process Performance Model (R2P2 Lifecycle Model) as adaptive model are described in Table 4.
Table 2 presents the potential causes of the traditional RR process which related to each research questions ( RQ1, RQ2, and RQ3). For example, the incorporation and innovation of the embedded checklists of ‘ Document Verification’ criterion and/ or sample templates is a potential cause of traditional RR process related to the RQ3: How to continually improve R2P2 Lifecycle Model for better performance and quality of GAP reducing? effort and number of gap found are established in initial plan the scope, stakeholder and training activities.
C. i-DocVer: The toolkit of the R2P2 Lifecycle Model Note: * Critical to Quality (CTQs),*Critical to Process (CTPs)
The potential solution and the activities to perform which related to each research questions (RQ1, RQ2, and RQ3) are presented in Table 3. For example, the integrated i-DocVer toolkit into R2P2 Lifecycle Model to help stakeholder perform their documents verification related to the RQ3: How to continually improve R2P2 Lifecycle Model for better performance and quality of GAP reducing?
Table 4 presents the output of a development process of the re-engineering RR process performance model. For example, the
Fig. 6: The i-DocVer toolkit: Re-engineering Readiness Review
Process Performance Model (R2P2 Lifecycle Model)
The innovative document verification tool (i-DocVer) presented in Figure 6 is a semi-automatic tool and an assistant document verification toolkit for mapping of CMMI practices and objective evidence which are expected by an assistant tool for the internal team for self-assessment appraisal. The i-DocVer toolkit has a concept designed for continuous appraisal information as an adaptive model. The features of the i-DocVer are effort and number of GAP found, Critical to Quality (CTQs) and Critical to Process (CTPs), Process Performance Baseline (PPB), process control and capability report, correlation and regression, implementation plan, improvement impacts report, process control plan, opportunities for replication and standardization include process monitoring and auditing report.
V. LESSON LEARNED FROM THE R2P2 LIFECYCLE MODEL
DEVELOPMENT
From output stage 1, we analyze the opportunity for improvement the RR process as showing in Figure 7.
Following the normalized heatmap for the separate datasets of 30 appraisals as shown in Figure 7, there are 2 types of factors and 2 distinct groups of organizations. It is clear from this that we have 3 significant types of factors. The 3 groups of organizations are still there but it is not as distinct.
In Figure 8 presents a correlation plot on the numerical factors. The 30 appraisals: Correlation and root-causes are • • • •
The correlation coefficient r measures the strength of linear relationships: –1 r 1 When a relationship exists, the variables are said to be correlated Perfect negative relationship r = –1.0 Non-linear correlation r = 0 Perfect positive relationship r = +1.0 r2 Measures the percent of variation in Y explained by the linear relationship of X and Y
As we define RQ1, the figure 7 and 8 present the analyzed result of RR process. We learn the organizational characteristics, their correlation and root-causes of the RR process for RR process improvement in next stages.
VI.
SUMMARY
This paper has known the root causes and the weaknesses of traditional RR process from analyzing of 30 historical appraisals cases. Regarding the first research question: what is the weakness of traditional RR process, the resolutions to address are investigated. The major causes of RR process are shown thus; • People Skills and experience level are not relevant to RR process performance and quality. Some CMMI expertise can be bias when they work in RR procedure.
• Work environment and project characteristic and/or organization types are not relevant to RR process performance and quality, because of the RR process must be performed strictly to the defined MDD by Institute. The different characteristics are not distinguished to result of the implemented RR process.
Therefore, as possible solutions to cover those problems, we are offering the re-engineering of readiness review process with R2P2 Lifecycle Model for enhancement of process performance, its process quality and increasing of precise.
Predictability and reusability are desirable expectations of most organizations. These concepts are highly related to that of a controlled process – one with results within well-known limits. Having a good set of controlled processes will guarantee an organization the benefit of a better performance not only in one or two projects but in many of them. In a way, (project) success will not depend on a few individuals’ heroism but rather, it will be a function of how well the organization performs as a whole – its processes are optimized.
Besides, we will use these lesson learned to create R2P2 Lifecycle Model in the next stage.
We will create the regression equation of the R2P2 Lifecycle Model. The equation’s factors are the number of gaps found, interpreting complexity, experiences of stakeholder, domain knowledge of stakeholder, and verification efficiency values. The result of these regression equation values and all 49 datasets of appraisals are expected to establish the R2P2 Lifecycle Model and enhance the i-DocVer to be a precise and reliable toolkit for the readiness review (RR) process in self-assessment appraisal. This research presented the 1st stage of root-cause analysis. Our next step is to evaluate and enhance the R2P2 Lifecycle Model and also implement the i-DocVer in the 2nd and the 3rd stages. Finally, the target is the adaptive R2P2 Lifecycle Model to continually improve the better quality of the readiness review (RR).
This research was supported/partially supported by all 49 datasets from cooperated company/organizations. We would like to thank not only the incorporate company/organizations but also our colleagues from appraisal partners and friends who provided insight and expertise that greatly assisted the research, although they may not agree with all of the interpretations/conclusions of this paper.