Can VVT capabilities mitigate programs implosion? How to sustain complexity increase by VVT capabilities Leardi Carlo Systems Engineering Validation Tetra Pak Packaging Solutions spa Modena, Italy Via Delfini, 1, 41123 (MO) carlo.leardi@incose.org Copyright Copyright Copyright ©by ©©held held held by by the author. theauthor. the author. Abstract— challenges and the adaptation in the industrial environment. One clear example is the implementation of data science in day- One of the most frequent statements about Systems Engineering to-day design. challenges is related to complexity increase. The theme of how Integration, Verification and Validation processes already drive affording the complexity increased pressure in efficient and a relevant percentage of total development costs. It’s impact sustainable way often emerges during workshops and webinars during usage and maintenance phases is increasing as correlated promoted by the VVTWG, Verification Validation and Testing to big data availability. AISE Working Group. This article proposes one viewpoint The article focuses on the following question: “How to avoid related to the opportunity to increase the VVT capabilities, programs implosion risk due to un-managed complexity methods, tools and skills, progressively, homogenously and increase in the VVT area?” “How can the set of VVT value-focused to significantly sustaining the pressure increase capabilities evolve towards the challenges?”. related to complexity. Initially the characteristics of complexity are deepened and Examples from the industrial environment are furnished and exemplified. The SE processes evolution are then discussed briefly discussed. regarding the specific aspects of complexity management. The capabilities dimensions are illustrated and exemplified. An Keywords—Verification Validation Testing , complexity industrial example regarding complex VVT activities management, capabilities, methods, tools, skills management by Design Structure Matrices is briefly I. INTRODUCTION mentioned. As a conclusion, it is stated that a progressive, homogenous increase in VVT capabilities: methods, tools and The last decades highlighted the passage from the awareness of skills, can significantly contribute to sustaining the pressure complexity increase to a daily-job issue for the systems increase related to complexity. engineers’ community and, in special way, for the VVT practitioners. II. Systems boundaries expand, specialized topics embrace several programs, enhanced by the transformation from Systems to III. CHARACTERISTICS OF COMPLEXITY Systems of Systems. The technological innovation acceleration “Programs complexity is constantly increasing”. This introduces new and more powerful opportunities. However, it statement is often used to address one of the most recurrent requires adaptation and specialization to the VVT practitioners. threat to day-to-day systems engineering successful Increasing quantities of data and not homogeneous information applications. This is especially true when the Verification and gets ready available while focused value propositions are Validation processes are addressed together with the functional required at decision points for the overall stakeholders’ chain. testing contained in the Integration one and the alternative System of Systems complexity are not only related to selection of the decision one. These systems engineering dimensions. It also relates to evolution dynamic, knowledge processes intrinsically overlapping, propose recurrent activities uncertainty, sub-systems interconnections, technology to create, review and finalize their deliverables. The European evolution, communication density and pointy customers’ needs. Systest Project assessed as around 60% of the budget is The evolution from document to model-based systems allocated to activities directly or un-directly related to VVT. engineering sustains the front-loading as well as models’ re- Complexity is however a concept which needs some better usability along the overall system life-cycle, including usage, specification to be understood. maintenance, update and disposal phases of the VVT processes. One, but not necessarily the more important, of the aspects of The verification and validation community leverages on a wide complexity is the programs dimensions in all their facets. set of well-established best practice. A relevant gap is although Programs are getting bigger and bigger as an effect of the registered among the development by academia and research developments transition from Systems to Systems of Systems. centers of new testing and analysis opportunities facing the new Well known figures are the exponential growth of code lines for SW systems and number of modules once a time intended as systems by themselves. E.G: In the liquid food industry, the programs involves more and more frequently a holistic viewpoint including from the raw materials acquisition to the recycling of the final package elements. Programs scope crosses different environments and includes many new stakeholders, rulers and standards owners. The increasing number of needs captured and their proper translation into systems requirements represent a day-to-day challenge for systems engineers. The second aspect of complexity is complication. One slim mechanical chronograph can present as much complexity as one huge industrial plant layout in terms of components and Picture #1: the Trajan column includes all the complexity their interactions. The density of technologies integrated into a single actuator, e.g. a phased movement, is larger than the one components: dimensions, complication, use case, mutability, of a similar application developed three decades ago. The mix sources uncertainties, amount of info well before the data science era. of different complications and development maturities creates further challenge to the development team. The innovation IV. HOW TO AFFORDING COMPLEXITY? acceleration introduces new technologies and forces the community of well-established VVT practitioners to change their working practices. In the beginning of Systems Engineering the focus were on Mutability is the third aspect. Requirements, although transition from waterfall to concurrent processes. Later, validated and pre-verified increase their tendency to change as formalized in “Vee”, “spiral” or “iterative”. The integration of an effect of the customer’s pressure which requires quick testing, SW, HW and Systems of Systems aspects introduced adaptation to new un-expected requests. E.g.: The replacement “W-model”, “Dual-V” and other similar approaches. of obsolete technologies/components and the continuous escalation of performances always lifts-up the targets and increases the validation effort. Target like as “not more than 1 out of several hundred thousand defect ratios at 95% confidence level are not any more un-usual. The paradigmatic shift from document to model-based systems engineering assume that configuration management is easier and shorter. The adoption of agile, spiral, incremental and in general lean concepts in products development introduces more mutable specification of requirements. Uncertainty increase is the fourth aspect. It leverages on all the five previously listed aspects. E.g. agreement and target validation effort is more than linear increased by the number Picture #2: an example of the Dual-V model [1] and the variety of stakeholders involved. Requirements conflictual or eventually un-feasibility is enhanced by scope All these combinations allow focusing on a limited part of the extension. The growth of interface requirements involves overall picture without forgetting the relations with the further attention to the system engineer. remaining part. Each single task finds it best place and the Last, but not surely the less important is the effect of the digital relations with the other entities are pre-defined. revolution, alias the increased availability of information. A From the other side, there is the tendency to incremental and in huge amount of data and info are available to the analysts. This general lean development concepts to reduce actual complexity flow of heterogeneous information requires powerful and wise to an affordable level. The issue is maintaining the integrity of analyses to extract the amount of knowledge necessary to the the System of Systems view. program to develop consciously the System of Systems without Pressure induced by increasing complexity does not however getting lost in analysis or deriving misleading directions for seem enough sustained by mixing tailored development decision process. processes advancements. The first drawback is that inserting additional complexity greater than the issues to be solved increases the dimension of the issues to be afforded. A second consideration is that additional resources skilled, with the right level of knowledge and charisma, to afford parallel tasks management are often simply un-affordable and too long to achieved. In order to avoid programs implosion one of the possible mitigation actions is sustaining the pressure of increase complexity by capability increase. V. SUSTAINING COMPLEXITY INCREASE PRESSURE NY ENHANCING CAPABILITIES Capability is the communized result of methodology, tools and skills. By methods, the fundamentals, each issue is afforded in a procedurally corrected way. They are typically developed by academia and research, validated and disseminated by standardization and regulatory agencies and finally deployed in industry with the initial help of consultancy. Picture #5: Human skills are well resumed by the Vitruvian human-centric concept. Skills are owned by the VVT practitioners. Tailoring and application of generic methodologies by the media of the tools to the specific industrial issues. One clear example is the impact of the digital revolution that highlighted the importance of the data science application. It is fundamentally a mix of well-known as well as advanced algorithmic methodologies. Such type of analyses is supported Picture #3: Methods set the directions and the ways to solve by specific HW, SW, communication and tailored, although the issues based on well known, computational and statistical methods. Analysts and statisticians are so required to update their day-to- Tools make available deploying the methodologies into an day practices to move towards a net-based, highly tailored way ordered, structured and integrated framework. of working. Sometimes they are got back to their experienced tracks, but usually new practices have to be applied. The industrial practitioners are required to acquire multi- dimensional skills together with deepening in specialized matters. The following challenges can be sustainable to the different aspects of the capability. The methodology evolution makes available always more powerful methods. Picture #4: Tools evolution brought in a few decades from multiple mechanical turning machines to AI driven multiple axis ones. Picture #6: Methods, tools and skills evolve in a connected holistic way E.g. The classical formulation of a validation target for Tools tend to include a wider set of methods ready available to continuous measure could be expressed as: “The performance the practitioners. KISS user interfaces and processes integrated shall be comparable to [unit]”. So, formulated, methodological drives are developed to sustain the selection the statistical methodology applied is a t-test of a sample where and the correct use of the methodologies. The continuous race the average is compared vs. the estimated target. Student's t- among open-sources and licensed SWs, the R story is a clear Test is one of the most commonly used techniques for testing a example, enables the acquisition of this aspect of the capability. hypothesis based on a difference between sample mean and a Without the power increase allowed by HW and SW evolution target. Explained in layman's terms, the t test determines a the application of most advanced methodologies, if not the one probability that one population is, on average, the same with of the ‘60s ones, could not be possible. respect to the stated target. The test was proposed by William Multidimensionality called by data science applications Gosset, English statistician whom published under the pen- requires the enlargement of skills and theoretical aspects name of “Student”, starting from the “The probable error of a domination by VVT practitioners. Without losing the basic mean”, 1907 Biometrika, a seminal work for twentieth century strengths, each practitioner is expected to focus on: mechanical, industrial statistics. Student formalized the t-distribution which physics, chemistry, communication and web based allows this standardized comparison at the bases of the more applications. diffused requirement archetype. The following SWOT scheme illustrates the potential combination of Strengths and Opportunities to sustain treats Regarding the passage of methodology from academia to lead by complexity increase by capabilities enhancements: industry it is wise to remember what said about W. Gosset: “To many in the statistical world "Student" was regarded as a Strengths Weaknesses statistical advisor to Guinness's brewery, to others he appeared to be a brewer devoting his spare time to statistics. ... though Enhanced and Limited resources there is some truth in both these ideas they miss the central integrated skills Time/budget point, which was the intimate connection between his statistical limitations research and the practical problems on which he was engaged. Opportunities Treats ... "Student" did a very large quantity of ordinary routine as well as his statistical work in the brewery, and all that in More powerful methods Complexity  addition to consultative statistical work and to preparing his Tools more inclusive various published papers.” and KISS Picture #7: SWOT analysis resume In order to have the first industrial manual of statistics we must although wait the 1947’s Davies: Davies, O. L. (Ed.): Statistical In particular, the dimensions, complication and mutability Methods in Research and Production. Oliver L Boyd, aspects of programs complexity can be afforded by more Edinburgh and London 1947. powerful methodologies and supported by inclusive and The application of such a statistical methodology is however process-integrated tools. highly expensive if applied to enlarged scopes. Other advanced Uncertainty management is one exercise which is funded on statistical techniques have then to be properly and consciously human skills and then can be solved by appropriate methods applied. E.g. the approximation of binary or count-based and tools. distributions to the normal one are not any more applicable. The issues deriving from digital revolution requires a full new General Linear or Hierarchical models, up to Laplacian effort in all three the capability dimensions. Eigenmaps are today available to properly compare test results One approach is a global program intended to produce a one in to targets in more complex situations. a time huge step intended to uniformly leverage the capabilities in the company. This type of intervention is assimilated, if not parallel, to a huge development process change/tailoring effort. Sudden big improvements in capabilities are however difficult and costly to be achieved. This approach requires relevant effort and time to identify the gaps, select and screen the necessary methodologies, update the best practices, acquire the tools, train the practitioners. At the same time, normal day-to-day business is running and it can be a serious issue to manage the current development while spreading and sustaining the seeds of future capabilities availability. A wiser attitude could be an incremental but continuously supported and followed-up capability increase focused on the weak areas. Typically, in a complex company there is already a limited bunch of practitioners aware and practically ready to Picture #7: Laplacian Eigenmaps graphical representation utilize relevant methodologies. Niche tasks, open-source codes and initiatives facilitate these spontaneous opportunities. Leveraging on already updated capability areas and identifying Effectiveness refers to documented and verified system high value implementation opportunities allows a general requirements fulfillment or to validated user needs. Efficiency growth in the organization at a sustainable effort and keeping relates to the effort spent, in terms of budget, time, skills and focused on the top issues. As soon as the “low hanging fruits” resources to achieve the previous result. are achieved, new areas and practitioners can be progressively The value flow, as addressed by the stakeholder’s needs identified and new implementation opportunities deployed. As elicitation, is identified and traced through its effectively soon as improvements are implemented, the additional value is achieved deliverables and the deviations of the ratio with the gradually stabilized. Monitoring the process allows to evaluate budget and schedule effort. the break-even point when further deployments are not any more sufficiently value-related. From time to time HW/SW acquisitions, trainings and value-related applications are prioritized accordingly to necessity. The effort is so diluted and returns value during the application. To be successful, capabilities enhancements can so be driven by a coordinated effort to introduce step-by-step improvements in all three the aspects: methods, tools and skills. VI. CASE STUDY: COMPLEX VVT STRATEGY AND PLAN MODELLED AND ELABORATED BY DESIGN STRUCTURE MATRICES. N2 matrices were introduced in the seventies to manage IBM Picture #6: Design Structure Matrices VVT strategy and plan Program and first published in a 1977 TRW internal report. graphical and analytic model VII. CONCLUSIONS Coordinated small steps incremental improvements in the three dimensions of capability: methods, tools and skills well integrated into a flexible and efficient development process are expected to effectively mitigate the complexity increase. The evidences, derived from the discussions and the activities of the AISE Verification Validation and Testing Working Group, shall be furtherly used for dissemination. REFERENCES [1] John O. Clark, “Systems Engineering form a Standards, V- Model, and Dual V-Model Perspective” Systems and Software technology Conference April 20. 2009. Picture #8: The original Lano’s N2 diagrams [2] Marco Vitruvio Pollione. De architectura. Liber III [3] Design Structure Matrix Methods and Applications Design Structure Matrices represent the evolution of the N2 By Steven D. Eppinger and Tyson R. Browning, 2012 diagrams to afford relevantly complexity in terms of schedule, [4] Avner Engel, Shalom Shachar "Measuring and optimizing components or multiple dimensions modeling. systems' quality costs and project duration" Systems A general, easily tailorable model is in this example provided Engineering Volume 9, Issue 3, pages 259–280, Autumn to the Systems Engineers in charge of Verification and (Fall) 2006. Validation processes. The aim is to make available a unique, [5] William Sealy Gosset, 1876-1937, in E S Pearson and M G Kendall, Studies in the History of Statistics and computational, graphical and communication environment Probability (London, 1970), 355-404. where managing the VVT, Verification, Validation and [6] Verification, Validation, and Testing of Engineered Testing, activities by identifying the value flow and its Systems, Wiley, 2010. Avner Engel. evolution during system development and, extensively, during system life-time.