One of the most frequent statements about Systems Engineering challenges is related to complexity increase. The theme of how affording the complexity increased pressure in efficient and sustainable way often emerges during workshops and webinars promoted by the VVTWG, Verification Validation and Testing AISE Working Group. This article proposes one viewpoint related to the opportunity to increase the VVT capabilities, methods, tools and skills, progressively, homogenously and value-focused to significantly sustaining the pressure increase related to complexity. Examples from the industrial environment are furnished and briefly discussed.
I. INTRODUCTION The last decades highlighted the passage from the awareness of complexity increase to a daily-job issue for the systems engineers’ community and, in special way, for the VVT practitioners.
Systems boundaries expand, specialized topics embrace several programs, enhanced by the transformation from Systems to Systems of Systems. The technological innovation acceleration introduces new and more powerful opportunities. However, it requires adaptation and specialization to the VVT practitioners. Increasing quantities of data and not homogeneous information gets ready available while focused value propositions are required at decision points for the overall stakeholders’ chain. System of Systems complexity are not only related to dimensions. It also relates to evolution dynamic, knowledge uncertainty, sub-systems interconnections, technology evolution, communication density and pointy customers’ needs. The evolution from document to model-based systems engineering sustains the front-loading as well as models’ reusability along the overall system life-cycle, including usage, maintenance, update and disposal phases of the VVT processes. The verification and validation community leverages on a wide set of well-established best practice. A relevant gap is although registered among the development by academia and research centers of new testing and analysis opportunities facing the new
“Programs complexity is constantly increasing”. This statement is often used to address one of the most recurrent threat to day-to-day systems engineering successful applications. This is especially true when the Verification and Validation processes are addressed together with the functional testing contained in the Integration one and the alternative selection of the decision one. These systems engineering processes intrinsically overlapping, propose recurrent activities to create, review and finalize their deliverables. The European Systest Project assessed as around 60% of the budget is allocated to activities directly or un-directly related to VVT. Complexity is however a concept which needs some better specification to be understood.
One, but not necessarily the more important, of the aspects of complexity is the programs dimensions in all their facets. Programs are getting bigger and bigger as an effect of the developments transition from Systems to Systems of Systems. Well known figures are the exponential growth of code lines for SW systems and number of modules once a time intended as systems by themselves.
E.G: In the liquid food industry, the programs involves more and more frequently a holistic viewpoint including from the raw materials acquisition to the recycling of the final package elements. Programs scope crosses different environments and includes many new stakeholders, rulers and standards owners. The increasing number of needs captured and their proper translation into systems requirements represent a day-to-day challenge for systems engineers.
The second aspect of complexity is complication. One slim mechanical chronograph can present as much complexity as one huge industrial plant layout in terms of components and their interactions. The density of technologies integrated into a single actuator, e.g. a phased movement, is larger than the one of a similar application developed three decades ago. The mix of different complications and development maturities creates further challenge to the development team. The innovation acceleration introduces new technologies and forces the community of well-established VVT practitioners to change their working practices.
Mutability is the third aspect. Requirements, although validated and pre-verified increase their tendency to change as an effect of the customer’s pressure which requires quick adaptation to new un-expected requests. E.g.: The replacement of obsolete technologies/components and the continuous escalation of performances always lifts-up the targets and increases the validation effort. Target like as “not more than 1 out of several hundred thousand defect ratios at 95% confidence level are not any more un-usual.
The paradigmatic shift from document to model-based systems engineering assume that configuration management is easier and shorter. The adoption of agile, spiral, incremental and in general lean concepts in products development introduces more mutable specification of requirements.
Uncertainty increase is the fourth aspect. It leverages on all the five previously listed aspects. E.g. agreement and target validation effort is more than linear increased by the number and the variety of stakeholders involved. Requirements conflictual or eventually un-feasibility is enhanced by scope extension. The growth of interface requirements involves further attention to the system engineer.
Last, but not surely the less important is the effect of the digital revolution, alias the increased availability of information. A huge amount of data and info are available to the analysts. This flow of heterogeneous information requires powerful and wise analyses to extract the amount of knowledge necessary to the program to develop consciously the System of Systems without getting lost in analysis or deriving misleading directions for decision process.
Picture #1: the Trajan column includes all the complexity components: dimensions, complication, use case, mutability, sources uncertainties, amount of info well before the data science era.
In the beginning of Systems Engineering the focus were on transition from waterfall to concurrent processes. Later, formalized in “Vee”, “spiral” or “iterative”. The integration of testing, SW, HW and Systems of Systems aspects introduced “W-model”, “Dual-V” and other similar approaches.
All these combinations allow focusing on a limited part of the overall picture without forgetting the relations with the remaining part. Each single task finds it best place and the relations with the other entities are pre-defined.
From the other side, there is the tendency to incremental and in general lean development concepts to reduce actual complexity to an affordable level. The issue is maintaining the integrity of the System of Systems view.
Pressure induced by increasing complexity does not however seem enough sustained by mixing tailored development processes advancements.
The first drawback is that inserting additional complexity greater than the issues to be solved increases the dimension of the issues to be afforded. A second consideration is that additional resources skilled, with the right level of knowledge and charisma, to afford parallel tasks management are often simply un-affordable and too long to achieved.
In order to avoid programs implosion one of the possible mitigation actions is sustaining the pressure of increase complexity by capability increase.
V. SUSTAINING COMPLEXITY INCREASE PRESSURE NY
ENHANCING CAPABILITIES Capability is the communized result of methodology, tools and skills.
By methods, the fundamentals, each issue is afforded in a procedurally corrected way. They are typically developed by academia and research, validated and disseminated by standardization and regulatory agencies and finally deployed in industry with the initial help of consultancy.
Picture #3: Methods set the directions and the ways to solve the issues Tools make available deploying the methodologies into an ordered, structured and integrated framework.
Picture #4: Tools evolution brought in a few decades from multiple mechanical turning machines to AI driven multiple axis ones.
Picture #5: Human skills are well resumed by the Vitruvian human-centric concept.
Skills are owned by the VVT practitioners. Tailoring and application of generic methodologies by the media of the tools to the specific industrial issues.
One clear example is the impact of the digital revolution that highlighted the importance of the data science application. It is fundamentally a mix of well-known as well as advanced algorithmic methodologies. Such type of analyses is supported by specific HW, SW, communication and tailored, although based on well known, computational and statistical methods. Analysts and statisticians are so required to update their day-today practices to move towards a net-based, highly tailored way of working. Sometimes they are got back to their experienced tracks, but usually new practices have to be applied. The industrial practitioners are required to acquire multidimensional skills together with deepening in specialized matters.
The following challenges can be sustainable to the different aspects of the capability.
The methodology evolution makes available always more powerful methods.
Picture #6: Methods, tools and skills evolve in a connected holistic way E.g. The classical formulation of a validation target for continuous measure could be expressed as: “The performance <xyz> shall be comparable to <target> [unit]”. So, formulated, the statistical methodology applied is a t-test of a sample where the average is compared vs. the estimated target. Student's tTest is one of the most commonly used techniques for testing a hypothesis based on a difference between sample mean and a target. Explained in layman's terms, the t test determines a probability that one population is, on average, the same with respect to the stated target. The test was proposed by William Gosset, English statistician whom published under the penname of “Student”, starting from the “The probable error of a mean”, 1907 Biometrika, a seminal work for twentieth century industrial statistics. Student formalized the t-distribution which allows this standardized comparison at the bases of the more diffused requirement archetype.
Regarding the passage of methodology from academia to industry it is wise to remember what said about W. Gosset: “To many in the statistical world "Student" was regarded as a statistical advisor to Guinness's brewery, to others he appeared to be a brewer devoting his spare time to statistics. ... though there is some truth in both these ideas they miss the central point, which was the intimate connection between his statistical research and the practical problems on which he was engaged. ... "Student" did a very large quantity of ordinary routine as well as his statistical work in the brewery, and all that in addition to consultative statistical work and to preparing his various published papers.” In order to have the first industrial manual of statistics we must although wait the 1947’s Davies: Davies, O. L. (Ed.): Statistical Methods in Research and Production. Oliver L Boyd, Edinburgh and London 1947.
The application of such a statistical methodology is however highly expensive if applied to enlarged scopes. Other advanced statistical techniques have then to be properly and consciously applied. E.g. the approximation of binary or count-based distributions to the normal one are not any more applicable. General Linear or Hierarchical models, up to Laplacian Eigenmaps are today available to properly compare test results to targets in more complex situations.
Picture #7: Laplacian Eigenmaps graphical representation Tools tend to include a wider set of methods ready available to the practitioners. KISS user interfaces and processes integrated methodological drives are developed to sustain the selection and the correct use of the methodologies. The continuous race among open-sources and licensed SWs, the R story is a clear example, enables the acquisition of this aspect of the capability. Without the power increase allowed by HW and SW evolution the application of most advanced methodologies, if not the one of the ‘60s ones, could not be possible.
Multidimensionality called by data science applications requires the enlargement of skills and theoretical aspects domination by VVT practitioners. Without losing the basic strengths, each practitioner is expected to focus on: mechanical, physics, chemistry, communication and web based applications.
The following SWOT scheme illustrates the potential combination of Strengths and Opportunities to sustain treats lead by complexity increase by capabilities enhancements:
Enhanced and integrated skills
More powerful methods Tools more inclusive and KISS Picture #7: SWOT analysis resume
Limited resources Time/budget limitations Treats Complexity In particular, the dimensions, complication and mutability aspects of programs complexity can be afforded by more powerful methodologies and supported by inclusive and process-integrated tools.
Uncertainty management is one exercise which is funded on human skills and then can be solved by appropriate methods and tools.
The issues deriving from digital revolution requires a full new effort in all three the capability dimensions.
One approach is a global program intended to produce a one in a time huge step intended to uniformly leverage the capabilities in the company. This type of intervention is assimilated, if not parallel, to a huge development process change/tailoring effort. Sudden big improvements in capabilities are however difficult and costly to be achieved. This approach requires relevant effort and time to identify the gaps, select and screen the necessary methodologies, update the best practices, acquire the tools, train the practitioners. At the same time, normal day-to-day business is running and it can be a serious issue to manage the current development while spreading and sustaining the seeds of future capabilities availability.
A wiser attitude could be an incremental but continuously supported and followed-up capability increase focused on the weak areas. Typically, in a complex company there is already a limited bunch of practitioners aware and practically ready to utilize relevant methodologies. Niche tasks, open-source codes and initiatives facilitate these spontaneous opportunities. Leveraging on already updated capability areas and identifying high value implementation opportunities allows a general growth in the organization at a sustainable effort and keeping focused on the top issues. As soon as the “low hanging fruits” are achieved, new areas and practitioners can be progressively identified and new implementation opportunities deployed. As soon as improvements are implemented, the additional value is gradually stabilized. Monitoring the process allows to evaluate the break-even point when further deployments are not any more sufficiently value-related. From time to time HW/SW acquisitions, trainings and value-related applications are prioritized accordingly to necessity. The effort is so diluted and returns value during the application.
To be successful, capabilities enhancements can so be driven by a coordinated effort to introduce step-by-step improvements in all three the aspects: methods, tools and skills.
VI. CASE STUDY: COMPLEX VVT STRATEGY AND PLAN MODELLED AND ELABORATED BY DESIGN STRUCTURE
MATRICES.
N2 matrices were introduced in the seventies to manage IBM Program and first published in a 1977 TRW internal report.
Picture #8: The original Lano’s N2 diagrams Design Structure Matrices represent the evolution of the N2 diagrams to afford relevantly complexity in terms of schedule, components or multiple dimensions modeling.
A general, easily tailorable model is in this example provided to the Systems Engineers in charge of Verification and Validation processes. The aim is to make available a unique, computational, graphical and communication environment where managing the VVT, Verification, Validation and Testing, activities by identifying the value flow and its evolution during system development and, extensively, during system life-time.
Effectiveness refers to documented and verified system requirements fulfillment or to validated user needs. Efficiency relates to the effort spent, in terms of budget, time, skills and resources to achieve the previous result.
The value flow, as addressed by the stakeholder’s needs elicitation, is identified and traced through its effectively achieved deliverables and the deviations of the ratio with the budget and schedule effort.
Picture #6: Design Structure Matrices VVT strategy and plan graphical and analytic model
Coordinated small steps incremental improvements in the three dimensions of capability: methods, tools and skills well integrated into a flexible and efficient development process are expected to effectively mitigate the complexity increase. The evidences, derived from the discussions and the activities of the AISE Verification Validation and Testing Working Group, shall be furtherly used for dissemination.