- Having a solid verification and validation Plan and a clear strategy to implement it plays a crucial role in successfully delivering any project e.g. governative, commercial or scientific. A specific tailoring is required but at the same ti me it is fundamental to share needs, experiences and methods spreading the best practices in the Italian Eco-S ystem and highlighting the commonalities. In this framework the VVT working group of AIS E hosted a 1-day workshop held in Bologna at the Area della Ricerca the 26th of May 2016 “Verification Validation and Testing: Passion and Deployment challenges”. This workshop coordinated by AIS E and organized by a Research Institute (INAF) and an Industrial partner (Tetra Pak Packaging S olutions) gathered together practitioners for different background (academia, research, industry, software vendors) for a full day discussion and collaborative workshop. This paper presents the outcomes and foresees future steps
INT RODUCTION
There is a general rule in software engineer that states that
to fix a problem during requirement analysis it costs 1$, to fix
the same problem during coding it costs 10$, but the cost to
fix it during the testing phase is 100$ [
On the base of this fact, the Italian Validation, Verification, and Test (VVT) group formed in late 2014. The group is one of the working group of AISE, Associazione Italiana di System Engineering, the Italian chapter of INCOSE. The group is counting a very heterogeneous group of people that belong to different environments. In the group it is easy to find experts and professionals in VVT p rocesses from the academic world, from the research business, and from industry. Its main aim is to share different experiences in different contexts, and to foster members to work together to improve the application of VVT processes. During its first meeting in 2014, the VVT group agreed to work on a list of six objectives. The first two points are: 1) to contribute to the development of a leaner application of VVT processes, 2) to contribute to the dissemination of VVT best practices between the industrial world and the academic world. In order to fulfil these two objectives the VVT group organized in May 2016 the first Italian Workshop on VVT.
Different kind of workshops on general System Engineering topics are common abroad and in Italy, and most of them are dedicated to requirements processes or to MBSE (Model Based System Engineering). There are a few examples of workshops fully devote to Verification and Test, but normally they are dealing with specific environment; one example is the “5th international workshop on verification and test of space systems” that was organized in May 2016 by the European Space Agency (ESA). In the VVT group we thought that a more general workshop could stem interesting discussion and could foster a direct sharing of knowled ge on VVT processes.
II.
WORKSHOP ORGANIZATION AND MODUSOPERANDI In this framework we, as members of the VVT group, decided to focus the Italian professionals interested to VVT around an event and we asked ourselves how to do it. We immediately agreed the gathering should be informal, open and it should produce an outcome that in principle could be the incipit for a deeper engagement of the community. We agreed the one-day workshop formula could work for our needs.
Organizing the workshop we asked ourselves “small” questions that immediately translated into big ones; e.g. we were wondering which kind of professionals could be interested in such a workshop, and which channel of communication will be more suitable to promote it. In the process, we ended-up wandering about the composition of our community (AISE members), about the Italian SE ecosystem, about the VVT professionals already in the market and the companies/ institutions where they belong; we asked ourselves if such professionals and their companies are aware of AISE activities and if and how this workshop could be appealing for them.
Doing that definitely, our point of view was not completely neuter and our vision was driven and somehow biased by o ur personal work environment and previous experiences. Luckily, the LOC members belong to different background and experiences (SR researcher at INAF - Istituto Nazionale di Astrofisica, CL quantitative Systems Engineer at Tetra Pak, LS head project engineer at SKA - SQUARE KILOMETRE ARRAY Organization). It would be great for the next time to have an even more extended and diverse board of organizer/mentor in order to have a more broader view of our community and to target in a more efficient way our next event.
At the best of our knowledge we delineated the profiles of the interested VVT professionals and we tailored the workshop on such pictures. During this process we identified four macro classes of possible stakeholders defining the work environment where they belong: Academia, Research, Industry, Software Houses.
In this macro category we grouped the professionals and practitioners (University Professors, Researchers, Post-Docs, Students) that analys e the System Engineering process es themselves and in particular the VVT process es. Since in Italy currently there is just one University offering a degree in SE (University of Rome "Tor Vergata" Postgraduate Master degree in Systems Engineering) we invited also Professors from Engineering departments such Department of Management, Economics, Industrial Engineering, Mechanical Engineering, that developed a strong interest in the System point of view. In this macro category we grouped all the stakeholders that apply SE and VVT practices as a tool in the research environment.
Doing science in the 21th century means to be able to design, develop, build, and test very large and expensive equipment, before to be able to exploit them scientifically. The project could be a satellite, a large ground facility, an accelerator of particles, but for each of these examples a system approach is mandatory to meet the requirements in term of budget, time and quality. The vast majority of representatives of this group registered to the workshop belonged to INAF. Naturally this branch of research had more space in the workshop.
In this macro category we grouped all the stakeholders belonging to the industry. This was probably the slice of stakeholder more difficult to reach because besides the companies already known for their involvement in AISE it was very difficult to reach other companies potentially interested. We attempted to reach other potential companies partnering with CRIT, a technology scouting society based on a consortium of industries based in the Emilia Romagna area, that offers services to private companies regarding technological R&D scouting, industrial best practices and innovation support. CRIT also helped us communicating the event.
A Software Houses definitely is an Industry; because of the relevance and the role of this particular industry in SE and in VVT process we decided to separate this kind of companies from other Industries. The software Houses build the SE tools that other Industries will use so somehow they play a different role. SW life cycle has the biggest cost in verification, as stated in the introduction, and a better application of System Engineering process are an effective way to control these costs. The SW discipline has his own dedicated standard on ISO/IEC/IEEE 12207.
Once the players were identified, the most important goal of this workshop was to get professionals and practitioners from these different backgrounds together; to look for commonalities, to highlight good practices , and to depict a common scenario, the Italian eco-System, where the professionals work. At the same time we were in need to clearly hear the diverse voices from different stakeholders and get to know different work environments where VVT is applied. For this reason we organized the day in two sessions . In the morning we did listen to lecturers for each class of stakeholder giving a general overview of their work environment and focussing on a key statement that can somehow summarize their point of view of VVT practices. In the afternoon the workshop itself took place. To get things really going we needed diverse people talking each other about their VVT experiences in an informal way; we needed to keep track of the informal discussion; we need ed a common framework to highlight natually the commonalities. Therefore we divided the partecipants in 7 groups assuring some variety in the composition. We asked the partecipants to work together to identify the commonalities between different environments either positive (e.g. best practices) or negatives (e.g. not adequate expertise/education, “walls” to overcome). The basic idea was to create a visual map showing the connections between the four sectors (academia, research, industry and software houses) in term of commonalities, walls and quicksands relatives to VVT practices.
It was important in our point of view to produce an tangible outcome to direct the efforts of the partecipants and to obtain a product from the workshop. To stimulate this process the LOC, with the contribution of the lecturers, provided a draft of this map where the conveneers could scatch their ideas. A “head” was assigned to each group to explain the preliminary map and to lead the discussion; each group nominated a “writer” to present the outcome of the discussion to the other groups and then write it down for further refinements and the map’s wrap-up.
III.
In this section we will briefly resume some stats about the participation to the workshop. We got about 60 people that registered for the workshop and about 30 that showed up. The composition of the participant is resumed in Fig. 3 . It is clear that half of the participants belong to the research sector and within this sector astrophysics play a crucial role; almost 90% of the researchers belong to INAF (Istituto Nazionale di Astrofisica) or they have worked there. In the Industry sector, about 35%, we see two large groups composed from professionals affiliated to Tetra Pak and to Datalogic. The Software Houses and the Academia sector provide both about 7% of the participants. The professional in the first sector belong to IBM, Blizzard, ESTECO; the ones from Academia belong to Politecnico di Milano, Università di Modena e Reggio Emilia, DIMES Università della Calabria. SE for complexity management
Mith or real opportunity? Citation index
Academia Needs Demonstrative solutions
Spin offs
Specific fields knowCleodmgpeetences/Skills
confoundings
feedbacks
The level of participation was pretty good for this kind of
event especially thinking that the area of interest of this
workshop is very narrow (VVT practices) and the typical
number of participants for such events . (E.g. the SESE 2016
tour
http://aise-incose-italia.it/events/southern-europesystems-engineering-tour-sese/). The fact that we record such
large participation for INAF and Tetra Pak shows that the
promotion of an event plays a crucial role. Another interesting
fact is that recently INAF itself organized a workshop on SE
so probably this community or at least some members are
already adopting a system approach and VVT techniques and
were ready to jump in. There is also an historical reason. SE
was born with big projects like the ICBM (Inter Continental
Ballistic Missile) in the late 50s [
The workshop got an occasion to engage the community being a starting point for following activities.
IV.
THE WORKSHOP In this paragraph we discuss the key statement that each lecturer provided, we resume the outcome of the workshop and we present the updated conceptual map
Cesare Fantuzzi Director of the Doctoral School in Engineering of Industrial Innovation, Department of Engineering Sciences and Methods. The development of mechatronic systems involves the use of different disciplines.. Traditionally, every discipline is developed independently and then integrated to generate the final system. However, increasing complexity of automatic machine design makes unavoidable to integrate simultaneously these disciplines. The resulting integrated approach carries an intrinsic complexity into system design process. To achieve this goal, the methodology treats the problem of linking the conceptual with executable models to enable the validation by simulation. The key statement presented by Prof. Fantuzzi “System Engineering as a solution to manag e complexity in design processes applied to complex industrial automatic machines: just a myth or a real opportunity? showed a possible and credible solution of the problem based on MBSE and virtual commissioning simulation processes.
Sergio Terzi – Politecnico di Milano – Department of Management, Economics and Industrial Engineering: In this talk the observatory GeCo (Collaborative Process Management of Design) was presented. The study presented is based on a series of surveys among more than 400 small, medium and large Italian manufacturing companies from four industrial macro-areas. Those survey are fundamental to bridge the gap between the literature theory and the industrial practice about New Product Development (NPD) process and Innovation. The key statement reported by Prof. Terzi is: “In their design process companies have still to understand the potentialities of system oriented approaches. Test and experimentations are still made too late. Some excellences exist and they can be highlighted as good examples to the wider manufacturing and engineering context” Alfredo Garro, Ph.D. associated professor at the university of Calabria and visiting professor at NASA proposed , connected from Houston, an innovative approach to one of th e pillars of VVT strategies: “Formal Requirements Modelling for Simulation-Based Verification”. Writing good requirements is not sufficient for delivering systems value. Efficiently validating and verifying them is the key to success. Modelling requirements according to a standard and enabling all the tasks and the testing strategies in the SE processes by the model by associating requirements and architecture with behavioural models is what Prof. Garro proposed . The “W” aspect of systems life-cycles, explicitly including the usage, maintenance and disposal phases is considered.
Examples and live demos of requirements validation were proposed.
The research brought two different examples as case studies.
The first one related to a scientific instrumentation based on
ground (i.e. the optical telescope VLT) and the second one a
deep space instrument (i.e. the sorption cooler on the Plank
satellite). The distinction is important, because of the so
different operation phases, the validation and verification
processes are quite different in the two cases. Moreover, the
Plank Satellite followed a very mature set of ECSS standards.
Davide Fierro – INAF – Chief Engineer Office INAF. In his
talk he reported his personal experience during the complex
integration, verification, and test of the VST telescope [
Gianluca Morgante – INAF – Head Thermal Engineer for several ESA space missions had the unusual chance to look at the same problem from two different point of view. The “problem” was to design, build and test the Planck Hydrogen Sorption Cooler and the two points of view were the NASA and ESA different perspectives. Morgante was member of the US team at the Jet Propulsion Laboratory that developed the Planck cooler for 7 years (1998-2005) and then he moved back to Europe to conduct the ESA test campaign and to deliver the instrument. Planck was the first European mission dedicated to map, with unprecedented sensitivity, the microwave cosmic background radiation (CMB), that is the relic of the Big Bang and to investigate the Universe evolution (birth, expansion, future, content). The scientific goals required a thermal architecture based on a passive and active systems combination and results in the most complex cryogenic mission to date. Overall the cryo-chain performances are among the biggest technological achievement of the Planck mission therefore Planck leaves a large heritage and a wealth of lessons learned for future cryo missions. The specifications required were demanding in terms of operating temperature, cooling power and vibration levels and they should be satisfied within the mass and power allocations of an ESA medium size mission. No existing system was fulfilling these requirements. The ESA approach at the time was typically very conservative, meaning to relax the requirements and to use whatever technology was already available. The NASA plan to find a solution instead was very ambitious and obviously more risky. The approach implied to come up with a creative and innovative solution: to ass emble a team of people with different expertise able to design, build and verify such a system following an approach similar, under some aspects, to the Faster, Cheaper, Better philosophy. The FCB approach is now controversial due to failures suffered by some projects but it was also able to lead important missions, such as Mars Pathfinder, to a resounding success. The US team had to demonstrate to ESA and to themselves that the undertaken endeavour was solid and promising so they built a prototype for a balloon experiment to demonstrate the TRL. This attempt was unsuccessful but the lessons learned from that first attempt were the inception of a complete new concept for the Planck cryo-coolers and their VVT. Planck Sorption Cooler testing process was complex as for any space instrumentation. The verification of the system took a long time and many resources as depicted in Fig. 5. In summary the Planck mission lessons learned in terms of systemVVT can be condensed in few bullets: System Engineering and good VVT practices are fundamental for a mission success specially to avoid over-designing and over-testing Testing is critical. Test as much and as long as possible: anticipating tests at system level even with earlier models (QM’s, EBB’s, etc.) is really important to highlight possible issue that don’t show up at subsystemlevel. Researchers/scientists need to build a sound knowledge of System Engineering approaches and VVT practices. This is still a weak spot in the expertise of scientific team members in space projects. In particular, SE and VVT should become part of education programs in the Universities also for space science courses. When starting new projects, it is critical to involve competent personnel and build on previous missions lessons learned, avoiding to re-invent the wheel.
To demonstate TRL System Tests Operation End of Life
Cold End Qualification Model
(QM)
Complete functional and performance test campaign on ground 2006 - 2008 2009 - 2013
2013 Fig. 5. Planck/Sorption Cooler Life Cycle In the end it is still hard to find an answer to the starting question: “Test better to test less?”. Is there a definitive answer? For sure, VVT practices applied and declined case by case are one of the possible answer.
Fabrizio Villa – INAF – Senior Researcher, Head of Cryowaves Lab. Cryowaves is a made-up word from the fusion of CRYOgenics and microWAVES with the aim to identify a Science and Technology branch mainly focused to develop technologies and facilities to build and operate wideband high performance radio, microwave, and mm-wave receivers and antennas operated in cryogenic environment. Modern microwave and mm-wave astrophysics uses detectors that needs to be cooled at cryogenic temperatures to minimize the detector noise, then reaching the impressive sensitivity to observe celestial sources as required by science needs. Examples of this kind of technology is the ESA Planck Satellite (up to now the most advance cryogenic system fo r space) that measured the Big-Bang sound in intensity and polarization at microK level and ALMA Observatory (www.almaobservatory.org) that, for instance, recently permitted to take an image of a proto-planetary disk rotating around a distant star.
Thanks to the iALMA premiale and ESO ALMA upgrades plan Dot. Villa has been engaged in the development of a new prototype receiver for ALMA. The technology legacy of Planck permitted to create a laboratory at INAF/IASFBologna (Cryowaves laboratory) to exploit cryowaves at their best. The idea was to setup a state-of-the-art laboratory to integrate, characterize, calibrate complex scientific instrumentation at cryogenic temperatures. In this respect the laboratory and related activities (design and simulations) are well focused on the VTT and more general SE disciplines. The final goal is to set up a laboratory and a dedicated workgroup of people where facilities, technologies knowledges and skills are in place and ready to tackle challenges related to cryogenic, integration, test, verification for space technology and beyond. The SE and VVT best Single Component
Testing
Elegant BreadBoard
(EBB) 2002 - 2006
Flight Model 1
Flight Model 2 practices are expected to be applied at all level and during all the phases; the team has a demonstrate experience in the field of integration and test for spatial mission and for complex scientific instrumentation.
proposed by Dr. Ing. Cutrona Gaetano and Dr. Ing. Margini Andrea from Tetra Pak Packaging Solutions concern ed the potentialities of the Agile methodologies to facilitate the identification and introducing new VVT methodologies from the academic to the industrial work.
The deployment of SE methodologies in the industry and in the service market in the last decade, focused on the “Lean SE” and on the Systems Product Line Engineering. Combining these two aspects an interesting issue for the V&V practitioners is the evaluation of a value-related, sufficient but not redundant effort during the usage phase of the system lifecycle.
The intervention proposed during the workshop, highlighted the human factors and the varied system responsiveness. New individuals and organizations, which were only typically marginally informed and consulted during the development phase, are involved in the usage and maintenance phase. The final user, the customer and the maintenance are typical but not the only examples. The expected net present value (ENPV) estimations find their confirmation or contradiction. Finally, the “intended environment” confirms its rapidly changing behaviour.
The SCRUM framework [
A liquid food packaging case study has been proposed. The V&V activity is strongly driven by effective V&V plans and Test Reports. The sprint backlogs converge to minimization of the requirements. Development and verification teams integration and consistency among cycling artefacts are highlighted as the keys to success for the application of Agile V&V methodologies within SCRUM framework. The basic question arose during the workgroups discussion regards the identification of a minimal set of VVT tools and methodologies, part of the ones applied d uring the development phases, suitable to efficiently enable the usage and maintenance phase of the system.
Carlo Poloni – ESTECO – Prof. Carlo Poloni addressed during his intervention the following question: “What is the VVT potential role of the virtual prototyping?” Collaborative Multidisciplinary Optimization generates relevant info while simulations are strictly connected with physical experiments during verification and validation processes.
The general observation is that isolated simulation “pearls” are implemented and reported by experts on sub -systems, components and sometimes also on systems. They are although rarely integrated into a consistent V&V strategy. The models are developed, as necessary, and their results remain typically isolated in the overall development overview. Moreover, the required skills to design, run and document the applications are in the hands of highly skilled experts whom do not relate continuously with the rest of the development team.
Prof. Poloni highlighted the importance of a network of information which sustains the system conceptual and architecture phase with the simulations performed within the VVT strategy during the design and verification phase. Fig. 7. Simulations integrated by a network of information.
Raw Data, fundamental to the simulation process require a structured, coherent and visible effort of integration in order to feed the Optimization and analysis process. Looking at another dimension a models’ hierarchy helps to maintain the coherence between the “VEE” model vertical, i.e. system maturity, and the horizontal system evolution dimensions.
Simulations model often anticipate physical testing when prototypes maturity is not sufficient or not available at all. Their development cost is not irrelevant and the re-usability must be included into the V&V strategy.
The coherency among the models and the systems maturity is one of the key factors to optimize the info/effort ratio. Examples and success stories with different multi-domain integrated tools, analytics and graphics were proposed to the attendees.
Fioravanti – Blizzard – Dr. Ing. Fioravanti, representing Blizzard srl, proposed the “Simply … Agile” alias methodologies and tools to implement Agile methods in SW development. Basically the actually available Agile development processes are still too complex in order to reach their basic objectives. The main causes of failed SW developed projects are proposed and analysed. Among them: insufficient tests in the intermediate phase.
The renovated Agile mantra lies on a wise step-by-step approach where flexibility allows managing the growing complexity with the right VVT effort. Working in small but complete and closed steps allows proposing to the final user, by fast prototyping, allows testing and usage by the final users of the implemented functions . The discussion with the workshop participants regarded the reasons of un-successful and successful projects.
V.
The resumed outputs of the groups discussion, proposed to the attention of the overall attendees can be shown in the updated conceptual map Fig. 9 and resumed as in the following paragraphs.
Gr5
Group 5 “Registro” VVT and SE not teaèched at the university
Not applied in the research Approach properly the issues
Different mindset Bureaucracy “quicksand” in the projects
and in the organizations
Resistance to change and diffuse Negative events which can facilitate the alignment
Job rotations among the two worlds (G.PCruottreocntat,hmeipneurtseosntaolbkenorewc-ohvoewred) More similarities than differences among the represented viewpoints. Many of the practitioners belong , or belonged, to more than one environment at the same time for limited or longer periods. This aspect is remarked as one of the more powerful catapults to speed-up the innovation deployment. Spin-offs, PhD theses, temporary assignments in the industries, European research funded occasions can furnish the fertile ground for growing-up commonalities. Furthermore, a SW tool developer and vendo r, a research centre, an university, a PMI or a big industry, despite their different foundations and organizational characteristics, need organized development processes following the concepts of the systems engineering.
Tailoring is the link among the different environments around the same concepts. The exposed failures in VVT experiences and the recovering strategies proposed fro m different panellists during the discussion reinforced similarities more than differences. Starting from the requirements int erpretation ambiguities , through the “verification war”, and finally to the validation challenges , similar issues were presented and discussed. Tailoring for the VVT processes is one of the key. Great telescopes, one of a type prototype, mass productions o r production lines could require VVT processes tailoring. Bureaucracy and resistance to change characterize all the environments. Only a real understanding of how the SE processes can leverage the system value of the personal Test Better to Test Less Also a failed test brings useful info
(G. Morgante) ` `
Gr4
Group 4 “Valore” Institutional reflection pause Being informed about WIP Reciprocal info exchange Come out of your’s shel SW development integrated
Different objectives Pay attention to value return
Rationalize results (C. Poloni)
Human expertise and plan growth can bypass this attitude. E.g. the VVT expert is not any more an isolated, although fundamental passage but he is one of the main wheels that convey, together with the other persons and specialists , to the success. Prejudices do no t h e lp to bypass the resistance to learn and change. Reciprocal knowledge and real opportunities to cooperate are needed. The fundamentals and the practice of the systems engineering are not diffused enough in all involved environments. Common understanding and faster deployment of research deliverables passes through training and connecting the researcher and the industrial systems engineers on the SE fundamentals, e.g. the TRL concept, and on the interactions among the Integration, Verification, Validation processes with the other ones. Monitoring the VVT SE applications in order to furnish a methodological reference is highlighted as one of the factors to address the state of the art. E.g. Agile processes are not yet enough known and applied Even if modern organization in the Astrophysical business are starting to appreciate these methods, there is a big margin for improvement for a fruitful sharing of knowledge. The PMI must be supported by the research centres and big industries to tailor their own way to Systems Engineering.
Focus on the overall V&V strategies. In order to avoid the “margin on margin” impact and to apply the “test better to test less” approach. Single examples of success cannot enough to sustain the development value. A systematic approach in the V&V strategy is required. Clearness in the application of the SE methodologies, tools , and a critical attitude vs. what done helps to avoid replicating the common mistakes.
Integrate by small steps virtual simulations with traditional physical testing can be the way to success. An easy, kiss, start-up which doesn’t require a too big once in a time effort. It can be continuously extended in scope and deepened by incremental “natural” growths to tend towards maximum value achievement.
Documenting, despite the selected implementation (e.g. classical or model based) documentation control and configuration is essential. It can require a quantity of effort, but it is fundamental not to disregard it due lack of time/funds or perform it approximatively. The addressed direction is the one of value-related documentation deliverables that goes from: traditional word check-lists, laboratory logs to virtual models deeply integrated in the development cycle (a documented model itself).
Re-use of the experiences developed in the university or by the research, after their main objective, for didactic, exemplifying or for continuous testing in the environments different from the starting ones.
Continue to meet is felt as one of the main needs , talk and create a “common place” where exchanging past, on -going and future experiences. This is the key to overcome, by intellectual curiosity, the personal or organizational restrictions. Furthermore, to create reciprocal knowledge, trust and common opportunities.
VI.
NEXT ACT IVITIES Next step of the AISE VVT working group is to diffuse and leverage on the outcomes and deliverables of the workshop within the larger Systems Engineers and VVT practitioners community. The formalization of this paper goes in this direction.
The second step of this activity is planned for spring 2017. Enlarging the involved community, prioritizing and identify real common opportunities on the themes highlighted during the workshop are the ambitions of this second sharing occasion. Contacts are also on-going to diffuse the informatio n with the parallel initiatives in the INCOSE EMEA sector.
Acknowledgment
The authors of this work would like to acknowledge the invited speakers at the workshop. They would also acknowledge Tetra Pak Packaging Solutions and Cryowaves laboratory for make the workshop possible. This workshop was partly supported by the Italian Ministero dell'Istruzione, Universita' e Ricerca through the grant Progetti Premiali 2012 -- iALMA (CUP C52I13000140001).