Probabilistic Thinking to Support Early Evaluation of System Quality Through Requirement Analysis Mohammad Rajabalinejad*, Maarten G. Bonnema Faculty of Engineering Technology, University of Twente 2522 LW Enschede, Netherlands * M.Rajabalinejad@utwente.nl Abstract. This paper focuses on coping with system quality in the early phases of design where there is lack of knowledge about a system, its functions or its architect. The paper encourages knowledge based evaluation of system quality and promotes probabilistic thinking. It states that probabilistic thinking facili- tates communication between a system designer and other design stakeholders or specialists. It accommodates tolerance and flexibility in sharing opinions and embraces uncertain information. This uncertain information, however, is to be processed and combined. This study offers a basic framework to collect, pro- cess and combine uncertain information based on the probability theory. Our purpose is to offer a graphical tool used by a system designer, systems engineer or system architect for collecting information under uncertainty. An example shows the application of this method through a case study. Keywords: system; quality; uncertainty; design; evaluation Nomenclature  expected value  relative weight di a random number representing the system quality over the i-th requirement d ik a random number representing the system quality over the i-th requirement ac- cording to the k-th stakeholder  expected value  relative weight of requirements m number of stakeholders n number of requirements ri a random number representing the importance of the i-th requirement rik a random number showing the opinion of k-th stakeholder over i-th requirement sk a random number representing the importance of the k-th stakeholder sk j a random number showing the opinion of j-th stakeholder over k-th stakeholder sq a random number representing the system quality Var variance 1 Introduction To deliver a quality system, a system designer should first identify, clarify, and doc- ument system requirements [1]. These tasks are performed in the earliest phase of a project life cycle and in the presence of a high level of uncertainty [2]. These re- quirements are not fixed and may change throughout the development stages [3]. On the other hand, some requirements e.g. maximization of benefits are explicitly or im- plicitly present in all design phases, and different terminology may be used for them. For example, design objectives or concept drivers are commonly used in the concept phase while the program of requirements or design criteria are more likely to be used in the embodiment phase. It is important to note that the requirements keep the focus of the design team on the most important design aspects or main needs and they pro- vide references for the evaluation of system quality. Therefore, system requirements have explicit roles through the design process. It is mainly because of the presence of a systematic approach [4] in this process and also because of the societal demands for meeting the standards [5] in engineered products. These are reflected in tools, processes and standards. An example is the popular method called the house of quality which relates user requirements to design re- quirements in order to ensure quality end-products. To achieve quality systems, de- signers need to define proper system requirements as early as possible [6] as they help judging the relevance of new information. The evaluation of design alternatives are on the basis of these requirements. In oth- er words, every design alternative has to be able to address the initial requirements. As a result, these requirements form criteria for evaluation of system quality. These design criteria may change through the design process and may have different degrees of importance. To assess system quality, a system designer has to rank them at the early stages of the design process. Ranking methods is of great value in decision models, and the use of multi criteria decision models (MCDM) typically involve crite- ria ranking. 1.1 Information elicitation To define system requirements, identification of stakeholders is one of the earliest steps. A review research by Pacheco and Garcia [9] confirms that an incomplete set of stakeholders may lead to incomplete requirements. A system designer has to pay at- tention to the problems arising from the scope, understanding and validation of re- quirements [10, 11] in the course of communication with stakeholders. Figure 1 presents the functional diagram for identification of stakeholders and communication with them. It shows some new stakeholders may be realized through the course of communication with already-known stakeholders. To document the stakeholder’s needs and collected feedback , Salado and Nilchiani [12] suggest a set of questions for discovering new stakeholders in order to identify a complete set of stakeholders. Complex systems often include a relatively high number of stakeholders with different (conflicting) interests [13]. In such cases, the process of information elicitation, documentation and integration is a necessity to achieve informative con- clusions. Yes Communicate Identify Document the More stakeholders with stakeholders needs identified? stakeholders No Integrate the Realize the key collected requirements inforamtion Fig. 1. The process of identification of stakeholders, communication with them, inte- gration and prioritization of the collected data. 1.2 Ranking process Ranking of requirements (or criteria) based on their importance is well discussed in decision models. The use of multi criteria decision models typically involves a sys- tematic ranking process as for instance indicated in [4, 14]. The influence of the rank- ing process on final decisions is for example explained in [15]. A review of subjective ranking methods shows that different methods cannot guarantee accurate results. This inconsistency in judgment explains difficulty of assigning reliable and subjective weights to the requirements. A systematic approach for ranking is described in [16] that is a generalization of Saaty’s pairwise structure [17]. Given the presence of sub- jectivity in the ranking process, sensitivity analysis of the design criteria is used to study the influence of variation and the ranking process on the decisions made [18]. Furthermore, some approaches e.g. the task-oriented weighing approach is effectively used. This approach is meant to limit the subjectivity of criteria weighting [19]. It suggests an algorithm to rank criteria objectively while considering the uncertainty in criteria weight [20]. The approach is based on introducing fuzzy numbers that impos- es specified membership functions, which has been also used in [21, 22]. However, there is an obstacle for systems architectures or engineers in communi- cation of the proposed methods with different stakeholders. The stakeholders can be individuals, corporations, organizations and authorities, with different fields/ levels of knowledge and experience [2]. The stakeholders have interest in the project and they desire to express their knowledge and expertise to improve the design. They also have expectations which have to be addressed at the end. Besides, it is advised to designers to rely on the experts in order to manage design uncertainties since it is proven that experts provide frameworks for making knowledge based decisions under uncertainty [23, 24]. This offers a human solution in terms of preferred alternatives. The uncer- tainty in importance of design requirements is also of human nature which should be reflected in the weighting process. To address these, we present the principles of our method through the next section. 1.3 Evaluation of system quality To estimate the system quality, an intuitive method is used. Detail description of this method and its formulation emerge through the rest of this paper. It provides a consistent framework to value the system under uncertainty and observe how well the system addresses the stakeholder’s needs. This outcome provide valuable sources for the system designer or system engineer to monitor the strong and weak point of the system. Figure 2 presents the functional flow for evaluation of system quality in a pluralistic approach where the stakeholders’ opinion is fundamentally contributed to quality evaluation. In this perspective, communication plays an essential role and the proposed method aims to facilitate this communication. Propose a new alternative system Improve the A new design proposed No Yes alternative ? system No Communicate the Measure how much the Integrate the Measure the System quality is key requirements proposed system collected overal system satisfactory? with stakeholders addresses requirements information quality Yes The proposed system is chosen Fig. 2. The work flow for evaluation of system quality. 1.4 Presentation We aim to present a realistic and intuitive approach that can communicate to people with different fields of knowledge and expertise. The method must be transparent, easy to implement and readily adaptable by different users. For this purpose, graphs are used to effectively communicate with different users. The format presented in Figure 3 identifies the importance of a requirement according to a stakeholder’s opin- ion. The linguistic scale or the numeric scale can be used for the ease of communica- tion, and one can assign the range of possible importance to a certain requirements. Fig. 3. An example of a stakeholder’s opinion about the importance of the i-th stakeholder Si . A probability distribution function (PDF) is assigned to this recorded data. Symmetric opinions are assumed here in this paper as described in [25, 26] and the collected data is treated as a random variable with a Gaussian distribution aiming to achieve set of a stochastic weight factors. 2 Formulation 2.1 Ranking of Stakeholders Having m stakeholders, each stakeholder evaluates the importance of all the stake- holders. This information is presented by stochastic variables sk , sk ,..., sk , where 1 2 m1 sk j represents the opinion of j-th stakeholder over the importance of k-th stakeholder, and its expected value and variance are respectively shown by  sk j  and Var sk j  . The expected relative weight and variation for the importance of each stakeholder is achieved by the following equations. m 1 E[ k ]  m  s  kj (1)  sk  j 1 k 1 m 1 Var k   2  Vars  kj (2)  m     sk   j 1  k 1  2.2 Ranking of requirements Now m stakeholders assess the importance of the i-th requirement ri , and this infor- mation is represented by stochastic variables ri1 , ri2 ,..., rim , where rik presents the k-th stakeholder’s opinion over the importance of the i-th requirement. As a result, the overall expected value and variation of the opinions over the importance of the i-th requirement rik can be calculated by the following equations. m 1 ri  m  s r  k ik (3)  s  k 1 k k 1 m  s  Varr  k 2 ik Varri  k 1 2 (4)  m     sk    k 1  2.3 Evaluation of system quality A quality system must be able to address the initial requirements. Using the proposed method of this paper, the designer can quantify the stakeholders’ opinion and evaluate how successfully the system addresses those requirements. For this purpose, stake- holders evaluate the system quality with regard to the system requirements and this information is labeled as d1 , d2 ,..., di , where d i represents the stakeholders’ opinion over the i-th requirement. The collected data is shown by stochastic variables di1 , di2 ,..., dim , where d ik presents the k-th stakeholder’s opinion over the importance of the i-th requirement. As a result, the overall expected value and variation of the opinions over the system quality with regard to the i-th requirement d i is calculated by the following equations. m 1 di   m  r d  k ik (5)  rk  k 1 k 1 m  r  Vard  k 2 ik Vardi   k 1 2 (6)  m    rk    k 1  And the overall system quality ( sq) and its variation can be shown through the equations below. n  d  i E[ sq]  i 1 (7) n n  Vard  i Varsq  i 1 2 (8) n 1. Algorithm The block diagram of workflow for evaluation of system quality is shown by Figure 4. It shows three main steps to evaluate system quality. The first step, which is of essential importance, is to identify the stakeholders and their requirements. Then the stakeholders and the realized re- quirements are ranked. Having this data, the system quality is evaluated. Identify Rank stakeholders/ Evaluate the stakeholders/ requirements system quality requirements Fig. 4. The functional block diagram for the algorithm. The following steps present the ranking process for requirements: o List m stakeholders. o Draw tables and list stakeholder (s1 , s2 ,..., sm ) using the numeric or verbal for- mat shown in Figure 3. o Ask the stakeholders to fill the tables. This step concludes m series of tables. Use sk format to label the collected information. o Use Equation 1-2 and calculate  k  and Var k  . j o List n requirements. o Draw tables and list requirements (r1 , r2 ,..., rn ) using the numeric or verbal for- mat shown in Figure 3. o Ask the stakeholders to fill the tables. This step concludes m series of tables. Use rik format to label the collected information. o Use Equations 3-4 to calculate ri  and Varri  for each requirement ri . o Draw tables and list requirements using the numeric or verbal format shown in Figure 3. o Ask the stakeholders to evaluate the system against the requirements and fill the tables. This step concludes m series of tables. Use d ik format to label the col- lected information. o Use Equations 5-6 to calculate di  and Vardi  . o Use Equations 7-8 to calculate the overall system quality and its variation. o If new stakeholders or values are realized, reiterate from the first step. Reuse of the collected information is possible. This process integrates the collected information and results an overview to a sys- tem designer for sorting the requirements based on the stakeholders’ opinion. Next section presents an example application that shows the process and expected out- comes. 3 Example application This section presents an example application to describe the proposed method. This example presents a stair-mobility project. This example shows an early estimation of the design quality in early phases of a project lifecycle where usually a high uncer- tainty level is present. A company in cooperation with TUDelft defined this project, and a team of stu- dents worked on this project and an individual designer finalized it. The aim of this project was developing a concept for chair stair-lifts used by adults in the Western Europe with minor disabilities. This could represent a target group that start feeling pain in hips, knees or ankles but also consider fatigue and fear issues during the as- cend or descend of staircase. Based on the stakeholder’s requirements and designers’ vision, several require- ments were defined to ensure desired functions. For demonstration purpose, we refer to two of them: natural interaction and ergonomics. Natural Interaction prevents stig- mata surrounding stair lifts, and ergonomics ensures that the product generates a natu- ral interaction with its user. These requirements are illustrated in Figure 5. This figure shows the opinion of three stakeholders, and they quantified the stair lift system using our proposed method. Here in this paper they are evenly graded for demonstration purpose, and a numerical scale has been used in the figure. Natural Interaction Ergonomics Personal Personal Medical Medical Retailer Retailer Company Company 0 20% 40% 60% 80% 100%0 20% 40% 60% 80% 100% Figure 5. (a) Three stakeholders present their expert opinion on the proposed design for stair-lift system. Applying the algorithm explained in Section 2 results in Table 1. This table pre- sents the integrated and concluding results. Two design requirements and three expert opinions on these requirements are presented in this table. Table 1. This table presents the integrated results for requirements and system quality. System requirements System quality Expected values Standard Expected Standard for require- deviation of value of deviation of Requirements ments requirements quality quality (E[ r ])% i (Var[ r ])%i (E[sq])% (Var[sq] )% Natural interac- 71.6 17.3 tion Ergonomics 63.7 24.5 67.7 29.9 Conclusions This study describes a methodology to measure system quality on a pluralistic basis. It embeds the importance of design stakeholders and requirements. The proposed meth- od enables and encourages a designer to communicate with stakeholders or experts and collect certain or uncertain information, combine this information and valuate system quality. The application of this method has been shown through the ColdFacts project. The proposed approach promotes the probabilistic thinking and establishes the principals of a method for using uncertain information based on the probability theo- ry. This method facilitates information collection and information integration in large, complex or high-tech systems[13]. Furthermore, it can be integrated with some cur- rently used methods in system design or systems engineering. References 1. Haskins, C. Systems engineering handbook. 2006. INCOSE. 2. Rajabalinejad, M. and C. Spitas, Incorporating Uncertainty into the Design Management Process. Design Management Journal, 2012. 6(1): p. 52-67. 3. Ronkainen, I.A., Criteria changes across product development stages. Industrial Marketing Management, 1985. 14(3): p. 171-178. 4. Pahl, G., W. Beitz, and K. Wallace, Engineering design: a systematic approach. 1996: Springer Verlag. 5. Sen, P. and J.B. Yang, Multiple criteria decision support in engineering design. Vol. 4. 1998: Springer London. 6. Spitas, C., Analysis of systematic engineering design paradigms in industrial practice: A survey. Journal of Engineering Design, 2011. 22(6): p. 427-445. 7. Balachandra, R. and J.H. Friar, Factors for success in R&D projects and new product innovation: a contextual framework. Engineering Management, IEEE Transactions on, 1997. 44(3): p. 276-287. 8. Roozenburg, N.F.M. and J. Eekels, Product design: fundamentals and methods. 1995: John Wiley & Sons. 9. Pacheco, C. and I. Garcia, A systematic literature review of stakeholder identification methods in requirements elicitation. Journal of Systems and Software, 2012. 85(9): p. 2171-2181. 10. Christel, M.G. and K.C. Kang, Issues in requirements elicitation. 1992, DTIC Document. 11. Heemels, W., et al., The key driver method. Boderc: Model-Based Design of High-Tech Systems, edited by W. Heemels and GJ Muller, 2006: p. 27-42. 12. Salado, A. and R. Nilchiani, Contextual-and Behavioral-Centric Stakeholder Identification. Procedia Computer Science, 2013. 16: p. 908-917. 13. Heemels, W., E. vd Waal, and G. Muller, A multi-disciplinary and model- based design methodology for high-tech systems. Proceedings of CSER, 2006. 14. Whitten, J.L., V.M. Barlow, and L. Bentley, Systems analysis and design methods. 1997: McGraw-Hill Professional. 15. Barron, F.H. and B.E. Barrett, Decision quality using ranked attribute weights. Management Science, 1996. 42(11): p. 1515-1523. 16. Takeda, E., K.O. Cogger, and P.L. Yu, Estimating criterion weights using eigenvectors: A comparative study. European Journal of Operational Research, 1987. 29(3): p. 360-369. 17. Saaty, T.L. and L.G. Vargas, The logic of priorities: applications in business, energy, health, and transportation. 1982: Kluwer-Nijhoff. 18. Barzilai, J., Deriving weights from pairwise comparison matrices. Journal of the Operational Research Society, 1997. 48(12): p. 1226-1232. 19. Yeh, C.-H., et al., Task oriented weighting in multi-criteria analysis. European Journal of Operational Research, 1999. 119(1): p. 130-146. 20. Buckley, J.J., Ranking alternatives using fuzzy numbers. Fuzzy Sets and Systems, 1985. 15(1): p. 21-31. 21. Tsai, W.C., A Fuzzy Ranking Approach to Performance eEaluation of Quality. 2011. Vol. 18. 2011. 22. Mitchell, H.B., Rnking-Intuitionistic Fuzzy Numbers. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 2004. 12(03): p. 377-386. 23. Zimmermann, H.J., Fuzzy sets, decision making and expert systems. Vol. 10. 1987: Springer. 24. Rajabalinejad, M. Modelling dependencies and couplings in the design space of meshing gear sets. 2012. 25. Choy, S.L., R. O'Leary, and K. Mengersen, Elicitation by design in ecology: using expert opinion to inform priors for Bayesian statistical models. Ecology, 2009. 90(1): p. 265-277. 26. O'Hagan, A., J. Forster, and M.G. Kendall, Bayesian inference. 2004: Arnold London. Concept 1 Interaction Ergonomic (w=0.4) (w=0.6) Ex- Lim- Expected Limits pected its (%) value (%) (%) value (%) 80- Company 75 65 70-60 70 70- Retailer 65 50 60-40 60 Med. Per- 80- 75 76 80-70 sonal 70