This paper focuses on coping with system quality in the early phases of design where there is lack of knowledge about a system, its functions or its architect. The paper encourages knowledge based evaluation of system quality and promotes probabilistic thinking. It states that probabilistic thinking facilitates communication between a system designer and other design stakeholders or specialists. It accommodates tolerance and flexibility in sharing opinions and embraces uncertain information. This uncertain information, however, is to be processed and combined. This study offers a basic framework to collect, process and combine uncertain information based on the probability theory. Our purpose is to offer a graphical tool used by a system designer, systems engineer or system architect for collecting information under uncertainty. An example shows the application of this method through a case study.
Nomenclature relative weight a random number representing the system quality over the i-th requirement a random number representing the system quality over the i-th requirement according to the k-th stakeholder expected value relative weight of requirements number of stakeholders number of requirements a random number representing the importance of the i-th requirement a random number showing the opinion of k-th stakeholder over i-th requirement a random number representing the importance of the k-th stakeholder a random number showing the opinion of j-th stakeholder over k-th stakeholder a random number representing the system quality variance
Introduction To deliver a quality system, a system designer should first identify, clarify, and document system requirements [1]. These tasks are performed in the earliest phase of a project life cycle and in the presence of a high level of uncertainty [2]. These requirements are not fixed and may change throughout the development stages [3]. On the other hand, some requirements e.g. maximization of benefits are explicitly or implicitly present in all design phases, and different terminology may be used for them. For example, design objectives or concept drivers are commonly used in the concept phase while the program of requirements or design criteria are more likely to be used in the embodiment phase. It is important to note that the requirements keep the focus of the design team on the most important design aspects or main needs and they provide references for the evaluation of system quality.
Therefore, system requirements have explicit roles through the design process. It is mainly because of the presence of a systematic approach [4] in this process and also because of the societal demands for meeting the standards [5] in engineered products. These are reflected in tools, processes and standards. An example is the popular method called the house of quality which relates user requirements to design requirements in order to ensure quality end-products. To achieve quality systems, designers need to define proper system requirements as early as possible [6] as they help judging the relevance of new information.
The evaluation of design alternatives are on the basis of these requirements. In other words, every design alternative has to be able to address the initial requirements. As a result, these requirements form criteria for evaluation of system quality. These design criteria may change through the design process and may have different degrees of importance. To assess system quality, a system designer has to rank them at the early stages of the design process. Ranking methods is of great value in decision models, and the use of multi criteria decision models (MCDM) typically involve criteria ranking. 1.1
To define system requirements, identification of stakeholders is one of the earliest steps. A review research by Pacheco and Garcia [9] confirms that an incomplete set of stakeholders may lead to incomplete requirements. A system designer has to pay attention to the problems arising from the scope, understanding and validation of requirements [10, 11] in the course of communication with stakeholders.
Figure 1 presents the functional diagram for identification of stakeholders and communication with them. It shows some new stakeholders may be realized through the course of communication with already-known stakeholders. To document the stakeholder’s needs and collected feedback , Salado and Nilchiani [12] suggest a set of questions for discovering new stakeholders in order to identify a complete set of stakeholders. Complex systems often include a relatively high number of stakeholders with different (conflicting) interests [13]. In such cases, the process of information elicitation, documentation and integration is a necessity to achieve informative conclusions.
Ranking of requirements (or criteria) based on their importance is well discussed in decision models. The use of multi criteria decision models typically involves a systematic ranking process as for instance indicated in [4, 14]. The influence of the ranking process on final decisions is for example explained in [15]. A review of subjective ranking methods shows that different methods cannot guarantee accurate results. This inconsistency in judgment explains difficulty of assigning reliable and subjective weights to the requirements. A systematic approach for ranking is described in [16] that is a generalization of Saaty’s pairwise structure [17]. Given the presence of subjectivity in the ranking process, sensitivity analysis of the design criteria is used to study the influence of variation and the ranking process on the decisions made [18]. Furthermore, some approaches e.g. the task-oriented weighing approach is effectively used. This approach is meant to limit the subjectivity of criteria weighting [19]. It suggests an algorithm to rank criteria objectively while considering the uncertainty in criteria weight [20]. The approach is based on introducing fuzzy numbers that imposes specified membership functions, which has been also used in [21, 22].
However, there is an obstacle for systems architectures or engineers in communication of the proposed methods with different stakeholders. The stakeholders can be individuals, corporations, organizations and authorities, with different fields/ levels of knowledge and experience [2]. The stakeholders have interest in the project and they desire to express their knowledge and expertise to improve the design. They also have expectations which have to be addressed at the end. Besides, it is advised to designers to rely on the experts in order to manage design uncertainties since it is proven that experts provide frameworks for making knowledge based decisions under uncertainty [23, 24]. This offers a human solution in terms of preferred alternatives. The uncertainty in importance of design requirements is also of human nature which should be reflected in the weighting process. To address these, we present the principles of our method through the next section.
Communicate the key requirements with stakeholders
Measure how much the
proposed system addresses requirements
Integrate the collected information
Propose a new alternative system Improve the proposed system Measure the overal system quality
No
A new design alternative ?
Yes System quality is satisfactory?
No
Yes The proposed system is chosen 1.3
To estimate the system quality, an intuitive method is used. Detail description of this method and its formulation emerge through the rest of this paper. It provides a consistent framework to value the system under uncertainty and observe how well the system addresses the stakeholder’s needs. This outcome provide valuable sources for the system designer or system engineer to monitor the strong and weak point of the system. Figure 2 presents the functional flow for evaluation of system quality in a pluralistic approach where the stakeholders’ opinion is fundamentally contributed to quality evaluation. In this perspective, communication plays an essential role and the proposed method aims to facilitate this communication. We aim to present a realistic and intuitive approach that can communicate to people with different fields of knowledge and expertise. The method must be transparent, easy to implement and readily adaptable by different users. For this purpose, graphs are used to effectively communicate with different users. The format presented in Figure 3 identifies the importance of a requirement according to a stakeholder’s opinion. The linguistic scale or the numeric scale can be used for the ease of communication, and one can assign the range of possible importance to a certain requirements. A probability distribution function (PDF) is assigned to this recorded data. Symmetric opinions are assumed here in this paper as described in [25, 26] and the collected data is treated as a random variable with a Gaussian distribution aiming to achieve set of a stochastic weight factors. Now m stakeholders assess the importance of the i-th requirement ri , and this information is represented by stochastic variables ri1 , ri2 ,..., rim , where rik presents the k-th stakeholder’s opinion over the importance of the i-th requirement. As a result, the overall expected value and variation of the opinions over the importance of the i-th requirement rik can be calculated by the following equations.
1 m E[ k ] m skj sk j1 k1 Var k
1 m m 2 j1 Varskj sk k 1 m 2 sk k 1 (1) (2) (3) (4) 2.3 A quality system must be able to address the initial requirements. Using the proposed method of this paper, the designer can quantify the stakeholders’ opinion and evaluate how successfully the system addresses those requirements. For this purpose, stakeholders evaluate the system quality with regard to the system requirements and this information is labeled as d1, d2 ,..., di , where di represents the stakeholders’ opinion over the i-th requirement. The collected data is shown by stochastic variables di1 , di2 ,..., dim , where dik presents the k-th stakeholder’s opinion over the importance of the i-th requirement. As a result, the overall expected value and variation of the opinions over the system quality with regard to the i-th requirement di is calculated by the following equations.
1 m di m rk dik rk k 1 k 1 m rk 2 Vardik Vardi k 1 m 2 rk k 1 n di E[sq] i1 n Vardi Varsq i1 n n2
And the overall system quality (sq) and its variation can be shown through the equations below.
The block diagram of workflow for evaluation of system quality is shown by Figure 4. It shows three main steps to evaluate system quality. The first step, which is of essential importance, is to identify the stakeholders and their requirements. Then the stakeholders and the realized requirements are ranked. Having this data, the system quality is evaluated.
requirements
(5) (6) (7) (8) This section presents an example application to describe the proposed method. This example presents a stair-mobility project. This example shows an early estimation of the design quality in early phases of a project lifecycle where usually a high uncertainty level is present.
A company in cooperation with TUDelft defined this project, and a team of students worked on this project and an individual designer finalized it. The aim of this project was developing a concept for chair stair-lifts used by adults in the Western Europe with minor disabilities. This could represent a target group that start feeling pain in hips, knees or ankles but also consider fatigue and fear issues during the ascend or descend of staircase.
Based on the stakeholder’s requirements and designers’ vision, several requirements were defined to ensure desired functions. For demonstration purpose, we refer to two of them: natural interaction and ergonomics. Natural Interaction prevents stigmata surrounding stair lifts, and ergonomics ensures that the product generates a natural interaction with its user. These requirements are illustrated in Figure 5. This figure shows the opinion of three stakeholders, and they quantified the stair lift system using our proposed method. Here in this paper they are evenly graded for demonstration purpose, and a numerical scale has been used in the figure.
lca lan ied rso M eP
Applying the algorithm explained in Section 2 results in Table 1. This table presents the integrated and concluding results. Two design requirements and three expert opinions on these requirements are presented in this table. This study describes a methodology to measure system quality on a pluralistic basis. It embeds the importance of design stakeholders and requirements. The proposed method enables and encourages a designer to communicate with stakeholders or experts and collect certain or uncertain information, combine this information and valuate system quality. The application of this method has been shown through the ColdFacts project.
The proposed approach promotes the probabilistic thinking and establishes the principals of a method for using uncertain information based on the probability theory. This method facilitates information collection and information integration in large, complex or high-tech systems[13]. Furthermore, it can be integrated with some currently used methods in system design or systems engineering. 1. 2. 3. 4. 5. 6. 7. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26.
Whitten, J.L., V.M. Barlow, and L. Bentley, Systems analysis and design methods. 1997: McGraw-Hill Professional.
Barron, F.H. and B.E. Barrett, Decision quality using ranked attribute weights. Management Science, 1996. 42(11): p. 1515-1523.
Takeda, E., K.O. Cogger, and P.L. Yu, Estimating criterion weights using eigenvectors: A comparative study. European Journal of Operational Research, 1987. 29(3): p. 360-369.
Saaty, T.L. and L.G. Vargas, The logic of priorities: applications in business, energy, health, and transportation. 1982: Kluwer-Nijhoff.
Barzilai, J., Deriving weights from pairwise comparison matrices. Journal of the Operational Research Society, 1997. 48(12): p. 1226-1232.
Yeh, C.-H., et al., Task oriented weighting in multi-criteria analysis. European Journal of Operational Research, 1999. 119(1): p. 130-146. Buckley, J.J., Ranking alternatives using fuzzy numbers. Fuzzy Sets and Systems, 1985. 15(1): p. 21-31.
Tsai, W.C., A Fuzzy Ranking Approach to Performance eEaluation of Quality. 2011. Vol. 18. 2011.
Mitchell, H.B., Rnking-Intuitionistic Fuzzy Numbers. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 2004. 12(03): p. 377-386.
Zimmermann, H.J., Fuzzy sets, decision making and expert systems. Vol. 10. 1987: Springer.
Rajabalinejad, M. Modelling dependencies and couplings in the design space of meshing gear sets. 2012.
Choy, S.L., R. O'Leary, and K. Mengersen, Elicitation by design in ecology: using expert opinion to inform priors for Bayesian statistical models. Ecology, 2009. 90(1): p. 265-277.
O'Hagan, A., J. Forster, and M.G. Kendall, Bayesian inference. 2004: Arnold London.
Interaction (w=0.6)
Expected value (%)
Limits (%) Company Retailer Med. Personal 75 65
Expected value (%) 65 50 70-60 60-40 80-70