Methodology for Determining Stakeholders’ Criteria Weights in Systems Engineering Vikas Shukla and Guillaume Auriol CNRS, LAAS, 7 avenue du colonel Roche, F-31400 Toulouse, France Univ. de Toulouse, INSA, LAAS, F-31400, Toulouse, France vshukla@laas.fr gauriol@laas.fr Abstract. Multi criteria decision making involves evaluation of various alterna- tive solutions upon a set of criteria. The result of multi criteria decision making is the best alternative which secures the highest score with the predefined cri- teria. Usually, these criteria are weighted in an order to represent their stake in the final selection. In multi criteria decision making this step is very critical for the selection of the right product. Often these criteria are traced backed to a set of multi-disciplinary stakeholders participating in the evaluation process. Often these stakeholders differ upon their weighting of a particular criteria with other stakeholders. In this paper we provide a holistic criteria weighting method which allows to assimilate the different criteria weights from different stakeholders to provide a single set of criteria weights, uniformly acceptable by all stakeholders. This paper shows how the criteria weights can be simulated and used priorly to find the design solutions. The simulations results can be used to accept or refute the particular design alternative. 1 Introduction Decision making is pervasive phenomenon in systems engineering (SE) projects. Some decisions involve single decision maker (DM). The responsibility of such a decision depends upon the sole single DM. Often, SE decisions involve more than one DM, thus adding fabric of complexity to the problem. Decision making becomes more complex, when multiple criteria get involved in the problem. Such problems, where multiple criteria and multiple decision makers are involved in a problem are called multi criteria decision making or analysis (MCDM/MCDA) problems. A systems engineering project involves multiple stakeholders and multiple crite- ria decision analysis [1]. Such problems need attention and a formal methodology, to provide pertinent solutions. Variety of methodologies exist in the literature which pro- vide more or less acceptable solutions, such as the famous AHP technique [2], Multi- attribute utility theory [3], ELECTRE Methods [4–6], PROMETHEE [7], TOPSIS[8], etc. In this paper, we are concerned with only the first step of the decision making problem, i.e., criteria weighting. In this paper we present a methodology for criteria weighting in the context of a systems engineering projects. Our criteria weighting ap- proach can provide weights for a variety of decision making techniques. Our approach is based on the classical preference modelling technique given by Fishburn et al. [9]. Proceedings of the Posters Workshop at CSD&M 2013 1 Methodology for Determining Stakeholders’ Criteria Weights in Systems Engineering A systems engineering project typically involves a crowd of multi-disciplinary stake- holders. Success of a project depends upon the decisions and choices made during the analysis of architectures & alternatives, and during the selection of components during the detail design phase [10]. Systems engineering is about making the right decisions to achieve the development of a product which is exactly demanded by their clients and stakeholders. These right decisions do not come automatically from a thin air, rather precise metrics are needed to evaluate the appropriate alternatives and right design components. As these decisions are based on these multiple criteria, they need to be weighted to measure their impact on the final decision choice. This task of providing weights to the criteria may seem trivial for a single decision maker, but when multiple stakeholders are involved this task becomes fairly difficult. As the different stakehold- ers differ upon the weights for various criteria, while each stakeholder being correct in his own view. It is interesting to see that industries seldom use techniques which demand high cognitive loads on DMs. Even if a techniques is more correct technically but leads to high cognitive load, it will hardly find usage in industry. Industries prefer simple techniques to pacify majority of its non-technical stakeholders. Our technique claims to pose less cognitive load to the stakeholders as compared to the other technique. In this paper, we provide a technique which allows to assimilate the various criteria weights from the different stakeholders to provide a single array of criteria weights which can be uniformly accepted by all the different decision makers or stakeholders. The major contributions of this paper are as follows: — It provide a holistic way to integrate the different criteria weights of different stake- holders to provide a single weights using the classical preference modelling. — It show that how all stakeholders are uniformly satisfied with the proposed tech- nique. This paper is organized as follows: Section 2 presents the criteria weighting prob- lem. Section 3 presents the state of art of criteria weighting problems. Section 4 presents our novel approach. Section 5 presents an example of our approach. Section 6 discusses about the advantages and disadvantages of our approach, and the industrial readiness assessment. Section 7 concludes and presents future perspectives. 2 Criteria Weighting Problem in Systems Engineering In a systems engineering project, it is of great importance that most of stakeholders are satisfied with the various decisions taken during the product development and with the final resulting end product. A higher satisfaction among the stakeholders can be guaran- teed if the various stakeholders criteria weights are taken into account in a transparent and holistic manner. The criteria weighting is critical part of the MCDM problem during analysis of alternatives in a system engineering project. Different stakeholders own different views towards the different criteria, their perception of weights differ from each other which makes it very tedious and difficult to come up with an agreement on a particular set of criteria weights. This often causes the conflicts among the stakeholders, which may halt Proceedings of the Posters Workshop at CSD&M 2013 2 Methodology for Determining Stakeholders’ Criteria Weights in Systems Engineering the progress of the project, if it remains unresolved. The criteria weighting problem refers to the problem of generating a single array of criteria weights from multiple arrays of criteria weights emerging from different stakeholders. Often, such problems start with demanding the DMs’ weights for all the involved criteria and then provide a mechanism to find the mean weight. This approach assumes that the DMs’ know their particular weights, which is often not supported by any evidence. The criteria weighting problem faces four critical challenges: — Evidence of validity for criteria weights. — Transparency of DMs’ participation in criteria weighting. — Scalability of criteria weighting. — Acceptable Cognitive load on DMs. Evidence of validity for criteria weights refers to the means which can prove that the assumed criteria weight is correct for a DM or to be able to reason why it is correct for a particular DM. Transparency of DM’ participation in criteria weighting is necessary to atleast make sure that the criteria weights are not illicitly decided in a manner to favour a particular DM’ preferences, or in other words to make sure that every participant DM is satisfied with his bit of contribution in the process. The scalability of the techniques for a sufficiently large number of criteria which may arise in a systems engineering project. In a systems engineering project the number of broad criteria hardly rise more than ten. These broad criteria later can be divided in to multiple criteria in next level. The criteria weighting technique should be able to address this hierarchy of criteria. The Fourth most important challenge is about the amount of cognitive load that a technique poses on the DM. If majority of DMs find it difficult to use the technique for weighting the criteria, then the technique has less chances to be used in the process. Whereas, if a technique which poses less cognitive load on the DMs, can easily win over others and find acceptability in the approach, even if it provides less accurate results. 1 c2 c ··· n c   St1 w11 w12 · · · w1n St2  w21 w22 · · · w2n  w     ..  .. .. . . ..  =⇒ Gr w1 2 · · · wn w (1) . . . . .  Stm wm1 wm2 · · · wmn Eq (1) represents the criteria weighting problem mathematically. In Eq (1) Sti refers to i’th stakeholders, c j refers to the j’th criterion, and wi j refers to the weight of the i’th stakeholder for the j’th criterion. 3 State of art of Criteria Weighting Techniques In the literature of multi-criteria decision making, often the problem of criteria weight- ing comes along the multi-criteria decision analysis. The vast literature on criteria weighting techniques found mentions of large number of different techniques, but as in this paper we try to solve criteria weighting technique for systems engineering, we’ll Proceedings of the Posters Workshop at CSD&M 2013 3 Methodology for Determining Stakeholders’ Criteria Weights in Systems Engineering stay with the techniques which have earlier found usage in the systems engineering project. Criteria weighting methods can be divided into two types internal and external. Popularly known internal methods are Entropy method, Regression method, Variance method, and LINMAP method. The external methods can further be divided into two types with DMs matrix such as: Swing method, Utility method, and without DMs matrix such as: SMART Method [11], SMARTER method, Eigenvector, Minimum weighted squares method, weighted sum method (WSM), weighted product method (WPM) [12]. Reference point method for vector optimization, AHP method, PROMETHEE [7] method are equally popular to solve this problems. There are many other techniques based on the visual aids for selecting the right stakeholder weight. The previously mentioned techniques depend completely on human judgement about the preference weighting to get the weight, but seldom human can provide reasoning about the preference weight. To best of our knowledge, in the literature of decision making, there is no comprehensive way to reason the weight of the decision maker, to allocate systematically the weights at the various levels of the preferences. In this respect our work is very different from the previous work. We provide a mechanism to weight the human perception and link it with the mathematical formulations to de- rive the criteria weight. To have more robust criteria weighting, we have used multiple algorithms to find the weight. Another aspect of the criteria weighting problem is about the uniform satisfaction among the stakeholders, pointed out by the well known Arrow’s impossibility theo- rem [13], which states that DM can find no procedure that can combine individual’s rankings of alternatives to obtain single unified rankings. Recent research works have tried to address criteria weighting problem using many other different techniques such as card playing [14], using hybrid approaches of many different techniques, but in al- most all of them they started with assumption about the particular criteria weight. 4 Proposed Technique The proposed technique uses the classical preference modelling [9, 15, 16] for repre- senting stakeholder preferences.The proposed approach is four step process. We as- sume that all the necessary DMs (stakeholders) and criteria are already identified for the system under study. 4.1 Prerequisite to technique – ‘D’ is the set of decision makers (DMs) involved in the concerned conflict, where 2 ≤ |D| < ∞. – ‘C’ is the set of distinguishable criteria, satisfying 2 ≤ |C| < ∞. For each stakeholder set ‘C’ consists of three distinct subsets H, M, L, such that: {H}∪{M}∪{L} = {C}, and {H} ∩ {M} = {M} ∩ {L} = {H} ∩ {L} = {φ }. Where cardinality of each set can be different. The criteria subsets are such that the perceived utility is in the order p(H) > p(M) > p(L). – For each i ∈ D, there is set of preference relationships {i , >i , ∼i } defined over ‘C’. Proceedings of the Posters Workshop at CSD&M 2013 4 Methodology for Determining Stakeholders’ Criteria Weights in Systems Engineering – For each i ∈ D, a complete binary relation i on C, specifies DM i’s strong preference over C. If s,t ∈ C, then s i t means the DM i strongly prefers s to t. – For each i ∈ D, a complete binary relation >i on C, specifies DM i’s weak preference over C. If s,t ∈ C, then s >i t means the DM i weakly prefers s to t. – For each i ∈ D, a complete binary relation ∼i on C, specifies DM i’s indif- ference over C. If s,t ∈ C, then s ∼i t means that s to t are indifferent to DM i. – The relation  and > are asymmetric, ∼ is symmetric and reflexive and the triple {i , >i , ∼i } is complete. – The preference relationships can be transitive for a particular stakeholder. 4.2 Technique The proposed approach can be divided into four steps as follows: Criteria Categorization All the decision makers involved in the decision process should categorize the agreed set of criteria in to three sub sets high preference (H), medium preference (M), low preference (L), according to their perception of utility of the criteria. Hence, the sorting of criteria is such that the perceived utility is in the order p(H) > p(M) > p(L). Preference modelling over criteria Each DM i ∈ D, creates the preference matrix Pi , over the criteria j ∈ C, given by Eq.(2). The previously created three subsets h, m, l are used as reference for creating the preference matrix. 2c1 c j ··· c   c1 0 p12 · · · p1 j c2 p21 0 · · · p2 j    Pi = .  . . . (2) ..  .. .. . . ...   cj p j1 p j2 · · · 0 where the value pab of a DM i ∈ D is given by Eq(3) below:    2 If a is strongly preferred criterion than b    1 If a is weakly preferred criterion than b pab = 0 If a is indifferent to criterion b (3)     −1 If a is weakly disliked criterion than n  −2 If a is strongly disliked criterion than b As the matrix Pi is skew-symmetric, it is sufficient to express the value of pi j to get pi j and vice versa. Once Pi matrix is available, the criteria are ranked in order with maximum number of 2, 1, 0, −1 and − 2 respectively. It is possible for a DM to have two or more criteria securing exactly same ordinal ranking, depending upon the entries in matrix. Proceedings of the Posters Workshop at CSD&M 2013 5 Methodology for Determining Stakeholders’ Criteria Weights in Systems Engineering Simulating solution The criteria weighting evaluation is always a difficult step for high level stakeholders who have simply limited knowledges. So, we propose a simulation of solutions which would be retained if weights are maintained. This high level simulation is individually processed for each stakeholder. A stakeholder could display the effect of his/her choice of criteria weights. This simulation is based on data bound to some sys- tem components. at the beginning of a new project, system experts could draw a very simple organic (or functional) system breakdown. During this step, they can add on each high level component approximative values for criteria. This approach is suitable because system experts have skills on available technologies. It doesn’t need to pre- cisely know the features of each components, actually it doesn’t need to precisely know the real system breakdown. At this step, system experts can only categorize the parts of solutions into sets of criteria values, for example they can chose between low, medium or high. The aim of this simulation is to provide stakeholders an approximative solution if they confirm their criteria weighting. Let’s illustrate this approach with a very simple example. This one deals with the design of a new individual powered transport system proposed to city inhabitants for their short distance journeys. Let the stakeholders be: end user, technical engineer of the product development company, business engineer of the product development company, and city representative for public transport system. In a very simplified way, let design criteria be: economical, environmental, reliable, in- novative, and repairable. Without knowing the details of solution, design experts have some element of solution as the breakdown as concerned, for example: chassis, engine, energy tank, and transmission. For each element of this breakdown, they are able to give some categories of solutions. Fig.1 gives an example of partial breakdown. For each categories of solution, system experts could add approximative criteria values. The chassis could be in carbon, magnesium, aluminum or in polyethylene. For each technology, subject matter experts are able to approximatively weight each element of this structure. For each technology, the expert is able to make a “categorization” for alternative solutions based on the proposed criteria set, completely without the specifications, it is not accurate sizing, but he knows a ranking of basic technologies. This simulation does not attempt to validate a solution but tries to establish a more precise relationship between a criterion and a family of solution. Consider the choice of the two following stakeholders. The sales engineer wants a conveyance which is economical, reliable and robust. It will therefore fill the his preference matrix given by Eq.(2) accordingly. He may not have realized that his criteria preference matrix guides the choice solution to plastic chassis and using an old noisy and polluting engine. The ecological criterion may not be a priority for him, but the solution proposed would not be very attractive commercially. It would be difficult to place the product in the market demand. Similarly, considering the mayor of the city, responsible for pub- lic transport system. He wants an ecological and innovative solution, that represents modernity and dynamism of the city. He does not realize that his preference matrix may indicate towards the solution of the latest technologies that are likely to be less reliable and expensive to maintain for a product designed for intensive use. In these two exam- ples, we see that the perception of a stakeholder to the criteria is necessarily biased. The head of the city was not opposed to having a robust and reliable, but he can interpret Proceedings of the Posters Workshop at CSD&M 2013 6 Methodology for Determining Stakeholders’ Criteria Weights in Systems Engineering Fig. 1. Solution according business engineer center of interest these criteria with his knowledge, which are a priori limited on technologies. His per- ception represents the criteria “robust” and “reliable” with respect to its repository of life for the project in question: a modern car, a scooter, ... To summarize, this simula- tion is a means proposed for each stakeholder should help when choosing a value for the weight of importance of the design criteria. Simulation offers the ability to view a draft solution based on criteria elements informed by expert systems. Generating scores The scores are generated using a simple process consisting of a maximum of j number of moves, where j = |C|, in which every DM starts marking the criteria set C. The scores can be generated using the utility functions shown in Figure 2, depending upon the risk averseness, proneness or neutrality of the decision maker. Below, we define some risk neutral utility functions that we employ in our example in Section 5. 1. Every DM starting with the most preferred configuration to the least preferred state. 2. Individual score of every configuration is calculated and the one with the highest value. The scores of every configuration can be calculated using the equation (4), and normalized using (5). (k − j.i) score(ci ) = (4) k Proceedings of the Posters Workshop at CSD&M 2013 7 Methodology for Determining Stakeholders’ Criteria Weights in Systems Engineering Ui (Xi ) Score More Risk Averse 14 RA RP Less Risk Averse 12 10 8 RN1 RN2 6 Risk Neutral Less Risk Prone 4 RN3 2 More risk prone Xi 0 1 2 3 4 5 6 Rank Fig. 2. Decreasing Utility function [3] Fig. 3. Score vs. Rank where k, j are index terms used to generate different types of utility functions. score(ci ) Normalised score(ci ) = (5) ∑i ci Once the scores are calculated they are normalized to obtain the score. 5 Example To show the ease of usage of our technique, we present here an example of Criteria weighting problem. We take an example of a hybrid car. The design team in a auto- motive industry wants to design a hybrid car and they need to weight design criteria to proceed with the selection of alternatives. Let the set of DMs be D = {a, b, c, d, e}, the criteria set be C = {c1 , c2 , c3 , c4 , c5 , c6 }. The interpretation of various criteria is ex- plained in Table 1. Table 1. Design Criteria Table 2. Design Criteria categorization Criteria Description DM h m l c1 Flexibility of usage a c1 , c5 c2 , c4 c6 , c3 c2 Maintainability b c4 , c6 c1 , c2 c3 , c5 c3 Robustness c c3 , c4 c2 , c1 c5 , c6 c4 Aesthetic value d c6 , c3 c2 , c1 c4 , c5 c5 Environment Friendly c6 Ease of manufacture Step 1 The DMs categorize criteria set according to their perception of criteria as shown in Table 2. Proceedings of the Posters Workshop at CSD&M 2013 8 Methodology for Determining Stakeholders’ Criteria Weights in Systems Engineering Step 2 The DMs create their preference matrices according to their categorization from step 1 as shown in Table 2. The preference matrix of stakeholder a, b, c and d are shown in Eq.(6),(7), respectively.  c1 c2 c3 c4 c5 c6   c1 c2 c3 c4 c5 c6  c1 0 1 2 1 1 2 c1 0 1 2 −2 1 −2 c2  −1 0 2 1 −2 2   c2  −1 0 1 −2 2 −1    c  −2 −2 0 −1 −2 0    , Pb = c3 −2 −1 0 −2 1 −2   Pa = 3  (6) c4 −1 −1 1 0 −1 1  c4 2 2 2 0 2 1     c5 −1 2 2 1 0 1  c5 −1 −2 −1 −2 0 2  c6 −2 −2 0 −1 −1 0 c6 2 1 2 −1 2 0  c1 c2 c3 c4 c5 c6   c1 c2 c3 c4 c5 c6  c1 0 1 −1 −1 1 1 c1 0 −1 −1 1 1 −2 c2  −1 0 −1 −1 1 1   c2  1 0 −1 1 1 −2    c 1 1 0 1 2 2   , Pd = c3 1 1 0 2 2 −1   Pc = 3  (7) c4 1 1 −1 0 2 2  c4 −1 −1 −2 0 1 −2     c5 −1 −1 −2 −2 0 1  c5 −1 −1 −2 −1 0 −2  c6 −1 −1 −2 −2 −1 0 c6 1 1 1 2 2 0 Step 3 The DMs carry out the criteria marking according to their preference matrices. Criteria with higher preference are marked early and one with lower are marked later in the order of preference. c1 c2 c3 c4 c5 c6   a1 0 c b 0 d   2 0 0 d c a b      3 b c,a,d 0 0 0 0    Marking =    (8) 4 c,d b 0 a 0 0      5 0 0 b d c a    6 0 0 a 0 b,d c After the marking is done, the DMs agree on a set of score generating algorithm. By using the score generating functions as mentioned in Eq.(4) with suitable index terms k and d the scores are obtained. Then normalized using Eq.(5). Similarly other scores are obtained by the various risk averse, risk prone and risk neutral functions by changing the index terms k and d. The obtained scores are are shown in Table 3, using three risk neutral utility functions: Risk − N1, Risk − N2, Risk − N3; risk averse utility function: Risk −A, risk prone: Risk −P and a rank order centroid function (ROC). Figure 3, shows the resulting graphs obtained through various utility functions used. Once the criteria weight matrix is available, the final criteria can be obtained by either by the mean of weights obtained for each criteria or by accepting any particular Proceedings of the Posters Workshop at CSD&M 2013 9 Methodology for Determining Stakeholders’ Criteria Weights in Systems Engineering Table 3. Design Criteria scores Algorithms (Scores-Normalized Scores) Criteria Risk-N1 Risk-N2 Risk-N3 Risk-P Risk-A ROC c1 16/6 | (0.19) 20/7 | (0.182) 24/8 |(0.182) 24|(0.197) 33|(0.168) 10 |(0.163 ) c2 15/6 | (0.1785) 19/7 | (0.173) 23/8 |(0.1742) 18|(0.1475) 41 |(0.209) 10-(0.163 ) c3 14/6 | (0.167) 18/7 | (0.163) 22/8|(0.167) 21 |(0.172) 31|(0.163) 10 |(0.163 ) c4 16/6 | (0.1904) 19/7 | ( 0.2) 24/8|(0.182) 23 |(0.1885) 39|(0.1989) 10|(0.163 ) c5 9/6 | (0.107) 13/7 | (0.118) 17/8-(0.123) 12 |(0.0983) 17|(0.097) 10 |(0.163 ) c6 14/6 | (0.167) 18/7 |(0.164) 22/8 |(0.167) 24|(0.197) 31|(0.163 ) 10|(0.163 ) criteria array. In this example we took the mean of the four array of criteria weights and the resultant criteria weight array is represented by Eq.(9). c1 c2 c3 c4 c5 c6   sol 0.1838 0.176 0.1664 0.192 0.109 0.1716 (9) entropy 0.1558 0.2254 0.2266 0.1483 0.0379 0.2057 6 Discussion Our approach tried to provide a methodological solution to one of the ignored problems of multi-criteria decision making. Most of MCDM techniques mentioned in the litera- ture assume that they can choose criteria weight just by barely looking at them. Often, this is not the case. We provide a technique which allows multiple DMs to achieve an appropriate array of criteria weights, while remaining in synergy with other DMs crite- ria weights. One of the benefit of our approach is about the scalability of the technique, it can easily take in account a large number of DMs with different perception of criteria weights and large number of criteria. But with respect to systems engineering, it would not be fruitful to employ very large number of criteria. There are seldom more than twelve criteria in a project. If the number of criteria are too many it would be advised to create a hierarchy of criteria and then use our approach recursively. The addition to ease of application, our technique provides other multiple benefits. It helps a DM to understand his perception of criteria better and of the other DMs. It also helps to avoid conflicts among the DMs, which often arrive during the decision mak- ing process. The transparency of the approach allows easy negotiation of the criteria weights and hence maximum number of DMs are satisfied with their contribution. The low cognitive load that our technique demands can be important factor for acceptabil- ity of the technique. Our technique can be coupled with a variety of decision-making approaches, such as PROMETHEE, TOPSIS, WSM, WPM. The decision makers are free to propose their own utility functions or scoring al- gorithms based on the ordinal ranking achieved. This allows a more conducive envi- ronment for criteria weight negotiation, as the whole process is transparent, with no black-box process involved. Proceedings of the Posters Workshop at CSD&M 2013 10 Methodology for Determining Stakeholders’ Criteria Weights in Systems Engineering In our approach, we considered that all the DMs have equal weight, but this may not be the case in a real systems engineering project. It is possible that the different stakeholders in the projects have different importance. But this is only upto Project manager to allocate the weights to various DMs. Our approach offers possibility to weight the different DMs involved in a project. In this case only one preference matrix would be required, i.e., only of Project manager. While, weighting DMs it would be advised to use a set of risk averse and risk neutral utility functions to weight them, in order to avoid penalizing the low ranked DMs. There can be arguments that the utility score generation algorithms cannot be ac- cepted as weighting mechanism. But the literature shows that, every individual DM has a utility function, which he uses consciously or unconsciously while providing scores directly as it usually happens. Here we attempted to provide a formalism to use this conscious/unconscious utility function, in order to help other DMs to understand the perceptions of each other. The benefit that our approaches provides over other is that, it involves the DMs to methodologically provide the ordinal ranking by first demand- ing them to attempt to categorize them in three categories. This categorization provides input to the next step, which can again be validated by corresponding DM; depending on the preference of the various DM’s, a range of scoring algorithm can be applied and weights can be obtained. In the beginning our approach demands slightly more participation from the DMs, as compared to the other approaches, but once the weights are methodologically ob- tained they are certainly more reliable then the other contemporary approaches with least amount of conflict. Better decisions allow to design the right systems, with more stakeholders getting more confident about their product. 7 Conclusion and Future perspectives In this current work, we have provided a systems theory of how the criteria weights can be obtained using the classical theory of preference modeling. We call our technique Utility Rank Order Weighting (UROW) technique. This approach provides multiple benefits with compared to other existing approaches. Usually in systems engineering project, the engineers rely upon their intuition to provide weights, and later use other technique to combine the different DMs’ preferences. Our approach provides a formal- ism to this systems engineer intuition and hence provides the reasoning for the various weights achieved. Our approach is very easy to understand and use, and demands very low cognitive load from the engineers and stakeholders. It allows to formally provide the scores using the DM’ drawn utility functions: risk prone, risk averse, or risk neutral; it provides a mechanism to combine them together to come up with a uniformly accept- able solution. In future, we look forward to link the simulation of the DMs’ preferences with the design library, in order to shorten the decision time. Our approach can easily be applied to the class of methods which require information on the attributes to carry out a decision analysis. Proceedings of the Posters Workshop at CSD&M 2013 11 Methodology for Determining Stakeholders’ Criteria Weights in Systems Engineering References 1. Sage, A., Armstrong, J.: Introduction to systems engineering. Wiley series in systems engi- neering. Wiley (2000) 2. Saaty, T.L.: How to make a decision: the analytic hierarchy process. European journal of operational research 48(1) (1990) 9–26 3. Keeney, R., Raiffa, H.: Decisions with Multiple Objectives: Preferences and Value Trade- Offs. Cambridge University Press (1993) 4. Roy, B.: Classement et choix en présence de points de vue multiples (la méthode electre). Riro 2(8) (1968) 57–75 5. Roy, B.: Electre iii: Un algorithme de classement fondé sur une représentation floue des préférences en présence de critères multiples. Cahiers du CERO 20(1) (1978) 3–24 6. Roy, B., Bertier, P.: La méthode electre ii(une application au média-planning...). (1973) 7. Brans, J.P., Mareschal, B.: Promethee methods. Multiple criteria decision analysis: state of the art surveys (2005) 163–186 8. Hwang, C.L., Yoon, K., et al.: Multiple attribute decision making: methods and applications: a state-of-the-art survey. Volume 13. Springer-Verlag New York (1981) 9. Fishburn, P.: Utility theory for decision making. Technical report, DTIC Document (1970) 10. Ulrich, K., Eppinger, S.: Product design and development. McGraw-Hill (2008) 11. Edwards, W.: How to use multiattribute utility measurement for social decisionmaking. Systems, Man and Cybernetics, IEEE Transactions on 7(5) (1977) 326–340 12. Triantaphyllou, E.: Multi-criteria decision making methods: a comparative study. Volume 11. Kluwer Academic Publishers Dordrecht (2000) 13. Arrow, K.: Social choice and individual values. Volume 12. Yale university press (1963) 14. Figueira, J., Roy, B.: Determining the weights of criteria in the electre type methods with a revised simos’ procedure. European Journal of Operational Research 139(2) (2002) 317–326 15. Roy, B.: The outranking approach and the foundations of electre methods. Theory and decision 31(1) (1991) 49–73 16. Roy, B., Vincke, P.: Relational systems of preference with one or more pseudo-criteria: Some new concepts and results. Management Science (1984) 1323–1335 Proceedings of the Posters Workshop at CSD&M 2013 12