How to Analyze Modeling Approach Comparison Criteria

Abstract

One possible nal goal of de ning a set of criteria to de ne modeling approaches [1] is to help people, especially from industry, picking up the good approaches or artifacts according to their own purpose. The authors of the comparison criteria have managed to get several di erent assessments made by defenders of particular modeling approaches. From our point of view the experiment is mature enough to support a factorial analysis of the criteria themselves. The goal of this paper is to present how such an analysis could be conducted and illustrate its usefulness. We have identi ed several key modeling concepts but we only focus in this document on the assessment of modeling approaches.

Context

to study instances (methods), variables (comparison criteria) and their modalities [ 2 ]. Two methods are close if questionnaires have been answered the same way. The focus will not be on instances (methods) per se but more on sets: are there groups of methods? We are not much interested in methods themselves but groups of methods inside the whole set, that is: analyzing methods and observe how they regroup and under which factors. We want to study the relationships between variables and the associations between modalities. A modality is the \value" a variable can take. Qualitative variables and quantitative ordinal variables have discrete values, and usually a nite set of modalities. Two modalities are close if they have been taken together often. Two comparison criteria characteristics that are often cited together by a group of individuals will geometrically appear close on the plot graphs generated by the factor analysis. We are looking for such plots clouds. One or more synthetic continuous variable(s) can be looked for by a PCA to summarize the qualitative variables. and interpret the relations between them. Using a representation by modalities is easier to show how vector dimensions separate or aggregate the di erent criteria and gives more precision. A nal goal is to characterize methods subsets by modalities of comparison criteria using a Hierarchical Cluster Analysis which is the logical and common follow-up to an MCA. We want to regroup the methods in a few number of classes corresponding to "pro les" of Comparison Criteria. The result is a hierarchical tree easy to interpret. Methods would appear as leaves, clustering into small branches, then bigger ones, etc., up to a trunk. Classes can then be described by the criteria variables and/or their modalities, by the factorial dimensions, or the individuals/methods. 3

Conclusion

This paper presents how comparison criteria could bene t of advanced statistical methods such as MCA and Hierarchical Clustering (see [ 3 ] for more details). Such tools can help give insights about many questions relating to similarities and di erences between modeling approaches, and/or comparison criteria characteristics complex relationships. Data collection can be enhanced too with a little revamping of the questionnaire so it better feeds the statistical tables and minimizes loss of information.

References

1. G. Georg , S. Ali , D. Amyot , B. H. Cheng , B. Combemale , R. France, J. Kienzle , J. Klein , P. Lahire , M. Luckey , A. Moreira , and G. Mussbacher . Modeling approach comparison criteria . Technical report , McGill U., Montreal, CAN, 2001 .

2. F. Husson , J. Josse , and J. Pages . Principal component methods - hierarchical clustering - partitional clustering: why would we need to choose for visualizing data ? Agrocampus Applied Mathematics Department , 2010 .

3. F. Mayart , J.-M. Bruel , and B. Hamid . Correlation analysis of modeling approach comparison criteria: methodology proposal . Technical report , 2013 . Available at http://www.irit.fr/publis/MACAO/IRIT