Assessing Collaborative Modeling Quality Based on Modeling Artifacts?

Abstract

Collaborative modeling uses and produces modeling artifacts whose quality can help us gauge the e ectiveness and e ciency of the modeling process. Such artifacts include the modeling language, the modeling procedure, the products and the support tool or medium. To effectively assess the quality of any collaborative modeling process, the (inter-) dependencies of these artifacts and their e ect on modeling process quality need to be analyzed. Although a number of research studies have assessed and measured the quality of collaborative processes, no formal (causal) model has been developed to assess the quality of the collaborative modeling process through a combination of modeling artifacts. This paper develops a Collaborative Modeling Process Quality (CMPQ) construct for assessing the quality of collaborative modeling. A modeling session involving 107 students was used to validate and measure the quality constructs in the model.

Keywords

• Collaborative Modeling
• Modeling Process Quality
• Modeling Artifacts
• Instrument Validation
• Structural Equation Modeling

-

While a number of approaches have been developed to measure and evaluate the quality of a collaborative modeling process, e.g. its successfulness [ 1 ] and users' satisfaction [ 2 ], there has not been any study that integrates the assessment of various modeling artifacts to determine the quality of a collaborative modeling process. Driven by the need to determine the e ciency and e ectiveness of the modeling process, we propose an evaluation method that indeed integrates the assessments as an alternative method for determining the quality and successfulness of, and users' satisfaction with, a modeling process. Our assessment approach evaluates quality of the collaborative e ort through the quality of the modeling artifacts, used in and produced during collaborative modeling. Speci cally, we look at the following constructs: Perceived Quality of the Modeling language (PQML), Perceived Use of the Modeling Procedure (PUMP), Perceived Quality of the Modeling Product (PQEP) and Ease of Use of the Medium (Support tool) (EOUM) to develop an integrated approach and a Collaborative Modeling Process Quality Assessment (CMPQ) construct for assessing the quality of the collaborative modeling process. Secondly, we wanted to measure the (inter)dependencies (causation, correlations, etc.) of the modeling artifacts on each other and their e ect and impact on the overall quality of the modeling process. To this e ect, we apply Exploratory Factor Analysis (EFA) and Con rmatory Factor Analysis (CFA) techniques on a conceptual model (Model 1) and a competing model (Model 2). 3

Major Findings and Conclusion The rst observation about the results of the con rmatory analysis (CFA is that the (standardized) factor loadings of the the conceptual model (Model 1) and the competing model (Model 2) are close. In fact they are the same for the PQML and PUMP constructs while slight di erences are noticed for the PQEP and EOUM. This closeness of the results indicates that the Model used in the EFA was a good conceptual model. To determine the possibility of Model 2 being preferred to Model 1, we compare the model t indices of both models to determine which ones are near or better than the threshold values (see [ 3 ] for these threshold values). Through the two known statistical techniques: Exploratory Factor Analysis (EFA) and Con rmatory Factor Analysis (CFA) we can con rm that the approach is sound and the research instrument passes the validity and reliability tests. The contribution of this paper is thus two-fold. First, it develops a method of assessing collaborative modeling quality based on modeling artifacts used in, and developed during the collaborative modeling e ort. Second, a validated instrument for measuring the developed constructs and assessing the quality of the CMPQ construct is presented and was properly validated.

References

1. Davison , R.: An Instrument for Measuring Meeting Success: Revalidation and Modi cation . Information and Management 36 , 321 { 328 ( 1999 )

2. Briggs , R.O. , Reinig , B.A. , Vreede , G.J. de: Meeting Satisfaction for Technology Supported Groups: An Empirical Validation of a Goal-Attainment Model . Small Group Research 37, ( 2006 )

3. Straub , D. , Boudreau , M.- C. , Gefen , D. : Validation Guidelines for IS Positivist Research . Communication of the AIS 13 , 380 { 427 ( 2004 )