Evaluating Collaborative Modeling Processes

Abstract

Collaborative modeling is an approach aimed at enhancing productivity in Systems Design. Such an approach brings together stakeholders with varying degrees of skills and knowledge. Although much attention has been paid to the models created, little empirical work has focused on the modeling process itself, especially its evaluation. This raises the question whether an approach for analyzing and evaluating modeling processes exists yet. We aim to analyze and evaluate this neglected aspect. With the help of a three-tier framework, and by taking a game design theoretical approach to modeling, we identify the di erent aspects that drive the modeling process. We use this framework to develop an understanding of the inner structure of the modeling process with a view of evaluating it. We give some preliminary results to illustrate our framework and sketch an outline of future scienti c inquiry to re ne and tighten this framework.

Keywords

• Collaborative Modeling
• Modeling Process Evaluation
• Modeling Game
• Game Design Theory

-

1.1

Introduction

Common Conceptual Modeling Assumptions Some of the most popular conceptual modeling assumptions include the following: modeling is product-oriented and design centered, the modeling process involves two roles: domain expert, and model builder (systems analyst) each playing their di erent roles at di erent times of conceptual modeling. Quality assessment and measurement is often restricted to only the end-products - the models (see, for example, [ 7 ]). We contend and hypothesize that conceptual modeling should be not only a product-oriented and design-centered approach but should also be a conversational activity and should be human-centered. There are also intermediary products that need to be analyzed and evaluated.

The process-oriented modeling approach assumes that the produced models should contain \shared knowledge" and the modeling process is governed and directed by a number of modeling rules and goals [ 15 ]. Modelers, however, do not concisely and explicitly perform \step-wise" thinking, in particular for nonexperts, in a product-oriented approach to incorporate such shared-knowledge in their models. Viewing modeling as consisting of the process and the products , helps us study the commitments, agreements, negotiations, decision making and consensus, etc. of the modelers and the rules and goals governing this process. 1.3

The Research Problem, Questions and Objectives One of the problems identi ed in collaborative/group modeling is lack of an approach (and related tool-support) that can be used to study and improve the communicative acts that lead to the generation of the models. Taking a game-metaphorical approach to systems design (see, for example, [ 6 ]) has the potential of helping us determine the rules/goals driving the modeling games. To achieve this, we eventually aim to design collaborative modeling games(CMGs) in which the modeling process plays a signi cant role and human interaction and communication take center stage.

The following questions motivate our way of thinking in this research. The main research question is: How can we evaluate enacted modeling processes (in view of them achieving set goals)? To adequately answer this question, we raise the following sub-questions: What is the quality of modeling? How can we measure modeling process quality? Our long-term objectives is to analyze, evaluate and understand collaborative modeling games (CMGs) with the longer-term aim of supporting conceptual modeling with a tool. A more immediate objective is to evaluate currently available collaborative modeling sessions as if they are games to measure their e ectiveness and e ciency.

Figure 1 shows the design, execution, evaluation and validation cycle of the CMGs. 2

Related Work

It being such a broad and multi-disciplinary area, it is hard to review all the literature related to Collaborative (Group) modeling within the constraints of paper's

Design the CMG (Design Science Approach) size. We therefore highlight only works directly related to our own. Bostrom et al. [ 1 ] provide one of the earliest attempts to consider group facilitated meetings using a Group Support Systems (GSS) tool. This work is important since it shows how a GSS tool can be used to help stakeholders generate information, organize it, evaluate and select alternatives and nally communicate their actions. Although communication is one of the aspects talked about and the role of the facilitator is emphasized, communication is between the meeting participants and the facilitator. This is signi cantly di erent from our approach where communication plays a central role in the negotiation between the participants to reach agreement and a common shared understanding.

In [ 2 ] the authors draw on Electronic Meeting Systems (EMS) technology and re-engineering techniques to develop a method and a support tool for modeling business processes. This is a richer approach in user involvement and idea generation than other traditionally known collaborative modeling approaches and tools. It, however, lacks the theoretical rigour and underpinning for process modeling as it in uences only the quality of the generated models. The approach is thus product-quality oriented. The work in [ 5 ] and [ 10 ] was the rst attempt to critically analyze the role of communication in modeling and the modeling process. Our current work extends this work in emphasizing communication in the modeling process and trying to nd out out how modelers generate their models. It, however, di ers from it in that the current work employs the gaming approach to modeling to determine the rules and goals under which modeling processes take place.

The research work in this paper builds more on the work of Rittgen [ 11,12 ]. Rittgen observes that in a collaborative environment, participants engage in di erent types of conversations prior to the creation of an accepted model. Our work, however, di ers from Rittgen's in that we take a more holistic approach that looks at modeling as a game-design theoretic approach.

Conceptual Framework and Methodological Approach

In this section we present the basic conceptual framework to help us analyze the process of modeling. The developed framework is related to two previously developed frameworks: The Semiotic Quality (SEQUAL) framework of Krogstie et al. [ 7 ] and The Quality of Modeling framework QoMo of van Bommel et al. [ 15 ]. The basic concepts of our \RIM" (Rules, Interactions and Models) framework are shown in Fig. 2.

 

Interactions lead to production of models, and  generated (intermediate) models drive further  interaction.  Interactions 

Log  <<time, t>> 

 

Models  <<time, t>> 

A range of interactions over a period of time 

changes the rules of play and interactions are  guided and restricted by rules of play.   

Rules  <<time, t>> 

Some  rules/goals  of  modeling  apply  to intermediary and end‐products and these  products may lead to new rules/goals. 

The RIM framework is a three-tier framework that examines the communicative acts (interactions) in a modeling session, the rules/goals set, and the models produced as a result of the interaction and collaboration which is, metaphorically speaking, a sort of modeling game [ 6 ]. The di erent players work under a set of rules and goals. The rules/goals, interactions and models are all time-stamped to help us track and identify he interplay between any pair.

In addition to the framework above we use the collaborative evaluation (COME) framework given in Fig. 3 to evaluate the modeling process games (CMGs) using a number of artifacts to be evaluated in view of the CMGs. Employing the design science approach [ 4 ], we put these artifacts to use within the context of evaluation and improvement of the CMGs.

Validate Evaluation Approach

  Evaluation Activities & Criteria  Generate   (Rated/Weighted)evaluation:   Criteria (guidelines, metrics,  benchmarks) 

Evaluate by : 

Criteria (guidelines,  metrics, benchmarks)      s s e s ‐rs a e re‐assess

Validate: 

Criteria (guidelines,  metrics, benchmarks)  Evaluate Artifact   e t a r e n e g ‐ e R    h c a o r p p A   n o i t a u l a v E     ii n a r e t i r C    n o i t a u l a v E   e s U    

Evaluation 

Approaches 

Discursive 

Evaluation

Cause & Effect  diagrams

Ontological 

Analysis 

Lab, action &  interpretive  research 

 

Field, surveys &  case studies 

 

Participant/ 

Expert‐based 

evaluation   

Analysis, 

Verification,  etc. 

  In this section, we present an analysis of the results obtained from a collaborative modeling session in a pilot study. In this rst phase of our reserved project, the emphasis was on making the interactions and a few goals and rules using only the RIM framework. The business process scenario given out to modelers, was about developing a Hazardous Material Management System (HMMS) by the Materials Management Department (MMD) of a city council. Two researchers and three modelers(two systems analysts (SA) and one domain expert (DE)) participated in the actual modeling. Figure 4 shows one of the screen-shots from the modeling session video recording. The session (which took 18 minutes) was video recorded with good sound quality. The modelers were also given a digital writing pad, which was recorded alongside the video. This provided us with a full, synchronized recording of all raw data we could wish for.  

Fig. 4. Screenshot of a collaborative modeling session 4.2

Results and Analysis We rst transcribed the video recordings, then made an annotation and categorization of the speech acts. We mainly drew on Language-Action Perspective (LAP) theory, Speech-Act Theory (SAT), Discourse Analysis and the Communicative Action Theory (CAT), see for example, [ 3 ]. Table 1 shows a sample categorization of the speech acts.

Findings and Discussion

In this section we give some of our observations from the data obtained from the modeling session. The ndings are given within the framework and methodological approach in Fig. 2.

Setting the Agenda. It is noted that modelers, without the help of a facilitator, set their own agenda by structuring the modeling process in the following phases: (I) - Setting the main approach:choosing the language and subdivision of work (II) - Exploring and deciding which actors take part in the modeling process and (III) - Modeling the sub-processes.

Categorization of the Conversations. It was noted from the video and the transcription that the communication among the modelers can broadly be categorized as a negotiation. This was the same conclusion reached in [ 11 ]. This comes from the argumentations (argue for/against) resulting in either acceptance,i.e.agreement (support) or rejection of the proposals. Rejection indicates disagreement. More details are found in [ 14 ].

Categorization of Modeling Rules and Goals. From the transcription and observations it was noted that the rules and goals guiding the modeling process could be categorized as: imposed in the scenario or created within the modeling game. These rules and goals were further categorized as explicit -directly set and stated or implicit - indirectly stated and set. In [ 14 ] these rules and goals are explained in more detail. They include: rules that were set for the game are: Goal setting rule: creation goal, Goal setting rule: validation goal and rules that were set in the game are: Goal setting rule: grammar goal, Goal setting rule: creation goal and Goal setting rule: grammar goal. 6

Conclusions and Further Research

This paper has looked at a research program aimed at shedding light on the process (act) of modeling. We employ an interactive and collaborative modeling approach, within the context of communicative modeling of business processes, to one modeling case. We have developed a three-tier conceptual framework and a methodological approach which can be used to analyze and understand the communicative process of modeling. Three key concepts: interactions, rules/goals and the modeling products have been identi ed.

Contribution and Direction for Further Research. Our contribution in this research is a framework that can be used to analyze modeling games and an evaluation mechanism to measure the e ectiveness and e ciency of these modeling games. We intend to focus on developing an evaluation mechanism and its requirements using the framework in Fig. 3 and to develop a methodology to enable us draw scienti cally sound and de nitive conclusions about collaborative modeling processes.

References

1. Bostrom , R.P. , Clawson , V.K. , Anson , R.:Group Facilitation and Group Support Systems . In Jessup, L. M and Valacinch , J.S. M(eds .),Group Support systems: New Perspectives, New York Macmillan, pp. 146 { 168 , ( 1993 ).

2. Dean , D. , Orwig , R. , Lee , J. , Vogel , D. : Modelling with a Group Modelling Tool: Group Support, Model Quality and Validation . In System Sciences 1994: Collaboration Technology Organizational Systems and Technology: Proceedings of the TwentySeventh Annual HICCS conference , volume 4 , pages 214223 . IEEE Computer Society Press, Los Almotos CA, USA, January ( 1994 ).

3. Goldkuhl , G.: Conversational Analysis as a Theoretical Foundation for Language Action Approaches ? In H. Weigand, G. Goldkuhl, and A. de Moor, editors, Proceedings of the 8th International Working LAP2003 Conference , Tilburg, The Netherlands. Springer Verlag Berlin Heidelberg.

4. Hevner , A.R. , March , S.T., Park , J. , Ram , S. : Design Science in Information Systems Research. MIS Quarterly 28 ( 1 ): pp. 75 - 105 , ( 2004 ).

5. Hoppenbrouwers , S.J.B.A. , Proper , H.A., van Reijswoud , V.E. : Navigating the Methodology Jungle - the Communicative role of Modeling Techniques in Information System Development . Computing Letters , 1 ( 3 ): pp. 169 , ( 2005 ).

6. Hoppenbrouwers , S.J.B.A. : Community-based ICT Development as a Multi-player Game . In Proceedings of the Conference : What is an Organization? Materiality, Agency and Discourse , University of Montreal, Canada, May ( 2008 ).

7. Krogstie , J. , Sindre , G. , Jorgensen , H..: Process Models Representing Knowledge Action : A Revised Quality Framewok . European Journal of Information Systems , 15 : 91 - 102 , ( 2006 ).

8. March , S.T., Smith G.F. : Design and Natural Science Research on Information Technology. Decision Science Support Systems , 15 ( 4 ): pp. 251 - 266 , ( 1995 ).

9. Plei er, D., Niehaves , B. : Evaluation of Conceptual Models - A Structuralist Approach . Proceedings of the 13th ECIS: Information Systems in a Rapidly Changing Economy, ECIS 2005 , Regensburg, Germany, May 26-28, ( 2005 ).

10. Proper , H.A. , Hoppenbrouwers , S.J.B.A. , van Bommel, P. : A Fundamental View on the Act of Modeling . In J. Kizza, J. Aisbett , A. Vince , and T. Wanyama, editors, Advances in Systems Modelling and ICT Applications , volume 2 of Special topics in computing and ICT research . Fountain Publishers, Kampala, Uganda, August ( 2006 ).

11. Rittgen , P. : Negotiating Models . In J. Krogstie , A. L. Opdahl , and G. Sindre, editors, CAiSE 2007 , volume 4495 of LNCS , pages 561573 . Springer-Verlag Berlin Heidelberg, 2007 .

12. Rittgen , P. : Collaborative Modelling Architecture (COMA) . http://www.coma.nu/ COMA_Tool.pdf. [Accessed on: 08 /02/2009].

13. Rouwette , E.A.J.A. , Vennix J ., Van Mullekom, T. : Group Model Building E ectiveness. A Review of assessment Studies . Systems Dynamics Review , 18 ( 1 ), pp. 5 { 45 , ( 2002 ).

14. Ssebuggwawo , D. , Hoppenbrouwers , S.J.B.A. , Proper , H.A. : Analysizing a Collaborative Modeling Game . To appear in the Proceedings of the CAiSE Forum 2009 , Amsterdam, 8 {12 June, ( 2009 ).

15. van Bommel, P. , Hoppenbrouwers , S.J.B.A. , Proper , H.A. : QoMo: A Modeling Process Quality Framework Based on Sequal . In H. A. Proper , T. Halpin , and J. Krogstie, editors, Proceedings of the Workshop EMMSAD07 , held in conjunctiun with the 19th CAiSE07 conference , pp. 118 - 127 . Tapir Academic Press, Trondheim, Norway, ( 2007 ).