Evaluating Collaborative Modeling Processes Towards Understanding and Supporting Collaborative Modeling Games Denis Ssebuggwawo? Institute of Computing and Information Sciences, Radboud University Nijmegen Heyendaalseweg 135, 6525 AJ Nijmegen, The Netherlands, EU. D.Ssebuggwawo@science.ru.nl http://www.ru.nl/ Abstract. Collaborative modeling is an approach aimed at enhancing productivity in Systems Design. Such an approach brings together stake- holders with varying degrees of skills and knowledge. Although much attention has been paid to the models created, little empirical work has focused on the modeling process itself, especially its evaluation. This raises the question whether an approach for analyzing and evaluating modeling processes exists yet. We aim to analyze and evaluate this ne- glected aspect. With the help of a three-tier framework, and by taking a game design theoretical approach to modeling, we identify the differ- ent aspects that drive the modeling process. We use this framework to develop an understanding of the inner structure of the modeling process with a view of evaluating it. We give some preliminary results to illus- trate our framework and sketch an outline of future scientific inquiry to refine and tighten this framework. Key words: Collaborative Modeling, Modeling Process Evaluation, Mod- eling Game, Game Design Theory 1 Introduction 1.1 Background, Context and Motivation Collaborative or Group modeling [2,13] is a process that can enhance productiv- ity in Information Systems Design and Business Process Re-engineering. Mod- eling has been observed to contain not only the models (end-products) but also the process that generates these models. Much attention has been paid to the models (end-products) and their associated quality (see, for example, [7]), but little attention has been paid to the process that generates these models. The only work known to have made attempts to look at the process of modeling within a communicative perspective is that of Hoppenbrouwers et al. [5], [10] and Rittgen [11,12]. ? Supervisors: Prof. dr. H.A. (Erik) Proper (e.proper@acm.org) and Dr. S.J.B.A. (Stijn) Hoppenbrouwers (stijnh@cs.ru.nl) 2 Proceedings of CAISE-DC 2009 1.2 Common Conceptual Modeling Assumptions Some of the most popular conceptual modeling assumptions include the fol- lowing: modeling is product-oriented and design centered, the modeling process involves two roles: domain expert, and model builder (systems analyst) each playing their different roles at different times of conceptual modeling. Quality assessment and measurement is often restricted to only the end-products - the models (see, for example, [7]). We contend and hypothesize that conceptual mod- eling should be not only a product-oriented and design-centered approach but should also be a conversational activity and should be human-centered. There are also intermediary products that need to be analyzed and evaluated. The process-oriented modeling approach assumes that the produced models should contain “shared knowledge” and the modeling process is governed and directed by a number of modeling rules and goals [15]. Modelers, however, do not concisely and explicitly perform “step-wise” thinking, in particular for non- experts, in a product-oriented approach to incorporate such shared-knowledge in their models. Viewing modeling as consisting of the process and the products , helps us study the commitments, agreements, negotiations, decision making and consensus, etc. of the modelers and the rules and goals governing this process. 1.3 The Research Problem, Questions and Objectives One of the problems identified in collaborative/group modeling is lack of an approach (and related tool-support) that can be used to study and improve the communicative acts that lead to the generation of the models. Taking a game-metaphorical approach to systems design (see, for example, [6]) has the potential of helping us determine the rules/goals driving the modeling games. To achieve this, we eventually aim to design collaborative modeling games(CMGs) in which the modeling process plays a significant role and human interaction and communication take center stage. The following questions motivate our way of thinking in this research. The main research question is: How can we evaluate enacted modeling processes (in view of them achieving set goals)? To adequately answer this question, we raise the following sub-questions: What is the quality of modeling? How can we mea- sure modeling process quality? Our long-term objectives is to analyze, evaluate and understand collaborative modeling games (CMGs) with the longer-term aim of supporting conceptual modeling with a tool. A more immediate objective is to evaluate currently avail- able collaborative modeling sessions as if they are games to measure their effec- tiveness and efficiency. Figure 1 shows the design, execution, evaluation and validation cycle of the CMGs. 2 Related Work It being such a broad and multi-disciplinary area, it is hard to review all the liter- ature related to Collaborative (Group) modeling within the constraints of paper’s Proceedings of CAISE-DC 2009 3 Execute & Analyze the CMG (Modeling Sessions using the RIM Framework) (Discourse Analysis, Grounded Theory) Design the CMG Improve the CMG Measure & Evaluate the CMG (Design Science Approach) (COME Framework) Current Knowledge about modeling Validate the CMG Additions to (Expert/Practice Validation) Knowledge base Relevance & Applicability to Business Process Modeling Conceptual Modeling Knowledge Base (Theories, Methods, Frameworks, etc.) Fig. 1. Design cycle of collaborative modeling games(CMGs). size. We therefore highlight only works directly related to our own. Bostrom et al. [1] provide one of the earliest attempts to consider group facilitated meet- ings using a Group Support Systems (GSS) tool. This work is important since it shows how a GSS tool can be used to help stakeholders generate information, organize it, evaluate and select alternatives and finally communicate their ac- tions. Although communication is one of the aspects talked about and the role of the facilitator is emphasized, communication is between the meeting partici- pants and the facilitator. This is significantly different from our approach where communication plays a central role in the negotiation between the participants to reach agreement and a common shared understanding. In [2] the authors draw on Electronic Meeting Systems (EMS) technology and re-engineering techniques to develop a method and a support tool for mod- eling business processes. This is a richer approach in user involvement and idea generation than other traditionally known collaborative modeling approaches and tools. It, however, lacks the theoretical rigour and underpinning for process modeling as it influences only the quality of the generated models. The approach is thus product-quality oriented. The work in [5] and [10] was the first attempt to critically analyze the role of communication in modeling and the modeling process. Our current work extends this work in emphasizing communication in the modeling process and trying to find out out how modelers generate their models. It, however, differs from it in that the current work employs the gaming approach to modeling to determine the rules and goals under which modeling processes take place. The research work in this paper builds more on the work of Rittgen [11,12]. Rittgen observes that in a collaborative environment, participants engage in different types of conversations prior to the creation of an accepted model. Our work, however, differs from Rittgen’s in that we take a more holistic approach that looks at modeling as a game-design theoretic approach. 4 Proceedings of CAISE-DC 2009 3 Conceptual Framework and Methodological Approach In this section we present the basic conceptual framework to help us analyze the process of modeling. The developed framework is related to two previously de- veloped frameworks: The Semiotic Quality (SEQUAL) framework of Krogstie et al. [7] and The Quality of Modeling framework QoMo of van Bommel et al. [15]. The basic concepts of our “RIM” (Rules, Interactions and Models) framework are shown in Fig. 2. Interactions lead to production of models, and generated (intermediate) models drive further interaction. Interactions Models Log <> <> A range of interactions over a period of time Some rules/goals of modeling apply to changes the rules of play and interactions are intermediary and end‐products and these guided and restricted by rules of play. Rules products may lead to new rules/goals. <> Fig. 2. Basic concepts for integrated analysis of interactions, rules and models. The RIM framework is a three-tier framework that examines the communica- tive acts (interactions) in a modeling session, the rules/goals set, and the models produced as a result of the interaction and collaboration which is, metaphorically speaking, a sort of modeling game [6]. The different players work under a set of rules and goals. The rules/goals, interactions and models are all time-stamped to help us track and identify he interplay between any pair. In addition to the framework above we use the collaborative evaluation (COME) framework given in Fig. 3 to evaluate the modeling process games (CMGs) using a number of artifacts to be evaluated in view of the CMGs. Em- ploying the design science approach [4], we put these artifacts to use within the context of evaluation and improvement of the CMGs. Proceedings of CAISE-DC 2009 5 Validate Evaluation Approach Evaluation Artifact Evaluation Activities & Criteria Evaluation Approaches MODELING PROCESS Discursive Evaluation Generate Cause & Effect (Rated/Weighted)evaluation: Modeling Language diagrams Criteria (guidelines, metrics, Evaluation Approach (Constructs) Ontological benchmarks) Analysis Modeling Procedures Lab, action & (Methods) interpretive Criteria research Products Re‐generate Evaluate by : Field, surveys & (Models) Criteria (guidelines, case studies metrics, benchmarks) Support Tool (Instantiations) Participant/ Use Evaluation Criteria in re‐assess Expert‐based re‐assess Apply Evaluation evaluation Determine Features Analysis, Validate: Verification, etc. Criteria (guidelines, metrics, benchmarks) Artifact Information Domain Evaluate Artifact Fig. 3. Modeling process evaluation approach. 4 Preliminary Results and Data Analysis In this section, we present an analysis of the results obtained from a collaborative modeling session in a pilot study. In this first phase of our reserved project, the emphasis was on making the interactions and a few goals and rules using only the RIM framework. 4.1 Experimental Setup The business process scenario given out to modelers, was about developing a Haz- ardous Material Management System (HMMS) by the Materials Management Department (MMD) of a city council. Two researchers and three modelers(two systems analysts (SA) and one domain expert (DE)) participated in the actual modeling. Figure 4 shows one of the screen-shots from the modeling session 6 Proceedings of CAISE-DC 2009 video recording. The session (which took 18 minutes) was video recorded with good sound quality. The modelers were also given a digital writing pad, which was recorded alongside the video. This provided us with a full, synchronized recording of all raw data we could wish for. Fig. 4. Screenshot of a collaborative modeling session 4.2 Results and Analysis We first transcribed the video recordings, then made an annotation and cate- gorization of the speech acts. We mainly drew on Language-Action Perspective (LAP) theory, Speech-Act Theory (SAT), Discourse Analysis and the Commu- nicative Action Theory (CAT), see for example, [3]. Table 1 shows a sample categorization of the speech acts. Table 1. Categorization of conversational speech acts Time Actor Speech Act Category 02:00 SA1 So, where does ordering start? question 02:03 SA2 First, we have to decide who takes part in it. So we proposition can set that on top of the diagram? 02:10 SA1 There are numbers, so that’s easy, so probably pur- answer chasing officer is involved? 02:18 SA2 Eh.. I guess so (laughs) agreement with 03:54 SA2 No, no. It is number..(laughs)...number six, not five. argument against 11:52 SA2 Yeah. Yeah, OK, but I call it a way of signing... withdrawal Proceedings of CAISE-DC 2009 7 5 Findings and Discussion In this section we give some of our observations from the data obtained from the modeling session. The findings are given within the framework and method- ological approach in Fig. 2. • Setting the Agenda. It is noted that modelers, without the help of a facili- tator, set their own agenda by structuring the modeling process in the following phases: (I) - Setting the main approach:choosing the language and subdivision of work (II) - Exploring and deciding which actors take part in the modeling process and (III) - Modeling the sub-processes. • Categorization of the Conversations. It was noted from the video and the transcription that the communication among the modelers can broadly be categorized as a negotiation. This was the same conclusion reached in [11]. This comes from the argumentations (argue for/against) resulting in either accep- tance,i.e.agreement (support) or rejection of the proposals. Rejection indicates disagreement. More details are found in [14]. • Categorization of Modeling Rules and Goals. From the transcription and observations it was noted that the rules and goals guiding the modeling process could be categorized as: imposed in the scenario or created within the modeling game. These rules and goals were further categorized as explicit-directly set and stated or implicit - indirectly stated and set. In [14] these rules and goals are explained in more detail. They include: rules that were set for the game are: Goal setting rule: creation goal, Goal setting rule: validation goal and rules that were set in the game are: Goal setting rule: grammar goal, Goal setting rule: creation goal and Goal setting rule: grammar goal. 6 Conclusions and Further Research This paper has looked at a research program aimed at shedding light on the process (act) of modeling. We employ an interactive and collaborative modeling approach, within the context of communicative modeling of business processes, to one modeling case. We have developed a three-tier conceptual framework and a methodological approach which can be used to analyze and understand the communicative process of modeling. Three key concepts: interactions, rules/goals and the modeling products have been identified. • Contribution and Direction for Further Research. Our contribution in this research is a framework that can be used to analyze modeling games and an evaluation mechanism to measure the effectiveness and efficiency of these modeling games. We intend to focus on developing an evaluation mechanism and its requirements using the framework in Fig. 3 and to develop a methodology to enable us draw scientifically sound and definitive conclusions about collaborative modeling processes. 8 Proceedings of CAISE-DC 2009 References 1. Bostrom, R.P., Clawson, V.K.,Anson, R.:Group Facilitation and Group Support Systems. In Jessup, L.M and Valacinch, J.S.M(eds.),Group Support systems: New Perspectives, New York Macmillan, pp. 146–168, (1993). 2. Dean, D., Orwig, R., Lee, J., Vogel, D.: Modelling with a Group Modelling Tool: Group Support, Model Quality and Validation. In System Sciences 1994: Collabora- tion Technology Organizational Systems and Technology: Proceedings of the Twenty- Seventh Annual HICCS conference, volume 4, pages 214223. IEEE Computer Society Press, Los Almotos CA, USA, January (1994). 3. Goldkuhl, G.: Conversational Analysis as a Theoretical Foundation for Language Action Approaches? In H. Weigand, G. Goldkuhl, and A. de Moor, editors, Proceed- ings of the 8th International Working LAP2003 Conference, Tilburg, The Nether- lands. Springer Verlag Berlin Heidelberg. 4. Hevner, A.R., March, S.T., Park, J., Ram, S.: Design Science in Information Systems Research. MIS Quarterly 28(1): pp. 75-105, (2004). 5. Hoppenbrouwers, S.J.B.A., Proper, H.A., van Reijswoud, V.E.: Navigating the Methodology Jungle - the Communicative role of Modeling Techniques in Information System Development. Computing Letters, 1(3): pp. 169, (2005). 6. Hoppenbrouwers, S.J.B.A.: Community-based ICT Development as a Multi-player Game. In Proceedings of the Conference : What is an Organization? Materiality, Agency and Discourse, University of Montreal, Canada, May (2008). 7. Krogstie, J., Sindre, G., Jorgensen, H..: Process Models Representing Knowledge Action : A Revised Quality Framewok. European Journal of Information Systems, 15: 91-102, (2006). 8. March, S.T., Smith G.F.: Design and Natural Science Research on Information Technology. Decision Science Support Systems, 15(4): pp. 251-266, (1995). 9. Pleiffer, D., Niehaves, B.: Evaluation of Conceptual Models - A Structuralist Ap- proach. Proceedings of the 13th ECIS: Information Systems in a Rapidly Changing Economy, ECIS 2005, Regensburg, Germany, May 26-28, (2005). 10. Proper, H.A., Hoppenbrouwers, S.J.B.A., van Bommel, P.: A Fundamental View on the Act of Modeling. In J. Kizza, J. Aisbett, A. Vince, and T. Wanyama, editors, Advances in Systems Modelling and ICT Applications, volume 2 of Special topics in computing and ICT research. Fountain Publishers, Kampala, Uganda, August (2006). 11. Rittgen, P.: Negotiating Models. In J. Krogstie, A. L. Opdahl, and G. Sindre, editors, CAiSE 2007, volume 4495 of LNCS, pages 561573. Springer-Verlag Berlin Heidelberg, 2007. 12. Rittgen, P.: Collaborative Modelling Architecture (COMA). http://www.coma.nu/ COMA_Tool.pdf. [Accessed on: 08/02/2009]. 13. Rouwette, E.A.J.A., Vennix J., Van Mullekom, T.:Group Model Building Effective- ness. A Review of assessment Studies. Systems Dynamics Review, 18(1), pp. 5–45, (2002). 14. Ssebuggwawo, D., Hoppenbrouwers, S.J.B.A., Proper, H.A.: Analysizing a Collab- orative Modeling Game. To appear in the Proceedings of the CAiSE Forum 2009, Amsterdam, 8–12 June, (2009). 15. van Bommel, P., Hoppenbrouwers, S.J.B.A., Proper, H.A.: QoMo: A Modeling Process Quality Framework Based on Sequal. In H. A. Proper, T. Halpin, and J. Krogstie, editors, Proceedings of the Workshop EMMSAD07, held in conjunctiun with the 19th CAiSE07 conference, pp. 118-127. Tapir Academic Press, Trondheim, Norway, (2007).