=Paper=
{{Paper
|id=Vol-479/paper-2
|storemode=property
|title=Evaluating Collaborative Modeling Processes:Towards Understanding and Supporting Collaborative Modeling Games
|pdfUrl=https://ceur-ws.org/Vol-479/paper2.pdf
|volume=Vol-479
}}
==Evaluating Collaborative Modeling Processes:Towards Understanding and Supporting Collaborative Modeling Games==
https://ceur-ws.org/Vol-479/paper2.pdf
Evaluating Collaborative Modeling Processes
Towards Understanding and Supporting Collaborative
Modeling Games
Denis Ssebuggwawo?
Institute of Computing and Information Sciences, Radboud University Nijmegen
Heyendaalseweg 135, 6525 AJ Nijmegen, The Netherlands, EU.
D.Ssebuggwawo@science.ru.nl
http://www.ru.nl/
Abstract. Collaborative modeling is an approach aimed at enhancing
productivity in Systems Design. Such an approach brings together stake-
holders with varying degrees of skills and knowledge. Although much
attention has been paid to the models created, little empirical work has
focused on the modeling process itself, especially its evaluation. This
raises the question whether an approach for analyzing and evaluating
modeling processes exists yet. We aim to analyze and evaluate this ne-
glected aspect. With the help of a three-tier framework, and by taking
a game design theoretical approach to modeling, we identify the differ-
ent aspects that drive the modeling process. We use this framework to
develop an understanding of the inner structure of the modeling process
with a view of evaluating it. We give some preliminary results to illus-
trate our framework and sketch an outline of future scientific inquiry to
refine and tighten this framework.
Key words: Collaborative Modeling, Modeling Process Evaluation, Mod-
eling Game, Game Design Theory
1 Introduction
1.1 Background, Context and Motivation
Collaborative or Group modeling [2,13] is a process that can enhance productiv-
ity in Information Systems Design and Business Process Re-engineering. Mod-
eling has been observed to contain not only the models (end-products) but also
the process that generates these models. Much attention has been paid to the
models (end-products) and their associated quality (see, for example, [7]), but
little attention has been paid to the process that generates these models. The
only work known to have made attempts to look at the process of modeling
within a communicative perspective is that of Hoppenbrouwers et al. [5], [10]
and Rittgen [11,12].
?
Supervisors: Prof. dr. H.A. (Erik) Proper (e.proper@acm.org) and Dr. S.J.B.A.
(Stijn) Hoppenbrouwers (stijnh@cs.ru.nl)
2 Proceedings of CAISE-DC 2009
1.2 Common Conceptual Modeling Assumptions
Some of the most popular conceptual modeling assumptions include the fol-
lowing: modeling is product-oriented and design centered, the modeling process
involves two roles: domain expert, and model builder (systems analyst) each
playing their different roles at different times of conceptual modeling. Quality
assessment and measurement is often restricted to only the end-products - the
models (see, for example, [7]). We contend and hypothesize that conceptual mod-
eling should be not only a product-oriented and design-centered approach but
should also be a conversational activity and should be human-centered. There
are also intermediary products that need to be analyzed and evaluated.
The process-oriented modeling approach assumes that the produced models
should contain “shared knowledge” and the modeling process is governed and
directed by a number of modeling rules and goals [15]. Modelers, however, do
not concisely and explicitly perform “step-wise” thinking, in particular for non-
experts, in a product-oriented approach to incorporate such shared-knowledge in
their models. Viewing modeling as consisting of the process and the products ,
helps us study the commitments, agreements, negotiations, decision making and
consensus, etc. of the modelers and the rules and goals governing this process.
1.3 The Research Problem, Questions and Objectives
One of the problems identified in collaborative/group modeling is lack of an
approach (and related tool-support) that can be used to study and improve
the communicative acts that lead to the generation of the models. Taking a
game-metaphorical approach to systems design (see, for example, [6]) has the
potential of helping us determine the rules/goals driving the modeling games. To
achieve this, we eventually aim to design collaborative modeling games(CMGs)
in which the modeling process plays a significant role and human interaction and
communication take center stage.
The following questions motivate our way of thinking in this research. The
main research question is: How can we evaluate enacted modeling processes (in
view of them achieving set goals)? To adequately answer this question, we raise
the following sub-questions: What is the quality of modeling? How can we mea-
sure modeling process quality?
Our long-term objectives is to analyze, evaluate and understand collaborative
modeling games (CMGs) with the longer-term aim of supporting conceptual
modeling with a tool. A more immediate objective is to evaluate currently avail-
able collaborative modeling sessions as if they are games to measure their effec-
tiveness and efficiency.
Figure 1 shows the design, execution, evaluation and validation cycle of the
CMGs.
2 Related Work
It being such a broad and multi-disciplinary area, it is hard to review all the liter-
ature related to Collaborative (Group) modeling within the constraints of paper’s
Proceedings of CAISE-DC 2009 3
Execute & Analyze the CMG
(Modeling Sessions using the RIM Framework)
(Discourse Analysis, Grounded Theory)
Design the CMG Improve the CMG Measure & Evaluate the CMG
(Design Science Approach) (COME Framework)
Current Knowledge
about modeling
Validate the CMG
Additions to (Expert/Practice Validation)
Knowledge base
Relevance & Applicability to Business Process
Modeling
Conceptual Modeling Knowledge Base
(Theories, Methods, Frameworks, etc.)
Fig. 1. Design cycle of collaborative modeling games(CMGs).
size. We therefore highlight only works directly related to our own. Bostrom et
al. [1] provide one of the earliest attempts to consider group facilitated meet-
ings using a Group Support Systems (GSS) tool. This work is important since
it shows how a GSS tool can be used to help stakeholders generate information,
organize it, evaluate and select alternatives and finally communicate their ac-
tions. Although communication is one of the aspects talked about and the role
of the facilitator is emphasized, communication is between the meeting partici-
pants and the facilitator. This is significantly different from our approach where
communication plays a central role in the negotiation between the participants
to reach agreement and a common shared understanding.
In [2] the authors draw on Electronic Meeting Systems (EMS) technology
and re-engineering techniques to develop a method and a support tool for mod-
eling business processes. This is a richer approach in user involvement and idea
generation than other traditionally known collaborative modeling approaches
and tools. It, however, lacks the theoretical rigour and underpinning for process
modeling as it influences only the quality of the generated models. The approach
is thus product-quality oriented. The work in [5] and [10] was the first attempt
to critically analyze the role of communication in modeling and the modeling
process. Our current work extends this work in emphasizing communication in
the modeling process and trying to find out out how modelers generate their
models. It, however, differs from it in that the current work employs the gaming
approach to modeling to determine the rules and goals under which modeling
processes take place.
The research work in this paper builds more on the work of Rittgen [11,12].
Rittgen observes that in a collaborative environment, participants engage in
different types of conversations prior to the creation of an accepted model. Our
work, however, differs from Rittgen’s in that we take a more holistic approach
that looks at modeling as a game-design theoretic approach.
4 Proceedings of CAISE-DC 2009
3 Conceptual Framework and Methodological Approach
In this section we present the basic conceptual framework to help us analyze the
process of modeling. The developed framework is related to two previously de-
veloped frameworks: The Semiotic Quality (SEQUAL) framework of Krogstie et
al. [7] and The Quality of Modeling framework QoMo of van Bommel et al. [15].
The basic concepts of our “RIM” (Rules, Interactions and Models) framework
are shown in Fig. 2.
Interactions lead to production of models, and
generated (intermediate) models drive further
interaction.
Interactions Models
Log <>
<>
A range of interactions over a period of time Some rules/goals of modeling apply to
changes the rules of play and interactions are intermediary and end‐products and these
guided and restricted by rules of play. Rules products may lead to new rules/goals.
<>
Fig. 2. Basic concepts for integrated analysis of interactions, rules and models.
The RIM framework is a three-tier framework that examines the communica-
tive acts (interactions) in a modeling session, the rules/goals set, and the models
produced as a result of the interaction and collaboration which is, metaphorically
speaking, a sort of modeling game [6]. The different players work under a set of
rules and goals. The rules/goals, interactions and models are all time-stamped
to help us track and identify he interplay between any pair.
In addition to the framework above we use the collaborative evaluation
(COME) framework given in Fig. 3 to evaluate the modeling process games
(CMGs) using a number of artifacts to be evaluated in view of the CMGs. Em-
ploying the design science approach [4], we put these artifacts to use within the
context of evaluation and improvement of the CMGs.
Proceedings of CAISE-DC 2009 5
Validate Evaluation Approach
Evaluation Artifact Evaluation Activities & Criteria Evaluation
Approaches
MODELING PROCESS Discursive
Evaluation
Generate
Cause & Effect
(Rated/Weighted)evaluation:
Modeling Language diagrams
Criteria (guidelines, metrics,
Evaluation Approach
(Constructs) Ontological
benchmarks)
Analysis
Modeling
Procedures Lab, action &
(Methods) interpretive
Criteria
research
Products
Re‐generate
Evaluate by :
Field, surveys &
(Models) Criteria (guidelines,
case studies
metrics, benchmarks)
Support Tool
(Instantiations) Participant/
Use Evaluation Criteria in
re‐assess Expert‐based
re‐assess
Apply Evaluation
evaluation
Determine
Features
Analysis,
Validate:
Verification, etc.
Criteria (guidelines,
metrics, benchmarks)
Artifact Information
Domain
Evaluate Artifact
Fig. 3. Modeling process evaluation approach.
4 Preliminary Results and Data Analysis
In this section, we present an analysis of the results obtained from a collaborative
modeling session in a pilot study. In this first phase of our reserved project, the
emphasis was on making the interactions and a few goals and rules using only
the RIM framework.
4.1 Experimental Setup
The business process scenario given out to modelers, was about developing a Haz-
ardous Material Management System (HMMS) by the Materials Management
Department (MMD) of a city council. Two researchers and three modelers(two
systems analysts (SA) and one domain expert (DE)) participated in the actual
modeling. Figure 4 shows one of the screen-shots from the modeling session
6 Proceedings of CAISE-DC 2009
video recording. The session (which took 18 minutes) was video recorded with
good sound quality. The modelers were also given a digital writing pad, which
was recorded alongside the video. This provided us with a full, synchronized
recording of all raw data we could wish for.
Fig. 4. Screenshot of a collaborative modeling session
4.2 Results and Analysis
We first transcribed the video recordings, then made an annotation and cate-
gorization of the speech acts. We mainly drew on Language-Action Perspective
(LAP) theory, Speech-Act Theory (SAT), Discourse Analysis and the Commu-
nicative Action Theory (CAT), see for example, [3]. Table 1 shows a sample
categorization of the speech acts.
Table 1. Categorization of conversational speech acts
Time Actor Speech Act Category
02:00 SA1 So, where does ordering start? question
02:03 SA2 First, we have to decide who takes part in it. So we proposition
can set that on top of the diagram?
02:10 SA1 There are numbers, so that’s easy, so probably pur- answer
chasing officer is involved?
02:18 SA2 Eh.. I guess so (laughs) agreement
with
03:54 SA2 No, no. It is number..(laughs)...number six, not five. argument
against
11:52 SA2 Yeah. Yeah, OK, but I call it a way of signing... withdrawal
Proceedings of CAISE-DC 2009 7
5 Findings and Discussion
In this section we give some of our observations from the data obtained from
the modeling session. The findings are given within the framework and method-
ological approach in Fig. 2.
• Setting the Agenda. It is noted that modelers, without the help of a facili-
tator, set their own agenda by structuring the modeling process in the following
phases: (I) - Setting the main approach:choosing the language and subdivision of
work (II) - Exploring and deciding which actors take part in the modeling process
and (III) - Modeling the sub-processes.
• Categorization of the Conversations. It was noted from the video and
the transcription that the communication among the modelers can broadly be
categorized as a negotiation. This was the same conclusion reached in [11]. This
comes from the argumentations (argue for/against) resulting in either accep-
tance,i.e.agreement (support) or rejection of the proposals. Rejection indicates
disagreement. More details are found in [14].
• Categorization of Modeling Rules and Goals. From the transcription
and observations it was noted that the rules and goals guiding the modeling
process could be categorized as: imposed in the scenario or created within the
modeling game. These rules and goals were further categorized as explicit-directly
set and stated or implicit - indirectly stated and set. In [14] these rules and goals
are explained in more detail. They include: rules that were set for the game are:
Goal setting rule: creation goal, Goal setting rule: validation goal and rules that
were set in the game are: Goal setting rule: grammar goal, Goal setting rule:
creation goal and Goal setting rule: grammar goal.
6 Conclusions and Further Research
This paper has looked at a research program aimed at shedding light on the
process (act) of modeling. We employ an interactive and collaborative modeling
approach, within the context of communicative modeling of business processes,
to one modeling case. We have developed a three-tier conceptual framework and
a methodological approach which can be used to analyze and understand the
communicative process of modeling. Three key concepts: interactions, rules/goals
and the modeling products have been identified.
• Contribution and Direction for Further Research. Our contribution in
this research is a framework that can be used to analyze modeling games and
an evaluation mechanism to measure the effectiveness and efficiency of these
modeling games. We intend to focus on developing an evaluation mechanism and
its requirements using the framework in Fig. 3 and to develop a methodology to
enable us draw scientifically sound and definitive conclusions about collaborative
modeling processes.
8 Proceedings of CAISE-DC 2009
References
1. Bostrom, R.P., Clawson, V.K.,Anson, R.:Group Facilitation and Group Support
Systems. In Jessup, L.M and Valacinch, J.S.M(eds.),Group Support systems: New
Perspectives, New York Macmillan, pp. 146–168, (1993).
2. Dean, D., Orwig, R., Lee, J., Vogel, D.: Modelling with a Group Modelling Tool:
Group Support, Model Quality and Validation. In System Sciences 1994: Collabora-
tion Technology Organizational Systems and Technology: Proceedings of the Twenty-
Seventh Annual HICCS conference, volume 4, pages 214223. IEEE Computer Society
Press, Los Almotos CA, USA, January (1994).
3. Goldkuhl, G.: Conversational Analysis as a Theoretical Foundation for Language
Action Approaches? In H. Weigand, G. Goldkuhl, and A. de Moor, editors, Proceed-
ings of the 8th International Working LAP2003 Conference, Tilburg, The Nether-
lands. Springer Verlag Berlin Heidelberg.
4. Hevner, A.R., March, S.T., Park, J., Ram, S.: Design Science in Information Systems
Research. MIS Quarterly 28(1): pp. 75-105, (2004).
5. Hoppenbrouwers, S.J.B.A., Proper, H.A., van Reijswoud, V.E.: Navigating the
Methodology Jungle - the Communicative role of Modeling Techniques in Information
System Development. Computing Letters, 1(3): pp. 169, (2005).
6. Hoppenbrouwers, S.J.B.A.: Community-based ICT Development as a Multi-player
Game. In Proceedings of the Conference : What is an Organization? Materiality,
Agency and Discourse, University of Montreal, Canada, May (2008).
7. Krogstie, J., Sindre, G., Jorgensen, H..: Process Models Representing Knowledge
Action : A Revised Quality Framewok. European Journal of Information Systems,
15: 91-102, (2006).
8. March, S.T., Smith G.F.: Design and Natural Science Research on Information
Technology. Decision Science Support Systems, 15(4): pp. 251-266, (1995).
9. Pleiffer, D., Niehaves, B.: Evaluation of Conceptual Models - A Structuralist Ap-
proach. Proceedings of the 13th ECIS: Information Systems in a Rapidly Changing
Economy, ECIS 2005, Regensburg, Germany, May 26-28, (2005).
10. Proper, H.A., Hoppenbrouwers, S.J.B.A., van Bommel, P.: A Fundamental View
on the Act of Modeling. In J. Kizza, J. Aisbett, A. Vince, and T. Wanyama, editors,
Advances in Systems Modelling and ICT Applications, volume 2 of Special topics in
computing and ICT research. Fountain Publishers, Kampala, Uganda, August (2006).
11. Rittgen, P.: Negotiating Models. In J. Krogstie, A. L. Opdahl, and G. Sindre,
editors, CAiSE 2007, volume 4495 of LNCS, pages 561573. Springer-Verlag Berlin
Heidelberg, 2007.
12. Rittgen, P.: Collaborative Modelling Architecture (COMA). http://www.coma.nu/
COMA_Tool.pdf. [Accessed on: 08/02/2009].
13. Rouwette, E.A.J.A., Vennix J., Van Mullekom, T.:Group Model Building Effective-
ness. A Review of assessment Studies. Systems Dynamics Review, 18(1), pp. 5–45,
(2002).
14. Ssebuggwawo, D., Hoppenbrouwers, S.J.B.A., Proper, H.A.: Analysizing a Collab-
orative Modeling Game. To appear in the Proceedings of the CAiSE Forum 2009,
Amsterdam, 8–12 June, (2009).
15. van Bommel, P., Hoppenbrouwers, S.J.B.A., Proper, H.A.: QoMo: A Modeling
Process Quality Framework Based on Sequal. In H. A. Proper, T. Halpin, and J.
Krogstie, editors, Proceedings of the Workshop EMMSAD07, held in conjunctiun
with the 19th CAiSE07 conference, pp. 118-127. Tapir Academic Press, Trondheim,
Norway, (2007).