Collaborative Modeling: Towards a Meta-model for Analysis and Evaluation ?

Abstract

In this paper we discuss a meta-model for the analysis and evaluation of collaborative modeling sessions. In the rst part of the meta-model, we use an analysis framework which reveals a triad of rules, interactions and models. This framework, which is central in driving the modeling process, helps us look inside the modeling process with the aim of understanding it better. The second part of the meta-model is based on an evaluation framework using a multi-criteria decision analysis (MCDA) method. Central to this framework, is how modelers' quality priorities and preferences can, through a group decision-making and negotiation process, be traced back to the interactions and rules in the analysis framework.

Keywords

• Collaborative Modeling
• Modeling Process Quality
• Modeling Process Analysis
• Modeling Process Evaluation
• Group Support Tools

Introduction

A number of studies have, over the years, looked at collaborative modeling [ 1,2,3 ]. There have also been attempts to understand the modeling process [ 4,5 ]. Such modeling is driven by participants' communication. Human communication [ 6 ], in collaborative modeling, involves argumentation, negotiation and decision making. Often, participants need to agree, through negotiation and decision making, on what constitutes, for example, \quality" for the di erent modeling artifacts and how such quality should be assessed. However, how to assess the quality of the collaborative modeling process, especially with respect to the modeling artifacts, remains a largely unexplored area.

The current paper tries to develop a meta-model which can be used for both the analysis and evaluation of a collaborative modeling process and the relation between events in the process and the resulting artifacts. The meta-model links the modeling artifacts and the evaluation framework to the rules, interactions and models (RIM) framework [ 7 ] through the interactions which are governed by rules. The interactions, rules and models are a result of the communicative process, mainly through modelers'negotiation. Negotiation plays a key role in collaborative modeling. It is through negotiation that modelers reach agreement and possibly consensus. In this paper we limit our discussion to negotiation dialogues from argumentation theory.

Negotiation dialogue has been widely studied, see for example [ 8,9,10 ]. Its practical applications include multi-agent systems (MAS) [ 11,12,13,14 ] with wide applications in electronic commerce [ 15,16,17 ]. Negotiation dialogues start from a position of con ict and the goal is to establish some consensus or compromise for all the parties involved. Usually, participants have con icting objectives, interests, preference and priorities. Through the process of negotiation, they get a compromise position that everyone is comfortable with. This is what happens in a multi-actor (collaborative and interactive) modeling process. Modelers have con icting views, priorities and preferences and they engage in an argumentation process, that involves, propositions, (dis)agreements, acceptances and rejections, supports and withdraws, etc, to reach a compromise.

It should be noted that, although there are a number of factors that one may be interested in looking at in the analysis and evaluation of the modeling process, which in fact may in uence the quality of the modeling process, e.g., power struggle, leadership and the unspoken message or body language, etc., (see for example, [ 18 ]), our interest at the moment is in what we call \drivers" of the modeling process. Rules and/or goals, interactions, and models are hypothesized to be drivers of the modeling process. In this paper we concentrate on only these. 2

Modeling Process Analysis: The RIM Framework Stakeholders, in a collaborative modeling process, interact and communicate their ideas and opinions to other members through the communication process. Three key items concerning this communication are the rules, the interactions and the models. The rules, interactions and models (RIM) framework is based on these items and helps us look into the collaborative modeling process. This framework is depicted in Fig. 1. Details of the RIM framework are found in [ 7 ]. The RIM framework is a three-tier framework that examines the communicative acts (interactions) in a modeling session, the rules/goals set, and the models produced as a result of the interaction and collaboration. The di erent collaborative modeling players work under a set of rules and goals. The rules/goals, interactions and models are all time-stamped to help us track and identify he interplay between any pair. The interplay of rules, interactions and models is explained in Table 1. In order to analyze the interactive conversations and determine the structure of the speech-acts that result thereof, we need to apply a discourse analysis or conversation analysis technique. There are a number of methods which can be used, notably, speech-act theory by Searle [ 19 ]. Searle's aim in his \Theory of Speech Acts" [ 19 ] was to show that: \speaking a language is performing acts ( ) in accordance with certain rules for the use of the linguistic elements", and to formulate these rules. He argues that the minimal unit of an utterance is not a word or sentence but a \speech act". Two types of speech acts were identi ed in his theory: propositional act - which is the act of uttering words and illocutionary act - which is a complete speech act. An illocutionary act has two components: propositional content which describes what an utterance is about and illocutionary force describes the way it (utterance) is uttered. In addition, each illocutionary act has an illocutionary point which characterizes that particular type of speech act. Searle classi es utterances according to the illocutionary point and proposes ve classes of speech acts shown in Table 2.

However, as argued in [ 20 ], speech-acts are individual statements in the whole conversation and cannot be analyzed outside the whole conversation in which they occur. The language-action perspective (LAP) [ 21 ] is, therefore, a candidate in analysing the whole conversation in which the speech-acts are just components. We base our analysis of the communicative process on LAP to identify the conversational interactions that occur in a collaborative modeling process. Rules govern the interactions and production of the models. They guide collaborative modelers during the modeling process and can be set for (before) or in (during) the modeling process. They link the product of the conversations the model to the conversations and they are intended to guarantee both process quality and model quality. Rules are either explicitly stated or implicitly stated. The elements of a rule are given in Fig. 3 while Table 4 explains these elements. 2.3

Model Analysis: The structure

Models (intermediate or nal) are lists of propositions up to time t, i.e. conversational statements commonly agreed upon and shared by all the modelers. These model propositions are subject to selection criteria in order to determine which one makes it to the group (shared) model. In collaborative modeling a model proposition is either explicitly agreed with or implicitly not disagreed with. The structure of a model proposition component is shown in Fig. 4 while its elements are explained in Table 5.

Modeling Process Evaluation: An MCDA Framework In collaborative modeling a number of artifacts are used in, and produced during, the modeling process. These include the modeling language, the methods or approaches used to solve the problem, the intermediate and end-products produced and the medium or support tool that may be used to aid the collaboration, see for example [ 23 ]. The priorities of the individual decision makers need to be aggregated, so as to reach agreement and consensus on what should be the group's position as far as modeling process quality is concerned. Reaching agreement requires group decision making and negotiation. Group decision making and negotiation are special types of interactions during the modeling process. This is what provides a link between the analysis (RIM) framework and the evaluation (MCDA) approach. In Section 4, it will be shown how this link is exploited to get a uni ed framework for analysis and evaluation. In the evaluation, we use a Multi-criteria Decision Analysis (MCDA) method to evaluate the modeling artifacts. We speci cally use the single synthesizing (weighting) criterion preference approach - with Analytic Hierarchy Process (AHP) [ 24 ].

has is ameasure of is of Quality

(.nr) "ModelingArtifactIsEvaluatedInInteraction"

QualityCriteria (.name)

The structure of the evaluated modeling artifact component, within the MCDA evaluation framework, is shown in Fig. 5. The di erent concepts are explained in Table 6. One important observation about the modeling artifact and the evaluation framework is the link provided by the evaluated modeling artifact to the RIM framework through the interactions which are governed by rules. This is an important observation since it helps us to unify the two frameworks. 4

The Analysis and Evaluation Meta-model In this section we combine the components to form a uni ed model for the integrated analysis and evaluation (of process and results) of collaborative modeling. The aim of having a uni ed framework is twofold: 1) to trace the aws in the modeling process using the evaluation framework back to the analysis framework, 2) to automate the analysis and evaluation by a having support tool which can be used to both analyze and evaluate the modeling process. Although the analysis and evaluation frameworks can stand on their own, having a tool-support that can help modelers to analyze and evaluate the process and trace aws in the entire modeling process is more attractive than the individual frameworks. The components of the integrated frameworks are linked together in a meta-model shown in Fig. 6. The novelty of the meta-model is that it combines the analysis and evaluation frameworks, i.e., the RIM framework and the MCDA framework. This is easily visible in the meta-model where the triage of the rules (R), interactions (I) and models (M) in Fig. 1 is depicted through the rules, interactions and model proposition entities.

is de-activated by is activated by

is activated at Content has

has

SpeechAct Goal guides c(o.nntaaminse)exchange of is de-activated at ends at

Interaction (.name) stops at

starts at 5

Meta-Model in Use: Illustrative Examples To demonstrate the theoretical importance and practical signi cance of the model we provide below some illustrative examples. The examples are drawn from recorded communication/conversations that took place during a modeling session. 5.1

Application of the Meta-Model: The Analysis

Example 2. Rule analysis for Fig. 3 is based on the following excerpt of modeling session conversations. Extracted elements of a rule from the coded meta-data are given in Table 8. Time 01:25 01:30 08:43 08:45 08:48 15:18 15:19

Actor

M1 M2 M1 M2 M2 M1 M2

Speech Act

Let's create 5 swim lane diagrams.

Yes, isn't that what I just proposed? Sequences are started with the START symbol ... Yes ...

Use blocks to indicate activities.

So no decision diamonds in UML activity diagrams? No; well; maybe.

KEY: Int.: Interaction A.C.: Activation Content [A/D]: Activated/De-activated

M.P.: Model Proposition

Some explanation is in order for some of the concepts shown in Tables 7 and 8. The categories for coding the modeling conversations, i.e., the interaction names in both tables correspond to the dialogue types of Walton and Krable [ 25 ] whereas the topic names and rule categories, in Table 8, are explained in [ 7 ]. The validation goal is an example of an explicitly stated rule. This is activated at the start of the modeling session and remains so until de-activated at the end of the modeling session. The others are all implicitly stated and are (de-)activated during the interactions as shown by the (de-)activation content. It should be be noted that we use the terms \activation" and \de-activation" in the sense that modeler M1 starts the argument and modeler M2 concludes it in the sense of reaching a nal agreement. For each we identify, respectively, the interaction, content and time in (by, at) which the argument was started and concluded. Example 3. Model proposition analysis in Fig. 4 is based on the following excerpt. Extracted elements of a model proposition from the coded meta-data are given in Table 9.

Time 14:41 14:45 14:50 14:57 16:46 16:55 17:07 17:14

Actor

M1 M2 M2 M2 M1 M2 M1 M2

Speech Act

If there is no place, he can't order or there is no availability.

Yeah, true...

You cannot do decision diamonds in UML activity diagrams.

You can only have splits and joins of some sort, not the decisions as such.

We can also say that if the form isn't lled in well then it is rejected but...

Yeah ...

No-route and terminal point from "accept" in swim lane 7, with "no order" ...

OK..., Yes Example 4. Evaluation analysis in Fig. 5 is based on an evaluation instrument part of which is shown in Fig. 7. This instrument is used, rst by individual modelers, and then second by a team of modelers, to evaluate the modeling artifact (modeling language, modeling procedure, modeling products-the models and the support tool). The instrument shows, for example, how a modeling procedure is evaluated (using its selected quality criteria). These are assigned scores using the fundamental scale [ 24 ]. The quality criteria (quality dimensions of the modeling artifacts) are de ned in [ 23 ] and the process of assigning these quality criteria scores is explained therein. Upon reaching consensus through negotiation and decision making processes, modelers use these scores in the computation of priorities and the overall quality for the modeling artifacts as shown in Table. 10. 7/2/2010 3:01:18 PM Page 1 of 1 Numerical Assessment

Efficiency

Model Name: COME 9 8 7 6 5 4 3 2 1 2 3 4 5 6 7 8 9

Effectiveness

Compare the relative importance with respect to: Modeling Procedure Efficiency Effectiveness Satisfaction Commitment & Shared Understanding

Efficiency EffectivenesSatisfaction Commitmen 2.0 6.0 3.0 5.0 6.0

1.0

Incon: 0.07 The examples given, do illustrate how the analysis and evaluation frameworks can be used to, respectively, analyze and evaluate the modeling sessions. The interactions provide a driving force through the argumentations, negotiations, etc., for the modeling process while the rules and/or goals are a part and parcel of the structuring process during the modeling process, especially, when there is no facilitator. It has been observed in [ 7 ] that modelers structure the modeling process into pro-active rule and goal setting procedures and ad-hoc reactive rule and goal setting procedures. With this kind of structuring, it is possible to see how the rules are set for, and set in, the modeling session. Analysing the data from such a well-structured process helps us to pin-point to the types and categories of these rules and goals, the interaction types and it enables us to see how the modeling session unfolds and progresses and how models are created from (implicitly or explicitly) agreed upon statements. Identifying the drivers of the collaborative process in terms of rules, interactions and models is likely to enable development of guidelines that can be used in the development of an automated support tool for the analysis.

Figure 7 and Table 10 show, respectively, how the evaluation of the modeling process and the associated artifacts can be done and how the modelers' priorities can be aggregated. There are a number of modeling artifacts that are used in and developed during a collaborative modeling session. These include the modeling language, the modeling procedure, the models, and the support tool or medium. Analyzing what takes place during the modeling process, and what drives the modeling process won't be complete unless we assess and evaluate the quality of all these modeling artifacts. Evaluation is quite important since it gives assurance about the quality of these artifacts and through the meta-model we can trace the aws in the modeling process back to the analysis. One key observation is that the modeling artifacts' quality dimensions can be assigned quality scores during a negotiation and decision making (interactive) process using a multicriteria decision analysis technique, e.g., AHP [ 24 ], where the modelers' di erent priorities, preferences are reconciled and aggregated, and the overall quality is nally obtained by synthesizing the priorities. Rules and/or goals play a role since they direct and guide the modeling process. 6

Conclusion and Future Research

The contribution of the paper is twofold. First, it shows how the collaborative modeling process can be analyzed through the RIM framework and how it can be evaluated through the MCDA evaluation framework. Second, it develops a meta-model which uni es the analysis framework and the evaluation framework. To test the soundness of the meta-model, we provided illustrative examples from real modeling sessions. Though simple in description, these examples bring out well the concepts discussed for the meta-model. One key observation is that the types or names of the identi ed interactions are similar to those identi ed by Walton and Krabbe [ 25 ][ 26 ] in \Argumentation Theory", with the exception of the \eristic" dialogue.

Future Research Direction. For future research, we intend to apply the metamodel to modeling sessions, especially empirical tests with experts in industry to further test the theoretical signi cance and practical relevance and importance of the meta-model. More speci cally, we intend to further study and analyze the modeling process using a number of other factors other than those concentrated on in this paper, e.g., dialogue games and argumentation process through negotiation from a number of perspective, e.g., multi-agents, (see for example, [ 27,28 ]). We further intend to test our a priori hypothesis about the interdependencies of the modeling artifacts and how the quality of one a ects the quality of the other. We hypothesize that the the modeling language and the support tool are independent whereas the modeling products (models) and the modeling procedure are dependent variables in a multi-actor multi-criteria modeling session. Our intention is to empirically study this interdependency. Establishing this relationship is key in helping develop guidelines for a support tool that automates the analysis and evaluation of the modeling process.

References

1. Gjersvik , R. , Krogstie , J., F lstad, A.: Participatory Development of Enterprise Process Models . In: Krogstie, J. , Halpin , T. , Siau , K .(eds.), Information Modeling Methods and Methodologies , pp. 195 { 215 . IGI Global ( 2005 )

2. Stirna , J. , Persson , A. : Ten Years Plus with EKD: Re ections from Using an Enterprise Modeling Method in Practice . In: Proper, H.A. , Halpin , T.A. , Krogstie , J . (eds.), Proceedings of the EMMSAD 2007 , held in conjunctiun with CAiSE'07 , pp. 97 { 106 . Tapir Academic Press, Trondheim, Norway ( 2007 )

3. Barjis , J. , Kolfschoten , G.L. , Verbraeck , A. : Collaborative Enterprise Modeling . In E. Proper, F. Harmsen , and J.L.G Dietz (eds.): PRET 2009 , LNBIP 28, pp. 50 { 62 . Springer Heidelberg ( 2009 )

4. Veldhuijzen van Zanten, G. , Hoppenbrouwers , S.J.B.A. , Proper , H.A. : System Development as a Rational Communicative Process . J. of Systemics, Cybernetics and Informatics 2 ( 4 ), 47 - 51 ( 2004 )

5. Rittgen , P. : Negotiating Models . In Krogstie, J., Opdahl , A.L. , Sindre , G . (eds.) CAiSE 2007 . LNCS 4495, pp. 561 - 573 . Springer Heidelberg ( 2007 )

6. Clark , H.H. , Brennan , S.E. : Grounding in Communication . In: Resnick, L.B. , Levin , J.M. , Teasley , S.D. (eds.) Perspectives on Socially Shared Cognition , pp. 127 { 149 . American Psychology Association, Washington ( 1991 )

7. Ssebuggwawo , D. , Hoppenbrouwers , S.J.B.A. , Proper , H.A. : Interactions, Goals and Rules in a Collaborative Modeling Session . In: Persson, A. , Stirna , J . (eds.) PoEM 2009 . LNBIP 39, pp. 54 - 68 . Springer Berlin Heidelberg ( 2009 )

8. Pruit , D.G. : Negotiation Behaviour . Academic Press NewYork ( 1981 )

9. Rai a , H.: The Art and Science of Negotiation . Havard University Press, Cambridge, MA ( 1982 )

10. Rosenschien , J. , Zlotkin , G.: Designing Conventions for Automated Negotiations among Computers . MIT Press, ( 1994 )

11. Amgoud , L. , Prade , H.: Generation of Di erent Types of Arguments in Negotiation . In: Delgrande, J.P. , Schaub , T. (eds.), Proceedings of the 10th International Workshop on Non-Montonic Reasoning , Whistler, Canada, June 2-5, 2004 , pp. 10 { 15 ( 2004 )

12. McBurney , P. , van Eijk, R.M. , Parsons , S , Amgoud, L. : A Dialogue-game Protocol for Agent Purchase Negotiations . J. of Auto. Agents and Multi-agents Sys 7 ( 3 ), 235 { 273 ( 2003 )

13. Parsons , S , Jennings, N.R. : Negotiation Through Argumentation: A Preliminary Report. Proceedings of the 2nd International Conference on Multiagent Systems, Kyoto Japan , pp. 267 { 274 ( 1996 )

14. Parsons , S. , Sierra , C. , Jennings , N.: Agents that Reason and Negotaiate by Arguing . J. Logic Comput . 8 ( 3 ), 261 { 292 ( 1998 )

15. L^aasri, B. , La^asri, H., Lander , S. , Lesser , V.: A Generic Model for Intelligent Negotiating Agents . Int. J. of Cooperative Information Systems 1 , 291 { 317 ( 1992 )

16. Sandholm , T. , Lesser , V. : Issue in Automated Negotiation and E-Commerce. Extending the Contract Net Protocol . In Proceedings of the 1st International Conference on Muti-agent Systems , AAAIPress: Menlo Park, CA, USA, pp. 328 { 335 ( 1995 )

17. Sierra , C. , Dignum , F. : Agent Mediated Electronic Commerce: Scienti c and Technological Road-map . In Dignum, F. and sierra , C. (eds.), Agent Mediated Electronic Commerce 1991 . LNAI 1991 , pp. 1 { 18 . Springer-Verlag-Berlin ( 2001 )

18. Sackler , M.L. : The Unspoken Message . Modern Psychoanalysis 23 , 53 { 62 ( 1998 )

19. Searle , J.R. : Speech Acts. an Essay in the Philosophy of Language . London, Cambridge University Press ( 1969 )

20. Winograd , T. , Flores , F. : Understanding Computers and Cognition. A New Foundation for Design . Norwood, Ablex ( 1986 )

21. Goldkuhl , G.: Conversational Analysis as a Theoretical Foundation for Language Action Approaches ? In: Weigand, H. , Goldkuhl , G., de Moor , A . (eds.). In: 8th International Working Conference on the Language Action Perspective on Communication Modelling (LAP 2003 ), pp. 51 { 69 . Tilburg , The Netherlands ( 2003 )

22. Halpin , T. : Information Modeling and Relational Databases: From Conceptual Analysis to Logical Design . Morgan Kaufmann Pub ( 2001 )

23. Ssebuggwawo , D. , Stijn Hoppenbrouwers , S.J.B.A. , Proper , H.A. ( 2009 ). Evaluating Modeling Sessions Using the Analytic Hierarchy Process . In: Persson, A. , Stirna , J . (eds.) PoEM 2009 . LNBIP 39, pp. 69 - 83 . Springer Berlin Heidelberg ( 2009 )

24. Saaty , T.L. : The Analytic hierarchy Process . McGraw-Hill , New York ( 1980 )

25. Walton , D. , Krabbe , E.C.W.: Commitment in Dialogue: Basic Concepts of Interpersonal Reasoning . State University of New York Press, Albany, N.Y. ( 1995 )

26. Reed , C. , Norman , T.J.: Argumentation Machines: New Frontiers in Argument and Computation .Kluwer Academic Publishers, Dordrecht, The Netherlands ( 2004 )

27. Hindricks , K. , Jonker , C.M. , Tykhnov , D. : Negotiation Dynamics: Analysis, Concessions Tactics and Outcome . In: Proceedings of the 2007 IEEE/WIC/ACM International Conference on Multi Agent Technology , pp. 427 { 433 ( 2007 )

28. Markus , M.G. , Gerhard , W.: A Generic Framework for Argumentation Based Negotiation . In: Cooperative Multi Agents. LNCS 4676 , pp. 209 { 233 ( 2007 )