TOOL{A Modeling Observatory & Tool for Studying Individual Modeling Processes

Abstract

We present TOOL, a browser-based modeling tool integrated with a modeling observatory for studying individual modeling processes, e.g., when constructing a data model. To account for the richness and complexity of the cognitive processes involved in conceptual modeling, modelers' modeling processes demand study from multiple, complementary angles and perspectives. TOOL integrates a multi-modal data collection approach including (1) tracking modeler-tool interactions (via the user interface), (2) recording verbal data protocols of modelers' thinking out loud, (3) screen captures, and (4) surveying modelers|to provide a more complete picture at the individual and aggregate modelers level in the quest for identifying patterns of modeling processes and modeling di culties. We report on the current state of prototype development, discuss the tool and its modes of observation, and outline future work on supporting modelers and on meta-modeling in TOOL.

Keywords

• Conceptual Modeling
• Modeling Tool
• Modeling Observatory
• Prototyping

-

Viewed as an activity, conceptual modeling involves an intricate array of cognitive processes and performed actions and, hence, is construed as a complex task involving codi ed and tacit knowledge (cf. [ 8 ]). Despite its complexity and relevance (e. g., [ 10 ]), surprisingly little is known about individual conceptual modeling processes. Research on observing modeling processes has only recently seen increasing interest with contributions, e. g., focusing on business process modeling [ 5 ], learning tool support [ 9 ] or on neuro-adaptive modeling environments [ 14 ]. To learn more about how conceptual modeling is performed by modelers, which modeling di culties they encounter and why, and how to overcome these di culties by targeted modeling (tool) support, we have been researching and developing TOOL, a web-browser-based modeling observatory and tool.

TOOL is part of a long-term research program to better understand individual modeling processes and to develop targeted tool support for modelers while conceptual modeling [ 11 ]. Research on TOOL is based on the fundamental assumption that modeling processes deserve study from multiple complementary angles and perspectives|to account for the richness of the cognitive processes and performed actions involved in conceptual modeling and its complexity. Hence, TOOL implements means to realize mixed method research designs based on multi-modal data collection (e. g., [ 6, 8 ]). 2

TOOL Prototype Overview

TOOL comprises a web-based modeling tool for constructing conceptual models (see Fig. 1) and a modeling observatory for studying individual modeling processes and includes corresponding analysis tools. Two essential requirements drive the prototype development: (i) platform independence, and (ii) usability, in particular an intuitive (graphical) user interface. Design considerations, operating principles and essential requirements are outlined in, e. g., [ 12 ]. At present, the modeling tool implements two graphical modeling editors for constructing (i) data models with a variant of the Entity-Relationship (ER) Model and (ii) business process models implementing a subset of the Business Process Model and Notation|bpmn 2.0. Both graphical editors are supported by ad-hoc syntax validation to check the syntactic correctness of conceptual models. Syntax checking is currently based on explicit typing and connection rules provided by stencil sets which contain the abstract and concrete syntax as well as speci c functions for, e. g., designators of roles.

TOOL supports studies of individual modeling processes in (i) a laboratory or eld setting, and (ii) in a virtual setup when used as a modeling observatory (cf. [ 13 ]). The modeling observatory provides four data collection approaches to support the study of modeling processes: The observatory supports (1) tracking modeler-tool interactions as timed-discrete events which can be subsequently visualized as (a) step-by-step replays (up to four models at the same time), (b) heatmaps and (c) dot diagrams (see Fig. 2; further details are shown in [ 8 ]). Since modeler-tool interactions are a rather restricted mode of observation of individual modeling processes, we opted for additionally recording (2) verbal data protocols by asking modelers to think out loud while modeling|or subsequent to model creation (concurrent and retrospective think-aloud, see [ 3, 1 ]). To gain further insights into how modelers operate with the modeling tool respectively its graphical editors, TOOL supports recording (3) screen captures based on WebRTC to provide a video recording of the modeling process. Beyond these modes of observation, TOOL integrates a component for (4) creating surveys and for visualizing their results (see Fig. 2). Depending on the needs of a study, an observation work ow user interface allows for con guring the selection and sequence of observation modes (cf. [ 13 ]). A video demonstrator of TOOL is available at: https://vimeo.com/441854796/5237d3782a. Fig. 2. Overview of the visual analysis components of TOOL.

Discussion and Outlook

TOOL has been a research subject for the past seven years and has continuously been under development. TOOL has been made available to students of an introductory course on modeling business information systems, and is currently in use by students to work on modeling tasks in the course material. Performance, scalability and stability of the running prototype have shown robust for the past year uptime albeit with moderate systems loads (60 to 80 students). We have employed TOOL in two exploratory studies on individual data modeling processes. In a rst study, we observe eight learners of conceptual modeling working on a data modeling task to identify modeling di culties these modelers experience [ 6 ], while a second study observes eight experienced modelers to discuss similarities and di erences in modeling di culties comparing non-experienced and experienced modelers [ 7 ].

To overcome modeling di culties in labeling modeling elements (e.g., [ 2, 7 ]), we extend TOOL by implementing an automated feedback component based on Natural Language Processing (NLP) techniques to provide suggestions on labeling model elements at modeling time. The feedback component is among the rst implementations to integrate a web-based data modeling tool with NLP technology, i. e., the Stanford CoreNLP toolkit (cf. [ 4 ]) to automatically process and understand an arbitrary natural language description of a modeling task in terms of its morphological structure to identify words and phrases as suggestions for labels for model elements. A preliminary evaluation of the feedback component demonstrates its usefulness by providing sensible and adequate suggestions to modelers. Furthermore, a meta-modeling component is currently developed that allows to implement modeling languages as graphical meta-models rather than text-based stencil sets.

References

1. Batra , D. , Davis , J.G. : Conceptual data modelling in database design: similarities and di erences between expert and novice designers . International Journal of ManMachine Studies 37 ( 1 ), 83 { 101 ( 1992 )

2. Delfmann , P. , Herwig , S. , Lis , L. : Uni ed enterprise knowledge representation with conceptual models { capturing corporate language in naming conventions . In: 30th International Conference on Information Systems. Research Paper 45 . Phoenix, Arizona, USA ( 2009 )

3. Ericsson , K.A. , Simon , H.A. : Protocol analysis: Verbal reports as data . MIT Press, Cambridge, MA, 2nd edn. ( 1993 )

4. Manning , C.D. , Surdeanu , M. , Bauer , J. , Finkel , J. , Bethard , S.J. , McClosky , D. : The Stanford CoreNLP Natural Language Processing Toolkit . In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations . pp. 55 { 60 . Association for Computational Linguistics, Baltimore, Maryland ( 2014 )

5. Pinggera , J. , So er, P., Fahland , D. , Weidlich , M. , Zugal , S. , Weber , B. , Reijers , H.A. , Mendling , J.: Styles in business process modeling: an exploration and a model . Software and Systems Modeling 14 ( 3 ), 1055 { 1080 ( 2015 )

6. Rosenthal , K. , Strecker , S. : Toward a taxonomy of modeling di culties : A multimodal study on individual modeling processes . In: 40th International Conference on Information Systems . Munich, Germany ( 2019 )

7. Rosenthal , K. , Strecker , S. , Pastors , O. : Modeling di culties in data modeling: Similarities and di erences between experienced and non-experienced modelers . In: 39th International Conference on Conceptual Modeling (ER) . Vienna, Austria ( 2020 ), accepted for publication

8. Rosenthal , K. , Ternes , B. , Strecker , S. : Understanding individual processes of conceptual modeling: A multi-modal observation and data generation approach . In: Bork, D. , Karagiannis , D. , Mayr , H.C. (eds.) Modellierung 2020 . pp. 77 { 92 . Gesellschaft fur Informatik e.V., Bonn ( 2020 )

9. Sedrakyan , G. , De Weerdt , J. , Snoeck , M. : Process-mining enabled feedback: "tell me what i did wrong" vs. "tell me how to do it right" . Computers in Human Behavior 57 , 352 { 376 ( 2016 )

10. Sedrakyan , G. , Snoeck , M. , De Weerdt , J.: Process mining analysis of conceptual modeling behavior of novices { Empirical study using JMermaid modeling and experimental logging environment . Computers in Human Behavior 41 , 486 { 503 ( 2014 )

11. Strecker , S. : Enterprise Modelling Research Group at University of Hagen. In: Koschmider, A. , Weidlich , M. (eds.) 40 Years EMISA { Poster Track, Lecture Notes in Informatics , Vol. P- 304 . pp. 195 { 199 . Gesellschaft fur Informatik e.V., Bonn ( 2020 )

12. Ternes , B. , Strecker , S.: A web-based modeling tool for studying the learning of conceptual modeling . In: Schaefer, I. , Karagiannis , D. , Vogelsang , A. , Mendez , D. , Seidl , C. (eds.) Modellierung 2018. Lecture Notes in Informatics , Vol. 280 . pp. 325 { 328 . Gesellschaft fur Informatik e.V., Braunschweig, Germany ( 2018 )

13. Ternes , B. , Rosenthal , K. , Barth , H. , Strecker , S. : TOOL{Modeling Observatory & Tool: An Update . In: Joint Proceedings of Modellierung 2020 Short, Workshop and Tools & Demo Papers . pp. 198 { 202 . CEUR-WS , Vienna, Austria ( 2020 )

14. Weber , B. , Pinggera , J. , Neurauter , M. , Zugal , S. , Martini , M. , Furtner , M. , Sachse , P. , Schnitzer , D. : Fixation Patterns During Process Model Creation: Initial Steps Toward Neuro-adaptive Process Modeling Environments . In: 49th Hawaii International Conference on System Sciences (HICSS) . pp. 600 { 609 . IEEE, Piscataway, NJ ( 2016 )