How do we perform conceptual modeling? What are common modeling dificulties? Which tool support assists modelers in what respect? The paper at hand reports an update of the design and development of a modeling observatory integrated with a modeling tool in support of studying conceptual modeling. The modeling observatory implements a multi-modal observation approach including tracking modeler-tool interactions, recording verbal data from modelers while modeling and surveying modelers about their modeling processes. A configurable observation setup provides support for conducting studies into individual modeling processes and analyses of modeling processes at the individual and aggregate level. We report on the current state of prototype development, a proof of concept in two exploratory studies and an outlook on future work.
Conceptual modeling involves an intricate array of cognitive processes and performed actions including abstracting, conceptualizing, contextualizing, associating, visualizing, interpreting & sense-making, judging & evaluating, and, in group settings, communicating, discussing and agreeing [RTS19]. Learning and performing conceptual modeling is, hence, construed as a complex task based on codified and tacit knowledge [e. g., SS17] that involves mastering theoretical foundations, modeling languages and methods, applying them to practical problems as well as critically thinking and reflecting upon an application domain [WO80]. Despite its complexity and relevance, we know surprisingly little about how conceptual modeling is performed by modelers, how the learning of conceptual modeling proceeds, which modeling dificulties modelers experience and why, and how to overcome these dificulties by targeted modeling (tool) support [e. g., Se16].
We have been developing TOOL, a web-based modeling observatory and tool for studying modeling processes since 2013 [e. g. Te19; TS18] as part of a long-term research program aiming to better understand modeling processes and the learning of conceptual modeling— following the overarching objective of enabling us to design and implement targeted tool support for modelers at diferent stages of their learning and mastering of conceptual modeling. The research program is based on the fundamental assumption that modeling processes demand and deserve study from several complementary perspectives—to account for the richness of cognitive processes involved in conceptual modeling and its complexity. Hence, TOOL implements a multi-modal observation and data generation approach complementing diferent modes of observation. Based on insights from evaluating the design and implementation of TOOL in two exploratory studies [cf. RS19], we extended TOOL with a configurable observation setup supporting a standardized and comparable data collection tailored to the purpose and needs of studies into individual modeling processes. 2
Two essential requirements drive the software development: (1) platform independence to the greatest possible extent and (2) usability (intuitive graphical user interface, GUI). Hence, in an early design decision, we opted for a web application and an extensible modeling tool with regard to modeling languages. At present, the modeling tool implements two graphical modeling editors: (1) a variant of the Entity-Relationship Model (ERM) for data modeling and (2) a subset of the Business Process Model and Notation (bpmn) 2.0 for business process modeling. Modeling languages are implemented as stencil sets containing the abstract and concrete syntax as well as concept specific functionalities, e. g., concerning semantics for designators. Accordingly, stencil sets provide explicit typing, connection rules, visual appearance, and other features that diferentiate a model editor from generic vector-oriented drawing tools.
Laboratory setup
Virtual setup
For studying modeling processes, TOOL supports complementary modes of observation and analysis tools, i. e., (1) tracking modeler-tool interactions as timed-discrete events for visualizing modeling processes as heatmaps, dot diagrams, and replays allowing for analyses, (2) recording verbal data protocols, and (3) conducting pre- and post-modeling surveys. Complementary observation modes can be selected and combined based on our fundamental assumption that modeling processes demand study from diferent angles. Frontend design considerations, operating principles and an earlier version of the data collection approach are outlined in [TS18] and [Te19].
To support studies on modeling processes, we have extended the modeling observatory with a customizable observation setup. TOOL is designed to support two main use scenarios for studying modeling processes: (a) a laboratory and (b) a virtual setup (see Fig. 1). The laboratory setup provides for observing individual modeling processes combining multiple observation modes, i. a., including recording verbal protocols and videotaping modelers while modeling, to gain a deeper understanding of individual modeling processes and in-depth insights into the reasoning of modelers during their modeling processes. Studies in this setup are expected to be accompanied by relatively small sample sizes as verbal protocol analysis is recognized as a labor-intensive approach. The virtual scenario enables observing conceptual modeling processes remote in large numbers using a tailored observation approach, e. g., tracking modeler-tool interactions complemented with surveying the modelers. Such a setting enables studies aiming to identify patterns of modeling processes and modeling dificulties. To support standardized and comparable data collection procedures, we have implemented a customizable observation setup. Depending on the purpose and needs of a study into individual modeling processes, observation parameters can be configured in the GUI of the modeling observatory: selecting modeling exercises; customizing general information, instructions and privacy statements; choosing the sequence and number of preand post-modeling surveys; selecting complementary observation modes; and tailoring the GUI of the modeling tool, e. g., hiding elements in the graphical modeling editors. Hence, the resulting observation workflow supports a standardized and comparable data collection procedure, and guides participants through the steps of a study.
(e.g., informed consent, information on data processing ) (e.g., select observation modes) (e.g., hide elements of the GUI)
Select a modeling exercise
Instructions
Privacy statements
Video-based modeling exercise
Video introduction into modeling tool n Pre-modeling
survey(s) Legend:
In progress Initiates Configure workflow
Select observation
modes (e.g., verbal recording, screen capturing)
Tailor user interface n Post-modeling survey(s)
Please note that due to privacy and security issues, the tool can only be accessed via a VPN connection to the university network at the following link: http://tool.fernuni-hagen.de. 3
The design and implementation of TOOL have been evaluated in two exploratory small-scale studies into modeling dificulties individuals experience when constructing a conceptual data model. In a first exploratory study identifying modeling dificulties in January 2019 [RS19], we observed eight learners of conceptual modeling working on a data modeling TOOL—Modeling Observatory and Tool 201 task using TOOL applying complementary observation modes: recording verbal protocols, videotaping modelers, tracking modeler-tool interactions and surveying subjects before and after modeling. TOOL assisted by providing the modeling tool and recording modeler-tool interactions, and it supported data analysis by visualizing modeler-tool interactions in dot diagrams and replays. However, the configurable observation setup and the customizable GUI were not supported in the modeling observatory yet—which required manual adjustments of both the modeling observatory and the modeling tool for conducting the study. In a second exploratory study in May to June 2019, we observed conceptual data modeling processes of experienced modelers to deepen our understanding of modeling dificulties. The study followed the same observation setup as in the first study complemented with recording the screen during modeling. TOOL supported data collection with an observation setup that we preconfigured in line with the chosen modes of observation, the observation procedure, and a GUI tailored to the needs of the study, e. g., hiding the syntax checking function. The analysis of the modeling processes is still in progress.
Conducting two exploratory studies demonstrated that TOOL supports not only the implemented observation modes but also assists in analyzing the collected data. Configurable observation setups and a customizable GUI provided by TOOL promise to keep data collection eforts to a lower level—in contrast to adjusting the implementation for each study or to using a combination of existing tools—and to contribute to a standardized and comparable data collection procedure. 4
TOOL needs further systematic testing and evaluation remaining on our research agenda. Since November 2019, TOOL is applied in an introductory university course using the implemented variant of the ERM with 200+ students per semester to investigate run-time stability under high load and the implementation of the tracking approach in a large-scale setup. In this setting, we are also preparing for a large-scale study in a virtual setup aimed at identifying patterns of modeling processes and modeling dificulties—to further evaluate the observation modes and the design and implementation of the configurable observation setup. The evaluation will be complemented with further small-scale studies observing experienced and non-experienced modelers aimed at deepening our understanding of modeling dificulties in data modeling.
Please note that the previous studies are limited to observing data modeling processes. In a next step, we prepare for future studies observing not only data modeling processes but also, e. g., business process modeling processes with the bpmn 2.0.
Based on insights from multiple future studies and a better understanding of individual modeling processes and modeling dificulties, we aim to extend TOOL by implementing tool support that systematically and deliberately assists modelers while modeling and that directly targets modeling dificulties. To provide further support complementing the [RS19] [RTS19] [Se16] [SS17] [Te19] [TS18] [WO80] already implemented feedback on syntax errors based on ad-hoc syntax validation, we are currently implementing tool support for data modeling suggesting identifiers for entity and relationship types and attributes based on natural-language processing (NLP).
Rosenthal, K.; Strecker, S.: Toward a Taxonomy of Modeling Dificulties : A Multi-Modal Study on Individual Modeling Processes. In: 40th International Conference on Information Systems (ICIS). Munich, Germany, 2019.
Rosenthal, K.; Ternes, B.; Strecker, S.: Learning Conceptual Modeling: Structuring Overview, Research Themes and Paths for Future Research. In: 29th European Conference on Information Systems (ECIS). Stockholm, Sweden, Research Paper 137, 2019.
Serral, E.; De Weerdt, J.; Sedrakyan, G.; Snoeck, M.: Automating Immediate and Personalized Feedback: Taking Conceptual Modelling Education to a Next Level. In: 10th International Conference on Research Challenges in Information Science (RCIS). IEEE, Grenoble, France, pp. 1–6, 2016.
Sedrakyan, G.; Snoeck, M.: Cognitive Feedback and Behavioral Feedforward Automation Perspectives for Modeling and Validation in a Learning Context. In (Hammoudi, S.; Pires, L.; Selic, B.; Desfray, P., eds.): Model-Driven Engineering and Software Development. 4th International Conference, MODELSWARD 2016, Rome, Italy. Vol. 692, Springer, Cham, pp. 70–92, 2017.
Ternes, B.; Strecker, S.; Rosenthal, K.; Barth, H.: A browser-based modeling tool for studying the learning of conceptual modeling based on a multi-modal data collection approach. In: 14. Internationale Tagung Wirtschaftsinformatik (WI). Siegen, Germany, pp. 1984–1988, 2019.