=Paper=
{{Paper
|id=Vol-2542/MOD20-TuD2
|storemode=property
|title=TOOL - Modeling Observatory & Tool: An Update
|pdfUrl=https://ceur-ws.org/Vol-2542/MOD20-TuD2.pdf
|volume=Vol-2542
|authors=Benjamin Ternes,Kristina Rosenthal,Hagen Barth,Stefan Strecker
|dblpUrl=https://dblp.org/rec/conf/modellierung/TernesRBS20
}}
==TOOL - Modeling Observatory & Tool: An Update==
https://ceur-ws.org/Vol-2542/MOD20-TuD2.pdf
Joint Proceedings of Modellierung 2020 Short, Workshop and Tools & Demo Papers
198 Modellierung 2020: Tools & Demo Papers
TOOL—Modeling Observatory & Tool: An Update
Benjamin Ternes,1 Kristina Rosenthal,1 Hagen Barth,1 Stefan Strecker1
Abstract: How do we perform conceptual modeling? What are common modeling difficulties? Which
tool support assists modelers in what respect? The paper at hand reports an update of the design
and development of a modeling observatory integrated with a modeling tool in support of studying
conceptual modeling. The modeling observatory implements a multi-modal observation approach
including tracking modeler-tool interactions, recording verbal data from modelers while modeling
and surveying modelers about their modeling processes. A configurable observation setup provides
support for conducting studies into individual modeling processes and analyses of modeling processes
at the individual and aggregate level. We report on the current state of prototype development, a proof
of concept in two exploratory studies and an outlook on future work.
Keywords: Conceptual modeling; Modeling tool; Tool development; Prototyping; Modeling process
1 Introduction
Conceptual modeling involves an intricate array of cognitive processes and performed
actions including abstracting, conceptualizing, contextualizing, associating, visualizing,
interpreting & sense-making, judging & evaluating, and, in group settings, communicating,
discussing and agreeing [RTS19]. Learning and performing conceptual modeling is, hence,
construed as a complex task based on codified and tacit knowledge [e. g., SS17] that involves
mastering theoretical foundations, modeling languages and methods, applying them to
practical problems as well as critically thinking and reflecting upon an application domain
[WO80]. Despite its complexity and relevance, we know surprisingly little about how
conceptual modeling is performed by modelers, how the learning of conceptual modeling
proceeds, which modeling difficulties modelers experience and why, and how to overcome
these difficulties by targeted modeling (tool) support [e. g., Se16].
We have been developing TOOL, a web-based modeling observatory and tool for studying
modeling processes since 2013 [e. g. Te19; TS18] as part of a long-term research program
aiming to better understand modeling processes and the learning of conceptual modeling—
following the overarching objective of enabling us to design and implement targeted tool
support for modelers at different stages of their learning and mastering of conceptual
modeling. The research program is based on the fundamental assumption that modeling
processes demand and deserve study from several complementary perspectives—to account
for the richness of cognitive processes involved in conceptual modeling and its complexity.
1 University of Hagen, Enterprise Modelling Research Group, Universitätsstr. 41, 58084 Hagen, Germany
{benjamin.ternes,kristina.rosenthal,hagen.barth,stefan.strecker}@fernuni-hagen.de
Copyright © 2020 for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
TOOL—Modeling Observatory and Tool 199
Hence, TOOL implements a multi-modal observation and data generation approach comple-
menting different modes of observation. Based on insights from evaluating the design and
implementation of TOOL in two exploratory studies [cf. RS19], we extended TOOL with a
configurable observation setup supporting a standardized and comparable data collection
tailored to the purpose and needs of studies into individual modeling processes.
2 TOOL presentation
Two essential requirements drive the software development: (1) platform independence to
the greatest possible extent and (2) usability (intuitive graphical user interface, GUI). Hence,
in an early design decision, we opted for a web application and an extensible modeling
tool with regard to modeling languages. At present, the modeling tool implements two
graphical modeling editors: (1) a variant of the Entity-Relationship Model (ERM) for data
modeling and (2) a subset of the Business Process Model and Notation (bpmn) 2.0 for
business process modeling. Modeling languages are implemented as stencil sets containing
the abstract and concrete syntax as well as concept specific functionalities, e. g., concerning
semantics for designators. Accordingly, stencil sets provide explicit typing, connection
rules, visual appearance, and other features that differentiate a model editor from generic
vector-oriented drawing tools.
Laboratory setup Virtual setup
Fig. 1: Use scenarios for observation setups using TOOL.
For studying modeling processes, TOOL supports complementary modes of observation
and analysis tools, i. e., (1) tracking modeler-tool interactions as timed-discrete events for
visualizing modeling processes as heatmaps, dot diagrams, and replays allowing for analyses,
(2) recording verbal data protocols, and (3) conducting pre- and post-modeling surveys.
Complementary observation modes can be selected and combined based on our fundamental
assumption that modeling processes demand study from different angles. Frontend design
considerations, operating principles and an earlier version of the data collection approach
are outlined in [TS18] and [Te19].
To support studies on modeling processes, we have extended the modeling observatory with
a customizable observation setup. TOOL is designed to support two main use scenarios
200 Benjamin Ternes, Kristina Rosenthal, Hagen Barth, Stefan Strecker
for studying modeling processes: (a) a laboratory and (b) a virtual setup (see Fig. 1). The
laboratory setup provides for observing individual modeling processes combining multiple
observation modes, i. a., including recording verbal protocols and videotaping modelers while
modeling, to gain a deeper understanding of individual modeling processes and in-depth
insights into the reasoning of modelers during their modeling processes. Studies in this setup
are expected to be accompanied by relatively small sample sizes as verbal protocol analysis is
recognized as a labor-intensive approach. The virtual scenario enables observing conceptual
modeling processes remote in large numbers using a tailored observation approach, e. g.,
tracking modeler-tool interactions complemented with surveying the modelers. Such a
setting enables studies aiming to identify patterns of modeling processes and modeling
difficulties. To support standardized and comparable data collection procedures, we have
implemented a customizable observation setup. Depending on the purpose and needs of
a study into individual modeling processes, observation parameters can be configured in
the GUI of the modeling observatory: selecting modeling exercises; customizing general
information, instructions and privacy statements; choosing the sequence and number of pre-
and post-modeling surveys; selecting complementary observation modes; and tailoring the
GUI of the modeling tool, e. g., hiding elements in the graphical modeling editors. Hence,
the resulting observation workflow supports a standardized and comparable data collection
procedure, and guides participants through the steps of a study.
(e.g., informed consent, information on data processing ) (e.g., select observation modes) (e.g., hide elements of the GUI)
Select a Privacy Configure Tailor user
Instructions
modeling exercise statements workflow interface
Select
Video-based Video introduction n Pre-modeling n Post-modeling
observation
modeling exercise into modeling tool survey(s) survey(s)
modes
(e.g., verbal recording, screen capturing)
Legend:
In progress
Initiates
Fig. 2: Overview of the configurable observation setup in TOOL.
Please note that due to privacy and security issues, the tool can only be accessed via a VPN
connection to the university network at the following link: http://tool.fernuni-hagen.de.
3 Proof of concept
The design and implementation of TOOL have been evaluated in two exploratory small-scale
studies into modeling difficulties individuals experience when constructing a conceptual
data model. In a first exploratory study identifying modeling difficulties in January 2019
[RS19], we observed eight learners of conceptual modeling working on a data modeling
TOOL—Modeling Observatory and Tool 201
task using TOOL applying complementary observation modes: recording verbal protocols,
videotaping modelers, tracking modeler-tool interactions and surveying subjects before and
after modeling. TOOL assisted by providing the modeling tool and recording modeler-tool
interactions, and it supported data analysis by visualizing modeler-tool interactions in dot
diagrams and replays. However, the configurable observation setup and the customizable GUI
were not supported in the modeling observatory yet—which required manual adjustments of
both the modeling observatory and the modeling tool for conducting the study. In a second
exploratory study in May to June 2019, we observed conceptual data modeling processes
of experienced modelers to deepen our understanding of modeling difficulties. The study
followed the same observation setup as in the first study complemented with recording the
screen during modeling. TOOL supported data collection with an observation setup that
we preconfigured in line with the chosen modes of observation, the observation procedure,
and a GUI tailored to the needs of the study, e. g., hiding the syntax checking function. The
analysis of the modeling processes is still in progress.
Conducting two exploratory studies demonstrated that TOOL supports not only the imple-
mented observation modes but also assists in analyzing the collected data. Configurable
observation setups and a customizable GUI provided by TOOL promise to keep data
collection efforts to a lower level—in contrast to adjusting the implementation for each
study or to using a combination of existing tools—and to contribute to a standardized and
comparable data collection procedure.
4 Outlook
TOOL needs further systematic testing and evaluation remaining on our research agenda.
Since November 2019, TOOL is applied in an introductory university course using the
implemented variant of the ERM with 200+ students per semester to investigate run-time
stability under high load and the implementation of the tracking approach in a large-scale
setup. In this setting, we are also preparing for a large-scale study in a virtual setup aimed at
identifying patterns of modeling processes and modeling difficulties—to further evaluate
the observation modes and the design and implementation of the configurable observation
setup. The evaluation will be complemented with further small-scale studies observing
experienced and non-experienced modelers aimed at deepening our understanding of
modeling difficulties in data modeling.
Please note that the previous studies are limited to observing data modeling processes. In a
next step, we prepare for future studies observing not only data modeling processes but also,
e. g., business process modeling processes with the bpmn 2.0.
Based on insights from multiple future studies and a better understanding of individual
modeling processes and modeling difficulties, we aim to extend TOOL by implementing
tool support that systematically and deliberately assists modelers while modeling and
that directly targets modeling difficulties. To provide further support complementing the
202 Benjamin Ternes, Kristina Rosenthal, Hagen Barth, Stefan Strecker
already implemented feedback on syntax errors based on ad-hoc syntax validation, we are
currently implementing tool support for data modeling suggesting identifiers for entity and
relationship types and attributes based on natural-language processing (NLP).
References
[RS19] Rosenthal, K.; Strecker, S.: Toward a Taxonomy of Modeling Difficulties : A
Multi-Modal Study on Individual Modeling Processes. In: 40th International
Conference on Information Systems (ICIS). Munich, Germany, 2019.
[RTS19] Rosenthal, K.; Ternes, B.; Strecker, S.: Learning Conceptual Modeling: Struc-
turing Overview, Research Themes and Paths for Future Research. In: 29th
European Conference on Information Systems (ECIS). Stockholm, Sweden,
Research Paper 137, 2019.
[Se16] Serral, E.; De Weerdt, J.; Sedrakyan, G.; Snoeck, M.: Automating Immediate
and Personalized Feedback: Taking Conceptual Modelling Education to a Next
Level. In: 10th International Conference on Research Challenges in Information
Science (RCIS). IEEE, Grenoble, France, pp. 1–6, 2016.
[SS17] Sedrakyan, G.; Snoeck, M.: Cognitive Feedback and Behavioral Feedforward
Automation Perspectives for Modeling and Validation in a Learning Context. In
(Hammoudi, S.; Pires, L.; Selic, B.; Desfray, P., eds.): Model-Driven Engineering
and Software Development. 4th International Conference, MODELSWARD
2016, Rome, Italy. Vol. 692, Springer, Cham, pp. 70–92, 2017.
[Te19] Ternes, B.; Strecker, S.; Rosenthal, K.; Barth, H.: A browser-based modeling
tool for studying the learning of conceptual modeling based on a multi-modal
data collection approach. In: 14. Internationale Tagung Wirtschaftsinformatik
(WI). Siegen, Germany, pp. 1984–1988, 2019.
[TS18] Ternes, B.; Strecker, S.: A web-based modeling tool for studying the learning
of conceptual modeling. In (Schaefer, I.; Karagiannis, D.; Vogelsang, A.;
Méndez, D.; Seidl, C., eds.): Modellierung 2018. Lecture Notes in Informatics.
Vol. 280, Gesellschaft für Informatik e.V., Braunschweig, Germany, pp. 325–328,
2018.
[WO80] Wedekind, H.; Ortner, E.: Systematisches Konstruieren von Datenbankanwen-
dungen: Zur Methodologie der angewandten Informatik. Carl Hanser Verlag,
Munich, 1980.