=Paper=
{{Paper
|id=Vol-3397/sci2
|storemode=property
|title=Neuro-Conceptualization: Visual Conceptual Modeling meets Neuroscience
|pdfUrl=https://ceur-ws.org/Vol-3397/sci2.pdf
|volume=Vol-3397
|authors=John Krogstie,Kshitij Sharma
|dblpUrl=https://dblp.org/rec/conf/emisa/KrogstieS23
}}
==Neuro-Conceptualization: Visual Conceptual Modeling meets Neuroscience==
https://ceur-ws.org/Vol-3397/sci2.pdf
Neuro-Conceptualization:
Visual Conceptual Modeling meets Neuroscience
John Krogstie and Kshitij Sharma
Norwegian University of Science and Technology, NTNU, Trondheim, Norway
Abstract
A lot of research has been done on the comprehension and development of conceptual models.
In other related areas such as linguistics and software engineering one has taken techniques
from neuroscience into use, to study the biological and neurological processes when working
with textual knowledge representations. This has only to a limited extent been the case when
it comes to visual conceptual models so far.
We will in this paper present ongoing research on the use of techniques from neuroscience to
investigate how we develop and comprehend visual conceptual models. Traditionally, neuro-
science techniques have been depending on EEG or even large MR-machines for techniques
such as fMRI, and we outline planned work, also for being able to study modeling tasks closer
to how they are actually performed by using multimodal data analysis.
Keywords 1
Novel directions Talk, Conceptual process modeling, NeuroIS
1. Introduction
NeuroIS is a research field in which neuroscience theories and tools are used to better understand
information systems phenomena. Existing research areas in NeuroIS is summarized in [14], where it
appears that the main focus is on the use of information systems. Lately also software development
tasks such as programming has been heavily studied in literature [12], whereas other tasks often linked
to IS-development such as visual conceptual modeling has so far to a very limited degree been studied
using techniques from neuroscience beyond the use of eye-tracking techniques [5].
Visual conceptual modeling is a central activity in information systems analysis and design. It
involves the construction of abstract models that capture the structure, behavior, and relationships of
real-world entities or concepts, and the two-dimensional layout allows to play with both the primary
and secondary notation [7] to convey meaning. Integrating neuroscience techniques into conceptual
modeling open up new opportunities for understanding how the human brain processes and represents
complex information, which can, in turn, inform and enhance the development of more effective
modeling approaches when looked upon together with insights form fields such as conceptual modeling,
linguistics, and cognitive psychology.
The use of neuroscience in conceptual modeling primarily focuses on understanding the neural
mechanisms underlying concept formation, representation, comprehension and manipulation. By
leveraging advanced neuroimaging techniques such as functional magnetic resonance imaging (fMRI2),
functional near-infrared spectroscopy (fNIRS3), and electroencephalography (EEG4), researchers can
13th International Workshop on Enterprise Modeling and Information Systems Architectures (EMISA) Stockholm, Sweden 2023
EMAIL: john.krogstie@ntnu.no, Kshitij.sharma@ntnu.no
ORCID: 0000-0003-4830-1876; 0000-0003-3364-637X
© 2023 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
2
fMRI: A magnetic resonance imaging (MRI) scanner measures blood oxygenation in the brain and exploits the different magnetic properties
of oxygenated and deoxygenated blood.
3
Functional Near-Infrared Spectroscopy (fNIRS) is a brain imaging technique that (like fMRI) uses hemodynamic responses to indirectly
capture neuronal activity. However, compared to fMRI, fNIRS is less expensive and more portable, offering higher ecological validity
4
Electroencephalograms (EEG) are recordings of the electrical activity of neurons in the brain. Using electrodes placed on the scalp, EEG
measures the summation of synchronous postsynaptic action potentials produced by a population of neurons with a very high temporal
precision (milliseconds)
examine brain activation patterns and connectivity while participants engage in tasks that require
conceptual reasoning or problem-solving. A challenge with some of the more advanced techniques from
neuroscience such as fMRI is that the accuracy of results comes at a cost, in particular on the ecological
validity of the trial situation and the cost-benefit of the technique used, thus we are aiming for using
less intrusive techniques in concert in combination with multimodal data analytics. A comprehension
description of current techniques in neuroscience as applied in informatics can be found in [12].
Moreover, the study of individual differences in conceptual processing and the neural basis of
expertise in specific domains can provide valuable information on the factors that contribute to the
development of expert-level conceptual reasoning and problem-solving abilities.
The use of neuroscience techniques in connection to conceptual modeling has the potential to
significantly advance our understanding of the neural basis of complex information comprehension,
processing and representation. By combining insights from both fields, researchers can develop more
effective conceptual modelling approaches that better align with the inherent capabilities and
constraints of the human brain.
As mentioned, techniques such as eye-tracking is used quite a bit for model comprehension and
modeling process analysis [1], but papers in the area primarily mention a more extensive treatment with
neuroscience techniques to be tried as the next step [15], although other techniques are gradually being
taken into use [13]. In this novel direction talk, we will present preliminary plans for experiments on
how to use input from a number of different sensors to do multimodal data analysis for proving better
understanding on how the brain is doing different modeling tasks.
2. Current research plan
We plan a number of experiments, starting with studying simple model comprehension tasks, being
extended to more complex tasks in a setting as close to a normal modeling situation as sensors improves.
A normal modeling situation will compared to laying still in an MR-machine involve movement that
typically introduce noise which it is hard to deal with in with some of the current sensors, but we see
this has improved over the last years and is expected to be further improved. As a start we envisage the
following experiments: (partly with inspiration from what that is recently done in the field of code
comprehension):
1. Investigate the usage of the brain when working with conceptual/visual models:
RQ 1: Which brain regions are activated during model comprehension (similar to what is done in
[9], where one looked at the brain regions active when performing computer program comprehension
tasks). Experimental task: a model (in a modeling language known to the participant) is presented to
the user, which is to use this for answering presented comprehension questions. Will start with a model
in one diagram (i.e., the whole model is visible, no need for scrolling/navigating in a hierarchy etc. to
avoid to much bodily movement by the participant. The models to use are similar to those used in
standard process model comprehension tasks [6].
2. Investigate the usage of the brain when working with conceptual/visual models as compared to
how it operates when using a text expressing the same information (with possibly some parallel to what
is done in [2] where differences in working with visual and textual programming languages were
investigated)
Main hypothesis: Different parts of the brain is used more intensively when working with visual
knowledge representation than when working with textual knowledge representations. Experimental
task: Have two domains, both presented as a model and a text, and comprehension questions for both
domains. Have a Latin-square set-up to give participants different settings, e.g., one group first see a
model of domain A, and then a text of domain B etc. In addition to study the parts of the brain used,
measure cognitive load and possibly also other characteristics (see below).
3: Investigate the usage of the brain when using different modeling languages (BPMN vs. UML AD
for process modeling for instance). Have a similar latin-square set-up based on models in both languages
representing two different domains.
4. How do layout and other aspects of secondary notation influence model comprehension (cf. [9],
where they looked at how layout and beacons in source code influence program comprehension). Need
to be detailed based on issues found in cognitive psychology, listed as empirical model quality issue in
the SEQUAL framework on model quality [4].
5. How can detected information on e.g., cognitive load be used to provide feedback tools to support
the modeler.
A more detailed set-up currently done in connection to the first two tasks also extending into
affective and behavioral aspects are presented below:
RQ1: What are the differences in affective, behavioral and cognitive processes across different
levels of model comprehension?
• Sub-RQ: what are the major brain regions responsible for visual model comprehension?
• Sub-RQ: how does the cognitive load evolve during the comprehension process?
• Sub-RQ: what are the roles of stress and physiological arousal leading to certain comprehension
performance level?
Prediction question: how accurately can we predict the comprehension level of the modelers using
the affective, behavioral, and cognitive measurements using deep learning networks? What are the
various dimensions of explainability in such a predictive model?
The experiment follows a time series repeated measure design as follows, with:
Where the models are represented both in text and as visual business process models, and the
participants are divided in a Latin-Square fashion. NASA TLX is a self-assessment of task load [11].
We aim to have around 60-70 participants in total. Participants will be recruited from NTNU student
and employee population. NTNU has more than 40000 students across all academic fields. The way the
study is to take care of protecting the participants privacy has been reported to and accepted by the
national authorities in this matter (NSD-approval).
Figure 1: Test setup
Figure 1 illustrates the set-up and the different sensors in the multimodal experimental setup. The
setup can provide a large variety of the information about the situation, not all necessary for the specific
task, which will be listed below, and also support a richer level of man-machine interactivity than e.g.,
the use of fMRI and other techniques dependent on more heavy machinery (e.g. MR-machines) pursued
in contemporary neuroscience.
2.1. EEG Measurements
Lower Alpha wave power (8—9 Hz): Relaxed, but not drowsy
Upper Alpha wave power (10—12 Hz): Normal, relaxed yet focused
Lower Beta wave power (13—17 Hz): Normal, relaxed yet focused
Upper Beta wave power (18—30 Hz): Active thinking, alertness
Gamma wave power (more than 30 Hz): Higher mental activity
Theta wave power (4—7 Hz) : Idling, inefficiency, related to ADHD
Fractal dimensions: Long term memory
Bursts in alpha wave: Performance
Alpha magnitude: Workload
Decreasing Alpha and Theta band power: Cognitive load
Phase coupling of Theta and Gamma: Short term memory
Alpha band power: Attention
Wavelet coefficients of Gamma band: Emotional intensity
Theta band power: Memory load
Increasing theta and decreasing alpha: Increasing performance
Gamma-band activity: Short term memory load
Decreasing Theta and Increasing Alpha Working memory
2.2. Eye Tracking Measurements
AOI5 hit: How much attention is paid to the different parts of the visual field.
AOI transition: Attention shift proportions.
AOI revisit on the screen: Failure of memory/Need for confirmation.
Local or global saccades (threshold on the saccade length): Focal6 or ambient7 cognitive
processing.
Short fixations on a specific part : Anticipation of finding information.
Number of saccades per unit time: Decreasing task difficulty/ Increasing mental
workload/ Increasing arousal.
Number of blinks per second: Time on task, mental workload, fatigue.
Blink duration: Drowsiness, mental workload.
Number of fixations per second Decreasing search efficiency/Expertise based
on the task / Difficulty in interpretation.
Fixation duration: Attention.
Skewness of fixation duration histogram: Cognitive processing.
Pupil diameter: Increased emotions, anticipation mental
workload/ Decreased drowsiness, fatigue.
Saccadic velocity: High cognitive load, task difficulty, memory
load.
Scanpath Velocity = number of forward saccades/number of backward saccades: Amount of
information processing.
5
AOI = Areas of interest that are defined by the researcher
6
Focal: short saccades and long fixations
7
Ambient = long saccades and short fixations
Saccade duration: Task difficulty/ Decrease in information
processing.
Saccade velocity: Low arousal and sleepiness
Saccade velocity skewness: Anticipation
Cross-recurrence (probability of looking at the same place at the same time): Increases
during dialogue episodes also increase during certain dialogue types and verbal and deictic
references. Correlates with collaboration quality and outcome.
Scanpath shape similarity (Overall shape, Same shape but different scale, Similar subparts):
Collaborative outcome.
Gaze similarity: Similar to cross recurrence but temporal
measurement.
Mean and SD pupil diameter; number of long fixations; saccade length: Cognitive load.
2.3. Wristband Measures
Wristband measures are mostly used in prediction, so we lack the measures that are directly
interpreted: HR = heart rate, EDA = electrodermal activation (skin conductance), BVP = blood volume
pressure, TEMP = skin temperature
EDA peak height / EDA peak rate / EDA slope: Cognitive load.
BVP power spectrum low/high ratio / BVP amplitude / # EDA responses detected/ EDA
mean/ EDA rising time / TEMP slope: Stress.
TEMP (mean, sd, kurtosis, skewness)/ EDA peaks / HR variability (mean, sd, kurtosis,
skewness): Emotional stress.
EDA change detection measures: Acute stress cycle (normal, aroused,
stressed, relaxed).
HR recovery rate changes (duration and counts): Chronic stress.
We can additionally compute the action units (AUs) from the faces of the participants capture using
cameras. Once we have these AUs then we can compute various emotions as the combination of these
AUs, such as Happiness, Sadness, Surprise, Fear, Anger, Disgust, and Contempt. Second, we can
compute the emotional profile (entropy of AUs, stability of emotions, emotional similarity between the
peers) of the participants similar to [12]. We have not space here to go in detail on the machine learning
interpretation of data, but this will be presented at the conference. We also note that using a large
number of inputs in parallel brings additional challenges in synchronizing the output of the different
sensors.
3. Concluding remarks
Whereas neuro-science techniques are being used for a number of tasks connected to IS usage and
programming, the application in connection to the use of visual conceptual models has so far been
limited. We have in this novel direction talk given an overview of a multi-modal approach for capturing
neuro-scientific data in connection to conceptual modeling, which we hope will bring up ideas in the
conference for how to bring this area of research forward.
4. References
[1] R.Batista Duarte, D. Silva da Silveira, V. de Albuquerque Brito, C. S. Lopes, A systematic
literature review on the usage of eye-tracking in understanding process models, Business Process
Management Journal, Vol. 27 No. 1, (2021) 346-367.
[2] S. Doukakis, Exploring brain activity and transforming knowledge in visual and textual
programming using neuroeducation approaches AIMS Neurosci., 6, pp. (2019)
[3] M. Giannakos, D. Spikol, D. Di Mitri, K. Sharma, X. Ochoa, R. Hammad (Eds.) The Multimodal
Learning Analytics Handbook, Springer, 2022
[4] J. Krogstie, Quality in Business Process Modeling, Springer. 2016
[5] J. Pinggera, M. Neurauter, S. Zugal, M. Martini, M. Furtner, P. Sachse, D. Schnitzer, Fixation
patterns during process model creation: initial steps toward neuro-adaptive process modeling
environments. In: Proceedings of the 49th Hawaii International Conference on System Sciences
(HICSS), 1016, pp. 600–609.
[6] H. Ritchi, M. Jans, J. Mendling, H. A. Reijers, The Influence of Business Process Representation
on Performance of Different Task Types. Journal of Information Systems 1 March; 34 (1) (2020)
167–194
[7] M. Schrepfer, J. Wolf, J. Mendling, H. A. Reijers, The Impact of Secondary Notation on Process
Model Understanding. In: Persson, A., Stirna, J. (eds) The Practice of Enterprise Modeling. PoEM
2009. Lecture Notes in Business Information Processing, vol 39. Springer, Berlin, Heidelberg.
2009.
[8] K. Sharma, S. Papavlasopoulou, M. Giannakos, Joint Emotional State of Children and Perceived
Collaborative Experience in Coding Activities. In Proceedings of the 18th ACM International
Conference on Interaction Design and Children. ACM 2019.
[9] J. Siegmund, C. Kästner, S. Apel, C. Parnin, A. Bethmann, T. Leich, G. Saake, A. Brechmann,
Understanding understanding source code with functional magnetic resonance imaging. In:
Proceedings of the 36th International Conference on Software Engineering - ICSE 2014. pp. 378–
389
[10] J. Siegmund, N. Peitek, C. Parnin, S. Apel, J. Hofmeister, C. Kästner, A. Begel, A. Bethmann, A.
Brechmann, Measuring neural efficiency of program comprehension. In: Proceedings of the 2017
11th Joint Meeting on Foundations of Software Engineering - ESEC/FSE 2017. , 2017,pp. 140–
150
[11] Task Load Index https://humansystems.arc.nasa.gov/groups/TLX/ Last accessed 16/3-2023
[12] B. Weber, T. Fischer, R. Riedl, Brain and autonomic nervous system activity measurement in
software engineering: A systematic literature review, Journal of Systems and Software, Volume
178 (2021)
[13] M. Winter, H. Neumann, R. Pryss, T. Probst, M. Reichert, Defining gaze patterns for process
model literacy – Exploring visual routines in process models with diverse mappings Expert
Systems with Applications, Volume 213, (2023)
[14] J. Xiong, M. Zuo, What does existing NeuroIS research focus on? Information Systems, Volume
89 (2020)
[15] M. Zimoch, T. Mohring, R. Pryss, T. Probst, W. Schlee, M. Reichert, Using Insights from
Cognitive Neuroscience to Investigate the Effects of Event-Driven Process Chains on Process
Model Comprehension. In: Teniente, E., Weidlich, M. (eds) Business Process Management
Workshops. BPM 2017. Lecture Notes in Business Information Processing, vol 308. Springer,
2017