=Paper=
{{Paper
|id=Vol-3139/paper10
|storemode=property
|title=Sensemaking in Multi-artefact Information Tasks
|pdfUrl=https://ceur-ws.org/Vol-3139/paper10.pdf
|volume=Vol-3139
|authors=Tianwa Chen
|dblpUrl=https://dblp.org/rec/conf/caise/Chen22
}}
==Sensemaking in Multi-artefact Information Tasks==
https://ceur-ws.org/Vol-3139/paper10.pdf
Sensemaking in Multi-artefact Information Tasks
Tianwa Chen1
1
School of Information Technology and Electrical Engineering, The University of Queensland, Brisbane, Australia
Abstract
Confronted with information silos and a growing volume of data in an increasingly interconnected data-
driven world, knowledge workers, including technical and business users, often have to navigate multiple
information artefacts to complete their tasks. These artefacts dispersed across various representational
formats, and various information systems, can lead to overlapping, redundant or even conflicting
information and inefficiency in information retrieval and knowledge workers’ understanding. Despite
a growing market of tools, there is a lack of understanding in the current body of knowledge of how
knowledge workers make sense of the multi-artefact information tasks and through what strategies.
Motivated by the human-centric nature of the problem, this PhD project employs experiments, both in
lab studies and on crowdsourcing platforms, and uses a number of behavioral and performance measures
to unpack the cognitive demands on knowledge workers as they make sense of dual artefact tasks and
multi-artefact tasks respectively. This project aims to propose an integrative model of sensemaking and
cognitive processing in multi-artefact information tasks. The findings contribute to a better understanding
of the sensemaking processes in various settings, inform modeling practice, and design supporting tools.
Keywords
Sensemaking, Business process modeling, Data curation, Data quality
1. Introduction
With the widespread problem of information silos and an increase in data accessibility, knowl-
edge workers, including technical and business users, often rely on multiple information artefacts
across different systems to complete their tasks. According to IDC [1], a typical knowledge
worker can spend 36% of the daily time searching for and consolidating information from
multiple artefacts, but workers can find the required information only 56% of the time. 61% of
knowledge workers regularly access four or more different artefacts to retrieve the information
they need for their work, and 15% access 11 or more. These artefacts dispersed across various
representational formats, and various information systems, can lead to overlapping, redundant
or even conflicting information and inefficiency in information retrieval and knowledge workers’
understanding.
In practice, given the process can be more diverse and exploratory when knowledge workers
navigate through multi-artefact information tasks, there has been a strong response from the
market with a plethora of tools to support the ‘human-in-the-loop’ [2]. However, despite there
being an increasing focus for researchers to study the behaviour of knowledge workers in many
Proceedings of the Doctoral Consortium Papers Presented at the 34th International Conference on Advanced Information
Systems Engineering (CAiSE 2022), June 06–10, 2022, Leuven, Belgium
$ tianwa.chen@uq.edu.au (T. Chen)
0000-0002-5135-0313 (T. Chen)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
contexts [3, 4], there has been little focus on the process of knowledge workers making sense
of these multi-artefact information tasks. The current body of knowledge does not adequately
explain knowledge workers’ sensemaking behaviours and strategies when interacting with
these tasks.
To explore this problem, we undertake exploratory studies to investigate knowledge workers’
behaviour in two settings that offer dual artefact and multi-artefact tasks respectively. For
the setting of dual artefact tasks, in the context of business process management systems
and business rule management systems, two commonly used artefacts are business process
models and business rule repositories. When presented separately, these two artefacts are
known to cause a lack of shared understanding,and conflicts and redundancies that can lead to
inefficiencies and even compliance breaches [5]. Although a number of integrated modeling
approaches for business processes and rules have been proposed, there is limited knowledge on
how these approaches affect worker behavior and task performance.
As for the setting of multi-artefact tasks, there is increasing evidence that knowledge workers,
including data scientists, engineers and analysts, can spend in excess of 80 percent of their
time and effort engaged in the data curation process in a typical data science project [6]. These
cost-intensive processes constitute a number of artefact tasks and are considered a drain on
analytic functions within organizations. Due to the inherent complexity of these tasks, the
bulk of data curation tasks still cannot feasibly and efficiently be addressed by machine-based
algorithms [7] without human intervention (e.g., manual inspection) [3]. Moreover, the existing
tools that support data curation tasks are often domain-focused and challenging to use in
coordination with other program functionalities. Therefore, given the increasing demand for a
more cost-efficient data curation process, researchers have started to look at how knowledge
workers engage with data (e.g., [3]). However, there is a paucity of research focusing on how
knowledge workers interact with various artefact information tasks and what processes they
follow while carrying out data curation activities.
Accordingly, motivated by the human-centric nature of the problem, this PhD project employs
exploratory studies to investigate the behavior of knowledge workers engaging with various
artefact information tasks in the context of a dual artefact information tasks setting and also in
multi-artefact information tasks settings. This project aims to propose an integrative model of
sensemaking and cognitive processing in multi-artefact information tasks. We approach the
design of the research through a sensemaking lens and consider foundational sensemaking
constructs of information foraging and information processing.
2. Research Goal
The project is divided into three studies, including experiments in controlled lab studies and
crowdsourcing platforms to understand the cognitive demands on knowledge workers as
they make sense of multi-artefact information tasks. It uses a number of behavioral and
performance measures through the use of eye-tracking and electroencephalography (EEG)
devices in controlled lab experiments.
The first study aims to investigate knowledge workers’ behavior in dual artefact tasks when
the form of integrated representation of the artefacts (namely business process models and
business rules) and task complexity changes.
The second study aims to understand how knowledge workers engage with multi-artefact
tasks in the data curation process. We will first investigate the data curation process specifically
related to data quality detection and how to build repeatable and efficient data curation processes
harnessing the collective intelligence of a group of knowledge workers.
The third study aims to propose an integrative model of sensemaking and cognitive processing
in multi-artefact information tasks by consolidating the research results learnt from the lab
studies and existing frameworks and theories in sensemaking and cognition processing. In
addition, we will test the research model by collecting empirical data from a crowdsourcing
platform.
3. Related Work
3.1. Sensemaking
The sensemaking methodology was introduced by Dervin in 1972 to design a human communi-
cations system, which was later developed as the model of the sensemaking triangle representing
how a person makes sense of the situation through a space-time context [8]. Russell et al.’s
“learning loop complex” [9] first proposed the cost structure of the sensemaking model, which
describes the process people use to understand and encode data to answer tasks-specific ques-
tions. Since these seminal works, literature in various domains has contributed to theories and
models of sensemaking.
More recently, there has been an increased focus on understanding how sensemaking operates
in the era of increasingly complex information artefacts [10]. For instance, researchers have used
a sensemaking perspective to understand how individuals make sense of the fairness assessment
system in ML [11], reusing knowledge [12], debugging strategies [13] and supporting knowledge
acceleration for programming [14].
Cognitive constructs of attention and memory have a natural and strong affinity to the two
phases in sensemaking models. Cognitive load theory [15, 16] provides proven mechanisms
through which these constructs can be operationalized. For example, attention and search
behaviour has been measured through eye-tracking devices, which can capture data on visual
scanning (eye movement) and attention (eye fixations) [17]. This data, in turn, can be used for
various behavioural measurements, such as cognitive load, visual association, visual cognition
efficiency, and intensity [18].
While there is a long history of the use of eye-tracking technology in medical and psychology
studies [19], the use of it in the context of data work with human-machine teaming is relatively
recent. However, it holds great promise for a deeper understanding of user behaviour in complex
tasks. To our best knowledge, existing sensemaking studies are focused on qualitative or
perceptionary measures with limited use of behavioural and performance measures. Hence, we
considered the use of eye-tracking devices in a controlled experiment as a novel and objective
means to capture and expose sensemaking behaviours and the interactive process of how
knowledge workers explore multi-artefact tasks in different settings.
3.2. Dual Artefact Information Tasks in the Case of Business Process Models
and Business Rules
In dual artefact tasks setting, our study considers the specific context of business process and
business rule modeling – two complementary approaches for modeling business activities,
which have multiple integration methods [20] to improve their individual representational
capacity. In summary, the integration methods can be categorized into three approaches with
distinct format and construction, namely: text annotation, diagrammatic integration, and link
integration [21]. Text annotation and link integration both use a textual expression to describe
the business rules and connect them with the corresponding section of the process model.
With link integration, visual links can explicitly connect corresponding rules with the relevant
process section. Diagrammatic integration relies on graphical process model construction, such
as sequence flows and gateways, to represent business rules in the process model. Each of these
methods has strengths and weaknesses, as summarized in [5], and thus a potential impact on a
knowledge worker’s understanding of a process.
3.3. Multi-artefact Information Tasks in Data Curation
The importance and scope of data curation have increased multi-fold in the era of big data, due
to the prevalence of external and repurposed data in data science projects. A primary reason
for data curation is the large proportion of externally acquired datasets with different quality
levels. In fact, even internal data may have to be repurposed [22] to meet the specific needs of
a certain data science project. In either case, the data curation process constitutes a number
of multi-artefact tasks, which may include selection, classification, transformation, filtering,
imputation, integration/fusion, or validation [23].
Currently, three main approaches are evident in the context of data curation, namely: ad-
hoc/manual, automated, and crowd-sourced approaches. The manual approach is the most
common approach [23, 24]. However, data quality issues constitute a major challenge for
knowledge workers using a manual approach as it is likely that multiple data quality issues
exist in large datasets, e.g. completeness, accuracy, and consistency [3, 25].
To study knowledge workers, recent research outlined the work cycle of data scientists,
ranging from discovery to design [3]. We note that utilizing a crowd-sourcing approach for
building data curation processes from multiple crowd-sourced tasks is currently under-studied
and a key objective.
4. Study 1 - Sensemaking in Dual Artefact Tasks – The Case of
Business Process Models and Business Rules
In this study, we investigate how user behavior occurs in dual artefact tasks when the form of
integrated representation of the artefacts (namely business process models and business rules)
and task complexity changes. Using a sensemaking lens in our study, we can delineate the
behavior between developing model understanding and task accomplishment.
4.1. Study Design
We use an experimental research design. In line with sensemaking foundations, we segment
the experiment into two phases, namely a searching and encoding phase (we term this as the
understanding phase) and a task specific information processing phase (termed the answering
phase). The understanding phase commences when the participant first fixates on the experiment
screen, and the answering phase commences when the participant starts to type the answer in
the question area for the first time (see Fig. 1). Due to space limitations, the complete experiment
instruments are available for download 1 .
Figure 1: Visual experiment design [26]. The divided areas of interests (AOIs) with names for analysis
purposes are not displayed to the participants.
The experiment data consists of a pre-experiment questionnaire, eye tracking log data, and
task performance data. The eye tracking data was collected through a Tobii Pro TX300 eye
tracker2 , which captures data on fixations, gaze, saccades, etc., with timestamps. To capture
sensemaking behavior, we used measurements related to fixation durations and frequencies,
measurements related to AOI specific fixations, and transitions between AOIs.
The experiment instruments included a tutorial, the treatments and a questionnaire. Each
group of participants was first provided with a BPMN tutorial and was then offered a model
using one of the three different rule integration approaches. In the treatment, we used the three
integration approaches (one per each treatment group). The scenario of the model and rules
originated from a travel booking diagram included in OMG’s BPMN 2.0 documentation3 . We
ensured, through multiple revisions, that we created informationally equivalent models for all
three integration approaches, and all confounding factors were constant, including the same
eye-tracking lab equipment and tutorial content. We did not limit the experiment duration nor
a word count limit on participants’ answers. The model was adjusted to ensure consistency
1
The experiment materials can be downloaded from bit.ly/3N5Kr6O
2
For more specifications of the eye tracker, please visit https://www.tobiipro.com/product-listing/tobii-pro-tx300/
3
Model originated from OMG’s BPMN 2.0 examples can be viewed in http://www.omg.org/cgi-bin/doc?dtc/10-
06-02
of format for each of the integration approaches, while providing some diversity in terms
of constructs and coverage to gain further insights into the relationship between integration
approaches and task complexity.
4.2. Current Progress
More details of the current results can be found in the publication [26]. Our results show
that link representation shows better task performance in terms of accuracy and efficiency,
especially as task complexity increases. Additionally, our results provide some evidence that
diagrammatic integration has better task performance on local questions in terms of accuracy,
but also requires the most effort in the initial information foraging (understanding) phase.
The findings from this study also form the basis of our investigation for the next step. We
will use complementary approaches such as cued retrospective ‘thinking-out-loud’ [27] and
biosensors (e.g. electroencephalography, captured by Emotive4 ) to provide further explanations
on the sensemaking behavior and cognition process. We also consider the limitations of the
current research, where we only included the basic constructs in business process models,
whereas advanced loop and nesting structures may introduce further complexities in sensemak-
ing. Therefore, we will also analyze the change in knowledge workers’ behavior over longer
tasks with more variability in task complexity to help further reveal insights into sensemaking,
and this may especially be valuable for training and work allocation purposes.
5. Study 2 - Sensemaking in Multi-artefact Information Tasks –
The Case of Data Curation
As the first step, we aim to understand how knowledge workers engage with multi-artefact
tasks in data curation specifically related to data quality detection.
5.1. Study Design
To capture knowledge worker sensemaking behaviours while discovering data quality issues,
we used an experimental design method in a lab study with purpose-built experiment platforms
that mimic typical data exploration tools. The lab setting [25] enabled us to use advanced
tracking devices (e.g., eye-trackers and activity loggers) to capture the interaction behaviors.
Our interface design is typical of several existing data exploration platforms that provide
UI areas with a similar arrangement, see e.g., Talend Cloud API (jupyter.org), RapidMiner
(rapidminer.com), or PowerBI (powerbi.microsoft.com). The UI of our data curation platform
has three main panels: the DataOps area as the internal functions on the left, the working
console area in the middle, and the data view and toolkit area to view and record data quality
annotations on the right (see Fig. 2). We custom built two experiment platforms, with one
requiring manual coding to undertake data quality discovery and the other offering built-in
functions. On both experiment platforms, we kept all other variables constant and provided
4
For more information about Emotiv, please see https://www.emotiv.com/
equivalent information with the same interface design, including the same dataset and a set of
pre-defined functions.
(a) Experiment platform with coding [28].
Figure 2: User interface of the experimental platforms.
To provide internal data curation resources, we pre-define 21 DataOps, ranging from import-
ing essential libraries to complex Boolean operations involving regular expressions [28]. This
set of DataOps is sufficient to complete all tasks in our experiment (i.e., participants do not
necessarily need to refer to external materials). The dataset includes 13,000 records and four
columns (ID, name, contact number and join date). We chose five most recognised and common
types of data quality issues [28] and injected them into the dataset with the help of Parallel
Data Generation Framework [29] to provide the ground truth. The size of the data and injected
number of errors removed the option of manual annotation.
The participants were required to complete the task of identifying and annotating the data
quality issues. They are only allowed to use the given browser throughout the experiment.
The experiment commences with a pre-experiment survey, followed by a tutorial outlining
definitions and examples of data quality issues and a practice example, and then they start the
formal experiment whenever they feel ready. At the end of the experiment, the participants are
asked to complete a post-experiment survey. The surveys based on [30] captured participant
perceptions on the experiment tasks and helped ensure internal validity.
5.2. Current Progress
More details of the current results can be found in the publications [28, 31, 32]. Our findings
show that the approaches taken by the knowledge workers participating in our study were
often diverse and complementary in that they were able to identify different data quality issues
with different levels of effectiveness and robustness. This bears implications for automatically
creating aggregated data curation process through crowd intelligence.
However, the current work is not without limitations as it was based on a lab experiment,
and we only focused on detecting data quality issues. Therefore, in the next step, we will
conduct experiments with real crowd workers to fully understand the sensemaking process
in the complex artefact tasks of data curation, and build effective, robust, and repeatable data
curation processes by learning from a crowd of knowledge workers.
6. Study 3 - Sensemaking and Cognitive Processing Model in
Multi-Artefact Information Tasks
The work is still in the early stages. Based on the existing frameworks and theories in sensemak-
ing and cognition processing, we plan to consolidate all research findings we found in studies 1
and 2 to propose a integrative sensemaking and cognitive processing model in multi-artefact
information tasks. We will test the proposed research model and hypotheses using empirical
data collected from a crowdsourcing platform.
7. Expected Contributions
This research project allows us to understand the cognitive demands on knowledge workers as
they make sense of multi-artefact information tasks. Expected contributions include bridging
the gap of the current limitation of understanding the sensemaking process of knowledge
workers in multi-artefact information tasks, contributing to sensemaking theory, informing
modelling practice, providing guidelines on training to knowledge workers, and the design of
supporting tools and tasks.
Acknowledgments
This research is supported by UQ RTP reserach scholarship. I would like to thank Prof. Shazia
Sadiq, Prof. Marta Indulska, and A/Prof. Gianluca Demartini for supervising this PhD project.
References
[1] D. Schubmehl, D. Vesset, The knowledge quotient: Unlocking the hidden value of infor-
mation using search and content analytics, White paper, IDC (2014).
[2] N. Sambasivan, S. Kapania, H. Highfill, D. Akrong, P. Paritosh, L. M. Aroyo, “everyone
wants to do the model work, not the data work”: Data cascades in high-stakes ai, in:
Proceedings of the 2021 CHI, 2021, pp. 1–15.
[3] M. Muller, I. Lange, D. Wang, D. Piorkowski, J. Tsay, Q. V. Liao, C. Dugan, T. Erickson,
How data science workers work with data: Discovery, capture, curation, design, creation,
in: Proceedings of the 2019 CHI Conference, 2019.
[4] D. J. Piorkowski, S. D. Fleming, I. Kwan, M. M. Burnett, C. Scaffidi, R. K. Bellamy, J. Jor-
dahl, The whats and hows of programmers’ foraging diets, in: Proceedings of the CHI
Conference, 2013, pp. 3063–3072.
[5] W. Wang, M. Indulska, S. W. Sadiq, Cognitive efforts in using integrated models of business
processes and rules., in: CAiSE Forum, 2016, pp. 33–40.
[6] D. Patil, Data Jujitsu, O’Reilly Media, Inc., 2012.
[7] S. Sadiq, T. Dasu, X. L. Dong, J. Freire, I. F. Ilyas, S. Link, M. J. Miller, F. Naumann, X. Zhou,
D. Srivastava, Data quality: The role of empiricism 46 (2018) 35–43.
[8] B. Dervin, A theoretic perspective and research approach for generating research helpful to
communication practice, Sense-making methodology reader: selected writings of Brenda
Dervin (2003) 251–268.
[9] D. M. Russell, M. J. Stefik, P. Pirolli, S. K. Card, The cost structure of sensemaking, in:
Proceedings of the INTERACT’93 and CHI’93, 1993, pp. 269–276.
[10] D. M. Russell, G. Convertino, A. Kittur, P. Pirolli, E. A. Watkins, Sensemaking in a senseless
world: 2018 workshop abstract, in: Extended Abstracts of the 2018 CHI Conference on
Human Factors in Computing Systems, 2018, pp. 1–7.
[11] Z. Gu, J. N. Yan, J. M. Rzeszotarski, Understanding user sensemaking in machine learning
fairness assessment systems, in: Proceedings of the Web Conference, 2021, pp. 658–668.
[12] M. X. Liu, A. Kittur, B. A. Myers, To reuse or not to reuse? a framework and system for
evaluating summarized knowledge, Proceedings of the ACM on HCI 5 (2021) 1–35.
[13] V. Grigoreanu, M. Burnett, S. Wiedenbeck, J. Cao, K. Rector, I. Kwan, End-user debug-
ging strategies: A sensemaking perspective, ACM Transactions on Computer-Human
Interaction (TOCHI) 19 (2012) 1–28.
[14] M. X. Liu, S. Burley, E. Deng, A. Zhou, A. Kittur, B. A. Myers, Supporting knowledge
acceleration for programming from a sensemaking perspective, in: Sensemaking Workshop
at CHI Conference on Human Factors in Computing Systems, 2018.
[15] F. Chen, J. Zhou, Y. Wang, K. Yu, S. Z. Arshad, A. Khawaji, D. Conway, Robust multimodal
cognitive load measurement, Springer, 2016.
[16] J. Sweller, P. Ayres, S. Kalyuga, Measuring cognitive load, in: Cognitive load theory,
Springer, 2011, pp. 71–85.
[17] A. T. Duchowski, Gaze-based interaction: A 30 year retrospective, Computers & Graphics
73 (2018) 59–69.
[18] K. Rayner, Eye movements in reading and information processing: 20 years of research.,
Psychological bulletin 124 (1998) 372.
[19] M. A. Just, P. A. Carpenter, Eye fixations and cognitive processes, Cognitive psychology 8
(1976) 441–480.
[20] G. Knolmayer, R. Endl, M. Pfahrer, Modeling processes and workflows by business rules,
in: Business Process Management, Springer, 2000, pp. 16–29.
[21] T. Chen, W. Wang, M. Indulska, S. Sadiq, Business process and rule integration approaches-
an empirical analysis, in: International Conference on Business Process Management,
Springer, 2018, pp. 37–52.
[22] R. Zhang, M. Indulska, S. Sadiq, Discovering data quality problems, Business & Information
Systems Engineering 61 (2019) 575–593.
[23] T. Hey, A. Trefethen, The data deluge: An e-science perspective, Grid computing: Making
the global infrastructure a reality (2003) 809–824.
[24] E. Rahm, H. H. Do, Data cleaning: Problems and current approaches, IEEE Data Eng. Bull.
23 (2000) 3–13.
[25] C. Sutton, T. Hobson, J. Geddes, R. Caruana, Data diff: Interpretable, executable summaries
of changes in distributions for data wrangling, in: Proceedings of the 24th ACM SIGKDD
Conference, 2018, pp. 2279–2288.
[26] T. Chen, S. Sadiq, M. Indulska, Sensemaking in dual artefact tasks–the case of business
process models and business rules, in: International Conference on Conceptual Modeling,
Springer, 2020, pp. 105–118.
[27] T. Van Gog, F. Paas, J. J. Van Merriënboer, P. Witte, Uncovering the problem-solving
process: Cued retrospective reporting versus concurrent and retrospective reporting.,
Journal of Experimental Psychology: Applied 11 (2005) 237.
[28] T. Chen, L. Han, G. Demartini, M. Indulska, S. Sadiq, Building data curation processes
with crowd intelligence, in: International Conference on Advanced Information Systems
Engineering, Springer, 2020, pp. 29–42.
[29] Y. Tay, Data generation for application-specific benchmarking, Proceedings of the VLDB
Endowment 4 (2011) 1470–1473.
[30] S. G. Hart, Nasa-task load index (nasa-tlx); 20 years later, in: Proceedings of the human
factors and ergonomics society annual meeting, volume 50, 2006, pp. 904–908.
[31] L. Han, T. Chen, G. Demartini, M. Indulska, S. Sadiq, On understanding data worker
interaction behaviors, in: Proceedings of the 43rd International ACM SIGIR Conference
on Research and Development in Information Retrieval, 2020, pp. 269–278.
[32] S. Yu, T. Chen, L. Han, G. Demartini, S. Sadiq, Dataops-4g: On supporting generalists in
data quality discovery, IEEE Transactions on Knowledge and Data Engineering (2022).