With the widespread problem of information silos and an increase in data accessibility, knowledge workers, including technical and business users, often rely on multiple information artefacts across different systems to complete their tasks. According to IDC [1], a typical knowledge worker can spend 36% of the daily time searching for and consolidating information from multiple artefacts, but workers can find the required information only 56% of the time. 61% of knowledge workers regularly access four or more different artefacts to retrieve the information they need for their work, and 15% access 11 or more. These artefacts dispersed across various representational formats, and various information systems, can lead to overlapping, redundant or even conflicting information and inefficiency in information retrieval and knowledge workers' understanding.
In practice, given the process can be more diverse and exploratory when knowledge workers navigate through multi-artefact information tasks, there has been a strong response from the market with a plethora of tools to support the 'human-in-the-loop' [2]. However, despite there being an increasing focus for researchers to study the behaviour of knowledge workers in many contexts [3,4], there has been little focus on the process of knowledge workers making sense of these multi-artefact information tasks. The current body of knowledge does not adequately explain knowledge workers' sensemaking behaviours and strategies when interacting with these tasks.
To explore this problem, we undertake exploratory studies to investigate knowledge workers' behaviour in two settings that offer dual artefact and multi-artefact tasks respectively. For the setting of dual artefact tasks, in the context of business process management systems and business rule management systems, two commonly used artefacts are business process models and business rule repositories. When presented separately, these two artefacts are known to cause a lack of shared understanding,and conflicts and redundancies that can lead to inefficiencies and even compliance breaches [5]. Although a number of integrated modeling approaches for business processes and rules have been proposed, there is limited knowledge on how these approaches affect worker behavior and task performance.
As for the setting of multi-artefact tasks, there is increasing evidence that knowledge workers, including data scientists, engineers and analysts, can spend in excess of 80 percent of their time and effort engaged in the data curation process in a typical data science project [6]. These cost-intensive processes constitute a number of artefact tasks and are considered a drain on analytic functions within organizations. Due to the inherent complexity of these tasks, the bulk of data curation tasks still cannot feasibly and efficiently be addressed by machine-based algorithms [7] without human intervention (e.g., manual inspection) [3]. Moreover, the existing tools that support data curation tasks are often domain-focused and challenging to use in coordination with other program functionalities. Therefore, given the increasing demand for a more cost-efficient data curation process, researchers have started to look at how knowledge workers engage with data (e.g., [3]). However, there is a paucity of research focusing on how knowledge workers interact with various artefact information tasks and what processes they follow while carrying out data curation activities.
Accordingly, motivated by the human-centric nature of the problem, this PhD project employs exploratory studies to investigate the behavior of knowledge workers engaging with various artefact information tasks in the context of a dual artefact information tasks setting and also in multi-artefact information tasks settings. This project aims to propose an integrative model of sensemaking and cognitive processing in multi-artefact information tasks. We approach the design of the research through a sensemaking lens and consider foundational sensemaking constructs of information foraging and information processing.
The project is divided into three studies, including experiments in controlled lab studies and crowdsourcing platforms to understand the cognitive demands on knowledge workers as they make sense of multi-artefact information tasks. It uses a number of behavioral and performance measures through the use of eye-tracking and electroencephalography (EEG) devices in controlled lab experiments.
The first study aims to investigate knowledge workers' behavior in dual artefact tasks when the form of integrated representation of the artefacts (namely business process models and business rules) and task complexity changes.
The second study aims to understand how knowledge workers engage with multi-artefact tasks in the data curation process. We will first investigate the data curation process specifically related to data quality detection and how to build repeatable and efficient data curation processes harnessing the collective intelligence of a group of knowledge workers.
The third study aims to propose an integrative model of sensemaking and cognitive processing in multi-artefact information tasks by consolidating the research results learnt from the lab studies and existing frameworks and theories in sensemaking and cognition processing. In addition, we will test the research model by collecting empirical data from a crowdsourcing platform.
The sensemaking methodology was introduced by Dervin in 1972 to design a human communications system, which was later developed as the model of the sensemaking triangle representing how a person makes sense of the situation through a space-time context [8]. Russell et al.'s "learning loop complex" [9] first proposed the cost structure of the sensemaking model, which describes the process people use to understand and encode data to answer tasks-specific questions. Since these seminal works, literature in various domains has contributed to theories and models of sensemaking.
More recently, there has been an increased focus on understanding how sensemaking operates in the era of increasingly complex information artefacts [10]. For instance, researchers have used a sensemaking perspective to understand how individuals make sense of the fairness assessment system in ML [11], reusing knowledge [12], debugging strategies [13] and supporting knowledge acceleration for programming [14].
Cognitive constructs of attention and memory have a natural and strong affinity to the two phases in sensemaking models. Cognitive load theory [15,16] provides proven mechanisms through which these constructs can be operationalized. For example, attention and search behaviour has been measured through eye-tracking devices, which can capture data on visual scanning (eye movement) and attention (eye fixations) [17]. This data, in turn, can be used for various behavioural measurements, such as cognitive load, visual association, visual cognition efficiency, and intensity [18].
While there is a long history of the use of eye-tracking technology in medical and psychology studies [19], the use of it in the context of data work with human-machine teaming is relatively recent. However, it holds great promise for a deeper understanding of user behaviour in complex tasks. To our best knowledge, existing sensemaking studies are focused on qualitative or perceptionary measures with limited use of behavioural and performance measures. Hence, we considered the use of eye-tracking devices in a controlled experiment as a novel and objective means to capture and expose sensemaking behaviours and the interactive process of how knowledge workers explore multi-artefact tasks in different settings.
In dual artefact tasks setting, our study considers the specific context of business process and business rule modeling -two complementary approaches for modeling business activities, which have multiple integration methods [20] to improve their individual representational capacity. In summary, the integration methods can be categorized into three approaches with distinct format and construction, namely: text annotation, diagrammatic integration, and link integration [21]. Text annotation and link integration both use a textual expression to describe the business rules and connect them with the corresponding section of the process model. With link integration, visual links can explicitly connect corresponding rules with the relevant process section. Diagrammatic integration relies on graphical process model construction, such as sequence flows and gateways, to represent business rules in the process model. Each of these methods has strengths and weaknesses, as summarized in [5], and thus a potential impact on a knowledge worker's understanding of a process.
The importance and scope of data curation have increased multi-fold in the era of big data, due to the prevalence of external and repurposed data in data science projects. A primary reason for data curation is the large proportion of externally acquired datasets with different quality levels. In fact, even internal data may have to be repurposed [22] to meet the specific needs of a certain data science project. In either case, the data curation process constitutes a number of multi-artefact tasks, which may include selection, classification, transformation, filtering, imputation, integration/fusion, or validation [23]. Currently, three main approaches are evident in the context of data curation, namely: adhoc/manual, automated, and crowd-sourced approaches. The manual approach is the most common approach [23,24]. However, data quality issues constitute a major challenge for knowledge workers using a manual approach as it is likely that multiple data quality issues exist in large datasets, e.g. completeness, accuracy, and consistency [3,25].
To study knowledge workers, recent research outlined the work cycle of data scientists, ranging from discovery to design [3]. We note that utilizing a crowd-sourcing approach for building data curation processes from multiple crowd-sourced tasks is currently under-studied and a key objective.
In this study, we investigate how user behavior occurs in dual artefact tasks when the form of integrated representation of the artefacts (namely business process models and business rules) and task complexity changes. Using a sensemaking lens in our study, we can delineate the behavior between developing model understanding and task accomplishment.
We use an experimental research design. In line with sensemaking foundations, we segment the experiment into two phases, namely a searching and encoding phase (we term this as the understanding phase) and a task specific information processing phase (termed the answering phase). The understanding phase commences when the participant first fixates on the experiment screen, and the answering phase commences when the participant starts to type the answer in the question area for the first time (see Fig. 1). Due to space limitations, the complete experiment instruments are available for download 1 .
Figure 1: Visual experiment design [26]. The divided areas of interests (AOIs) with names for analysis purposes are not displayed to the participants.
The experiment data consists of a pre-experiment questionnaire, eye tracking log data, and task performance data. The eye tracking data was collected through a Tobii Pro TX300 eye tracker 2 , which captures data on fixations, gaze, saccades, etc., with timestamps. To capture sensemaking behavior, we used measurements related to fixation durations and frequencies, measurements related to AOI specific fixations, and transitions between AOIs.
The experiment instruments included a tutorial, the treatments and a questionnaire. Each group of participants was first provided with a BPMN tutorial and was then offered a model using one of the three different rule integration approaches. In the treatment, we used the three integration approaches (one per each treatment group). The scenario of the model and rules originated from a travel booking diagram included in OMG's BPMN 2.0 documentation 3 . We ensured, through multiple revisions, that we created informationally equivalent models for all three integration approaches, and all confounding factors were constant, including the same eye-tracking lab equipment and tutorial content. We did not limit the experiment duration nor a word count limit on participants' answers. The model was adjusted to ensure consistency of format for each of the integration approaches, while providing some diversity in terms of constructs and coverage to gain further insights into the relationship between integration approaches and task complexity.
More details of the current results can be found in the publication [26]. Our results show that link representation shows better task performance in terms of accuracy and efficiency, especially as task complexity increases. Additionally, our results provide some evidence that diagrammatic integration has better task performance on local questions in terms of accuracy, but also requires the most effort in the initial information foraging (understanding) phase.
The findings from this study also form the basis of our investigation for the next step. We will use complementary approaches such as cued retrospective 'thinking-out-loud' [27] and biosensors (e.g. electroencephalography, captured by Emotive4 ) to provide further explanations on the sensemaking behavior and cognition process. We also consider the limitations of the current research, where we only included the basic constructs in business process models, whereas advanced loop and nesting structures may introduce further complexities in sensemaking. Therefore, we will also analyze the change in knowledge workers' behavior over longer tasks with more variability in task complexity to help further reveal insights into sensemaking, and this may especially be valuable for training and work allocation purposes.
As the first step, we aim to understand how knowledge workers engage with multi-artefact tasks in data curation specifically related to data quality detection.
To capture knowledge worker sensemaking behaviours while discovering data quality issues, we used an experimental design method in a lab study with purpose-built experiment platforms that mimic typical data exploration tools. The lab setting [25] enabled us to use advanced tracking devices (e.g., eye-trackers and activity loggers) to capture the interaction behaviors. Our interface design is typical of several existing data exploration platforms that provide UI areas with a similar arrangement, see e.g., Talend Cloud API (jupyter.org), RapidMiner (rapidminer.com), or PowerBI (powerbi.microsoft.com). The UI of our data curation platform has three main panels: the DataOps area as the internal functions on the left, the working console area in the middle, and the data view and toolkit area to view and record data quality annotations on the right (see Fig. 2). We custom built two experiment platforms, with one requiring manual coding to undertake data quality discovery and the other offering built-in functions. On both experiment platforms, we kept all other variables constant and provided equivalent information with the same interface design, including the same dataset and a set of pre-defined functions.
(a) Experiment platform with coding [28]. To provide internal data curation resources, we pre-define 21 DataOps, ranging from importing essential libraries to complex Boolean operations involving regular expressions [28]. This set of DataOps is sufficient to complete all tasks in our experiment (i.e., participants do not necessarily need to refer to external materials). The dataset includes 13,000 records and four columns (ID, name, contact number and join date). We chose five most recognised and common types of data quality issues [28] and injected them into the dataset with the help of Parallel Data Generation Framework [29] to provide the ground truth. The size of the data and injected number of errors removed the option of manual annotation.
The participants were required to complete the task of identifying and annotating the data quality issues. They are only allowed to use the given browser throughout the experiment. The experiment commences with a pre-experiment survey, followed by a tutorial outlining definitions and examples of data quality issues and a practice example, and then they start the formal experiment whenever they feel ready. At the end of the experiment, the participants are asked to complete a post-experiment survey. The surveys based on [30] captured participant perceptions on the experiment tasks and helped ensure internal validity.
More details of the current results can be found in the publications [28,31,32]. Our findings show that the approaches taken by the knowledge workers participating in our study were often diverse and complementary in that they were able to identify different data quality issues with different levels of effectiveness and robustness. This bears implications for automatically creating aggregated data curation process through crowd intelligence.
However, the current work is not without limitations as it was based on a lab experiment, and we only focused on detecting data quality issues. Therefore, in the next step, we will conduct experiments with real crowd workers to fully understand the sensemaking process in the complex artefact tasks of data curation, and build effective, robust, and repeatable data curation processes by learning from a crowd of knowledge workers.
The work is still in the early stages. Based on the existing frameworks and theories in sensemaking and cognition processing, we plan to consolidate all research findings we found in studies 1 and 2 to propose a integrative sensemaking and cognitive processing model in multi-artefact information tasks. We will test the proposed research model and hypotheses using empirical data collected from a crowdsourcing platform.
This research project allows us to understand the cognitive demands on knowledge workers as they make sense of multi-artefact information tasks. Expected contributions include bridging the gap of the current limitation of understanding the sensemaking process of knowledge workers in multi-artefact information tasks, contributing to sensemaking theory, informing modelling practice, providing guidelines on training to knowledge workers, and the design of supporting tools and tasks.
This research is supported by UQ RTP reserach scholarship. I would like to thank Prof. Shazia Sadiq, Prof. Marta Indulska, and A/Prof. Gianluca Demartini for supervising this PhD project.