Understandability of Business Process Models

Abstract

Diferent factors afect the understandability of business process models, which depends both on the characteristics of the model but also on the model users' knowledge and skills. Researchers have conducted experiments to find the relationships among factors and indicators, collecting data using surveys and quizzes in problem-solving experimental tasks. However, in order to collect data from a critical number and variety of model users, experimental replications can be an expensive approach. This article proposes an understandability measurement approach to collect data through a survey. The proposal is based on existing quality models and instruments, which are analysed to design a minimal instrument that allows jointly collecting information from various understandability factors. The proposed measurement approach is part of ongoing research on discovering relationships between the multiple factors and business process model understandability indicators using data analytics and machine learning techniques.

Keywords

• understandability
• instrument
• business process models

1. Introduction

Comprehension is the primary goal of the pragmatic quality dimension in conceptual models [ 1 ]. In particular, the understandability of business process models addresses how easy it is to understand the information contained in a process model [ 2 ]. Understandability indicators of a model can be measured with an objective approach, i.e., by asking a model user about the information represented in the model, or with a subjective approach in terms of perceived understandability [ 3 ].

On the other hand, many factors can afect understandability, which is related to conceptual modelling and the personal characteristics of the model user. The experimental research [ 2, 4, 5 ] of these factors and their efect on understandability indicators have confirmed the need for an empirical approach but also revealed the dificulty of controlling many levels of many variables at the same time. Moreover, since some factors depend on the characteristics of the model users, covering a significant number and diversity of users would require a great number of experimental replications. Our approach is to collect data on understandability factors and indicators through a survey that can be continuously applied to increase the number and variety of model users incrementally. However, carrying on such a long-term efort requires carefully selecting the measurements related to personal factors and the understandability outcomes. In this article, we propose a measurement approach to collect multiple understandability factors and indicators in a minimalist way, aiming for the long-term collection of understandability data.

2. Background and Related Work

In 2020, Dikici et al. [ 3 ] proposed an understandability framework for business process models based on a systematic literature review. The framework groups the understandability factors into process model and personal factors and defines understandability indicators. Process model factors regard to inner characteristics of the model; for instance, a model with more concepts and relationships could be harder to understand than a simpler model. Personal factors regard the background of the model user; for instance, if the problem domain or the modelling notation are known to the model user, it would be easier to extract information about the model. The understandability indicators measure if the model user can extract information from the model and can be objectively measured or self-reported. Figure 1 depicts the framework’s factors and indicators from [ 3 ].

Some of the above factors have been measured in experimental assessments using diferent instruments. For instance, process model factors and measurements for the model graph are proposed by Mendling et al. in [ 6 ], while in [ 7 ] adds the evaluation of the labelling style of the model elements. Sanchez et al. [ 8 ] added the complexity measurement for process models. In [ 9 ], the process diagrams size is measured as the number of elements; however, it opens the question about the efect of the size of the model layout. Regarding the personal factors, Recker et al. [ 10 ] studies the impact of the individual characteristics of model users using demographic surveys. Finally, understandability indicators were also measured by Mendling [ 11 ] through quizzes about the information represented in business process models. The questions are designed ad hoc for each model, requiring that the model user interprets a set of model elements used in combination and follows the process’s flow through decision points.

3. Understandability Measurement Proposal

This proposal aims to support collecting data on the factors and indicators of understandability of business process models from as many model users as possible. To this end, we aim to move from an experimental approach to data collection to a survey approach. The goal is to cover a wide variety of personal profiles that may afect understandability. In a survey context, the set of questions about the model user’s background and profile must be minimal to reduce dropout risk. On the other hand, process models must be characterised in terms of their understandability factors. However, the understandability characteristics of models should be independent of the business process modelling tool since it is impossible to control the respondents’ settings. Finally, it is needed to collect understandability indicators through quizzes.

The proposed understandability measurement approach is presented in Figure 1. As shown, we base the proposal on the quality framework by Dikici et al. [ 3 ], although some factors and indicators are not considered. In the following subsections, we comment on the procedure and measurements for each step.

3.1. Step 1: Initial Survey

This step regards collecting the personal information of the model user. We ask the model users to characterise themselves for each of the personal factors in Figure 1. The following factors are collected using 5-point Likert Scale questions: modelling expertise, knowledge in process modelling notation, cognitive abilities, and domain familiarity. We followed this approach since it was the same adopted in experimental instruments [ 2, 5 ]. On the other hand, the professional background is selected from a list.

The discarded factors for the first step regard the learning style research: ( PF5 - Learning Style and PF6-Learning Motive and PF7 - Learning Strategy). The assumption that people can be grouped into diferent learning style categories has scarce support from objective studies [ 12 ]. These elements are scratched in orange in Figure 1.

3.2. Step 2: Model Review

In this step, the model users have access to review a business process model. The survey can present many business process models to the model user in a sequential manner, as shown in Figure 1. The business models considered in the current version of the instrument were carefully designed to present a combination of diferent values of the process model factors of the framework. The factors taken into account are PMF2-Structural complexity, PMF5-Visual layout, PMF6-Model element labelling and PMF10-Modelling construct type used, that are starred in Figure 1. The combination of diferent levels of these factors generated seven diferent models. However, the number of models could be increased by considering the non-starred factors in the model design, which is future work.

3.3. Step 3: Understandability Assessment

Finally, in Step 3 we measure the understandability indicators. The objective indicators of understandability are measured through a quiz that tests whether the model user can extract information from the model. The type of questions is true or false, and no more than six questions per model are designed. The understandability efectiveness indicator for each reviewed model is calculated as the number of correct responses. To make diferent models comparable, the quizzes for other models must have the same number of questions (four questions by each model) and a similar dificulty level. The response time for each answer is recorded to calculate the understandability task eficiency as the meantime for all the correct answers. The true or false questions as well the efectiveness measurements are based on the works presented in [ 5 ] and [ 11 ]. Since the authors did not report validity threats, we replicated the approach.

To consider the perceived understandability factors, i.e., cognitive load, perceived usefulness, ease of use, and intention to use, it would be necessary to introduce 13 questions, as presented in [ 10 ]. Since we believe this would add an overwhelming load to the model user, we opted not to consider self-reported understandability scores in order to better utilize the subjects’ time to review more models. This decision is shown in the blue striped factors in Figure 1. The ifnal survey instrument and a sample questionnaire are available in an open repository 1, and implemented in the Model Comprehensibility Survey System (MUSS)2.

4. Conclusions and Future Work

In this paper, we presented the process for designing a survey-based measurement approach for understandability. Based on existing understandability quality frameworks and experimental instruments, we designed a three-step process and a 37-questions instrument to collect understandability indicators. We also implemented a support tool for the process that automatically measures the model characteristics. The future work is focused on applying the measurement approach to capture a significant amount of data for its analysis with data analytics and machine learning techniques and on updating the instrument based on the findings. This work has been developed with the financial support of the Spanish State Research Agency and the Generalitat Valenciana under the projects MICIN/AEI/10.13039/501100011033, GV/2021/ 072, and INNEST/2021/57, and co-financed with ERDF and the European Union NextGenerationEU/ PRTR, the National Agency for Research and Development (ANID)/ Scholarship Program/ Doctorado Becas Chile/ 2020-72210494 and Santiago Grisolía fellowship under the project GRISOLIAP/2020/096.

References

[1] O. I. Lindland , G. Sindre , A. Solvberg , Understanding quality in conceptual modeling , IEEE Software ( 1994 ) 42 - 49 .

[2] H. A. Reijers , J. Mendling , A Study Into the Factors That Influence the Understandability of Business Process Models , IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans ( 2011 ) 449 - 462 .

[3] A. Dikici , O. Turetken , O. Demirors , Factors influencing the understandability of process models: A systematic literature review , Information and Software Technology ( 2018 ) 112 - 129 .

[4] R. Gabryelczyk , A. Jurczuk , Does Experience Matter? Factors Afecting the Understandability of the Business Process Modelling Notation , Procedia Engineering 182 ( 2017 ).

[5] J. Mendling , M. Strembeck , J. Recker , Factors of process model comprehension-Findings from a series of experiments, Decision Support Systems ( 2012 ) 195 - 206 .

[6] J. Mendling , Metrics for Process Models: Empirical Foundations of Verification, Error Prediction, and Guidelines for Correctness , Lecture Notes in Business Information Processing , Springer-Verlag, 2008 .

[7] J. Mendling , H. Reijers , J. Recker , Activity labeling in process modeling: Empirical insights and recommendations , Information Systems 35 ( 2010 ) 467 - 482 . Vocabularies, Ontologies and Rules for Enterprise and Business Process Modeling and Management .

[8] L. Sánchez-González , F. García , F. Ruiz , J. Mendling , Quality indicators for business process models from a gateway complexity perspective , Information and Software Technology 54 ( 2012 ) 1159 - 1174 .

[9] H. Störrle , On the impact of layout quality to understanding UML diagrams: Size matters , in: Model-Driven Engineering Languages and Systems , Springer, Cham, 2014 , pp. 518 - 534 .

[10] J. Recker , Continued use of process modeling grammars: the impact of individual diference factors , Eur J Inf Syst 19 ( 2010 ) 76 - 92 .

[11] J. Mendling , J. Recker , H. A. Reijers , H. Leopold , An Empirical Review of the Connection Between Model Viewer Characteristics and the Comprehension of Conceptual Process Models , Information Systems Frontiers 21 ( 2019 ) 1111 - 1135 .

[12] P. A. Kirschner , Stop propagating the learning styles myth , Computers & Education 106 ( 2017 ) 166 - 171 .