Teaching and learning conceptual modeling have been reported to be dificult tasks. In the past few decades, there have been a myriad of studies and resources on how to simplify teaching, diferent teaching methods, and frameworks proposed for conceptual modeling education. Nevertheless, there exist gaps in helping educators construct facile study modules to help activate students toward achieving their learning goals. In order to make the choices on pedagogy, method, and assessment for an educator easier, we present three courses containing a business process management teaching module at diferent levels of mastery (introductory, familiar, and advanced) in blended contexts at a higher education institute. We examine and deliberate on several aspects of the course design. Through correlation analysis, this study explores whether (1) there are any discernible efects of the design on outcomes by analyzing the association between formative and summative scores; (2) the course design employed in all three study modules possesses the necessary flexibility to accommodate various levels of mastery that are intended to be achieved; and (3) the utilization of adaptive release of learning material efectively stimulates student engagement and participation.
CEUR
ceur-ws.org CEUR Workshop Proceedings ICT staf, etc) and purposes in the clarification, redesign or automation of existing processes. Novice designers and business analysts will be placed at an advantage in organizations if they have developed strong modeling skill sets during their university education. However, teaching requirements formalization through CM comes with its own set of challenges[3]. Acquiring proficiency in PM is essential for efectively managing or automating business processes, but learning PM poses challenges due to its inherent complexity, requiring specific cognitive schemata and practical experience [4]. Enhancing the training of process modelers contributes to the acquisition of necessary skills for PM, potentially addressing the prevalent problem of subpar process model quality within organizations [5].
There have been several publications researching CM education and on the usefulness of
feedback in teaching CM in Higher education [6, 7, 8, 4]. But thus far, no standardized approach
has been established for teaching CM or incorporating feedback into the teaching process. While
there are myriad of educational materials available such as MOOCs, textbooks, and traditional
classroom courses for/on teaching CM as listed in [9], there is no one-size-fits-all approach.
Neither is there an understanding of how to support CM learning [10]. When ofering/designing
a course with PM, a teaching team needs to address (
The BPMN module was presented to business program cohorts enrolled in the academic year 2022-2023 at three diferent academic levels: Bridging program (preparatory for an Academic Master for Non-Academic Bachelor), Academic Bachelor , and Academic Master. The cohort following the Bridging program completed a Non-Academic Bachelor. The Academic Bachelor cohort is in their third year of Bachelor in Business Engineering. The Masters’ cohort is following a 1-year MBA. The courses are ICT Management (ICTM) in the Bridging program, Business Information Systems (BIS) in the Bachelor program, and Business Process Modelling and Information Systems Design (BPM & ISD) in the Masters program. All cohorts are considered novice process modelers with none or very little modeling experience.
The details on learning aims, ECTS, and summative evaluation of the three courses ICTM1, BIS2 and BPM&ISD3 can be found in the links below in the footnotes. The three courses have distinct levels of learning goal complexities which we encapsulate pedagogically using Bloom’s Taxonomy[13]. [6] applied the revised Bloom’s Taxonomy of educational objectives[13] for domain modeling, identifying and defining 6 cognitive process levels and 4 knowledge levels with examples in the context of learning CM. In designing the module, we enforce these scafolding levels and apply them to PM. We classified the contents starting from the evaluation (summative assessment:SA), the learning objectives, learning items, formative assessments (FAs) of the BPMN module of each course into diferent Bloom categories as defined in [ 6]. For example, the SA for the ICTM course only contains Multiple Choice questions (MCQs) which classifies it into the Understand, Analyse, and Evaluate levels. For the BIS and BPM&ISD courses, students have to model a given textual description using BPMN, which classifies these into the Create level. Additionally, BPM&ISD students have to solve a real case study. Table 1 lists the course characteristics and how the Bloom’s taxonomy levels map to course components and assessments. After following the courses ICTM, BIS, and BPM&ISD respectively, students are expected to have an introductory, moderate, and expert level of familiarity using BPMN.
At first, the complete BPMN material was designed with some fixed elements (lessons and practice exercise sessions) partitioned into BPMN Basics and Advanced subsections. In lieu of the university’s response to post-pandemic circumstances and culture, where a reduced number of students attended in-person classes but took the exam, most of the learning material was required to be online. Lessons were planned as videos with each video having a central focus on specific BPMN concepts. Interspersed between the 4 BPMN Basics and 7 BPMN advanced videos, the FAs (6 quizzes; 2-8 exercises) were planned to help the student with self-evaluation and self-regulation. These learning items were also ofered through an online component, thus giving autonomy to the learners to pace themselves and to work on it at home in case they cannot come to class.
In-person sessions were set up for solving the exercises. During these in-person sessions, after a quick review of the lectures, a few exercises are solved step-by-step to introduce how a textual description can be interpreted and a process diagram be realized. Then, students are given time to solve exercises by themselves and the instructor and teaching assistant are available to answer any individual dificulties. A few possible solutions, common dificulties, and errors are discussed for each of the exercises at the end. This is done to facilitate discussion and improve on the quality of models. Exercises discussed in the exercise sessions for the three module levels were chosen carefully giving particular attention to the dificulty level of the exercises. In ICTM, more basic exercises were seen, and quiz MCQs were discussed as to how models had to be read and comprehended. In the BIS course, a few quiz items were discussed with more emphasis on
modeling and understanding common errors for both basic and advanced BPMN. In the masters level course BPM&ISD, the exercise sessions were intense with a strong emphasis on advanced exercises. From table 1, one can notice that the total number of FA exercises varies per course according to the expertise level of PM needed to be mastered in that course. The higher the expertise, the FA exercises ofered are more advanced. This flexibility in the learning material is also reflected in the total number of extra exercises (quantity) and dificulty levels ofered during the in-person exercise sessions.
Efective learning is described as the process of appraising knowledge, understanding, and skills [14]. In higher education, assessments are used mainly for measuring efectiveness. Beyond evaluation, assessments can also influence several factors and purposes like feedback, motivation, self-regulation, helping promote learning [15] and pushing the learners towards achieving their learning goals. In the study modules at all three levels, to help with self-regulation, and for the students to benefit from a structured study to progress towards the learning goals of the BPMN module, FAs have been introduced in the courses as quizzes and modeling assignments (exercises). Aligning with the SA and learning goals of the three levels of BPMN modules, the number of FA exercises is reduced in ICTM (only 2 basic ones are ofered) and BIS (4 ofered- 2 basic, 2 advanced) as compared to 8 FA exercises (2 basic and 6 advanced) in BPM&ISD.
The Hattie and Timperley feedback model [16] advocates that feedback should be able to answer simple questions (the what and how) which direct the student in progressing towards the learning goals and what steps should be taken to reach them. Incorporating feedback into the teaching and learning processes and its influence on the learning outcomes in diferent learning contexts (traditional, blended, or online) have been well-established in the extant literature [17, 14]. With the increased use of technologies in education, feedback is also being automated by building it into the learning management systems (LMS) so that real-time, personalized feedback is available to learners [17]. In the case of the 6 FA quizzes, the feedback is directly built into the LMS quiz tool and video feedback via adaptive release for each FA exercise. In line with the Bloom’s taxonomy levels on SA and the learning objectives of the ICTM course, 4 videos (solution to FA exercises) with feedback on common errors are available to students (without adaptive release elements) to learn how to discern if a model is correct or not.
Specifically, in teaching how to make conceptual models and giving feedback to novice modelers, [8] suggests that knowing and learning about the kinds of modeling errors that are most likely produced, helps novice modelers in developing conceptual models of higher quality. Bogdanova [9] takes it a step further by classifying errors into an ontology and aligning it pedagogically with Bloom’s taxonomy and providing personalized feedback reports using a semi-automatic tool. Following this prescription, ofering personalized, individual feedback for every modeling exercise is not always feasible for instructors in Higher education when student cohort sizes are large. Ofering group, online, or just-in-time teaching feedback sessions have also been extensively proven to be great alternatives. To this end, during the in-person exercise sessions, individual and group feedback with an emphasis on common errors is provided. Similarly, for the FA exercises, feedback on the common mistakes and errors in the solutions has been ofered in feedback videos which become available via adaptive release when a formative modeling assignment has been submitted. The adaptive release of learning material was chosen in order to motivate students to solve the FA exercises before consulting the solutions directly.
The log data of the BPMN modules in the three courses was extracted from the institutional LMS and the breakdown of summative scores for the BPMN questions on the exam was received from the course instructors. The courses ran in the academic year 2022-2023 and were ofered in programs at the Faculty of Economics and Business at KU Leuven. Since the BPMN module was only a part of larger courses, for the summative scores we consider only the BPMN questions on the final exam. The percentage of students completing the FAs per course is shown in Table 2.
Many previous correlation studies searching for associations between FAs and SAs as listed in reviews [18, 19]. Usually, these studies present regression controlling for variables like gender, previous academic performance, and attendance [20]; reflect on whether frequency of FA attempts [21] influences summative scores; or by statistical test of the Pearson’s correlation coeficient[ 22]. In our dataset, we do not collect any personal data due to strict GDPR guidelines and to a large extent since such data as gender or prior achievement do not influence the teacher and cannot be influenced by the teacher. Attending a class in-person is also left to the student as a matter of choice or as a situational choice post-pandemic and hence this data was not available to collect as well. In the BPM&ISD course, since the scores on the case study can be influenced by group efects and not individual performance, we only include the written exam scores and do not include the score for the Case study in the correlation comparisons. Pearson’s correlation coeficient is used to indicate the relationship between the scores on the quizzes, the number of quiz attempts, the exercise attempts, the number of views on the feedback video, versus the summative scores on the BPMN questions.
The results of the correlation study are presented in Figure 1. The figure shows the correlations between the number of attempts, scores, and feedback video views associated with the FAs (quizzes and exercises) versus the SA score. The top four rows correspond to the average values of exercise attempts, quiz scores, quiz attempts, and (FA exercise) feedback video views. The pattern in the bars indicates whether the adaptive release of the material was implemented or not. In places where there are no bars, those FA exercises and/or feedback videos were not ofered in that particular course. A correlation of zero is clearly indicated. The color bar indicates the p-value giving the significance of Pearson’s tests; with darker color representing a higher p-value. When p<0.05 the correlation scores are significant.
The following are the key observations from the correlation study: (
In the BIS course, the pattern of Bloom-level correlation between FA and SA seemingly breaks down. The FA quiz scores and attempts show a higher correlation with SA scores as opposed to the FA exercise attempts. The SA of BIS course is at Bloom level - Create, whereas the high correlation occurs with the FA Quizzes which are at lower Bloom levels. It is possible that Bachelor level students are more focused on or have preferences for quizzes as opposed to Exercises as learning items or for self-regulation. A second possibility is this choice could be SA dependent - the SA of the complete course also included MCQs, so this cohort could have anticipated MCQs for the BPMN questions on the SA. The diferences in correlations (Figure 1) and the percentage of students accessing formative tests (Table 2) could also be an efect of the course design.
At the introductory or moderate knowledge/skill levels, only a few FA (basic) exercises (2
in ICTM and 4 in BIS) are ofered. Perhaps ofering a higher number of FA exercises (8 in
BPM&ISD) and a higher number of extra exercises during the exercise sessions (see Table 1)
would have shifted the students’ perspective towards solving more FA exercises. Moreover,
notice that the views on the feedback videos for advanced FA exercises (Ex 5 onward) whether
they are ofered via adaptive release (in BPM&ISD) or not (in ICTM) show a positive correlation
with the SA scores. These observations can help instructors improve the next run of these
courses: (
There are several limitations to this study. While most of the learning material was online and the activity was collected through log data, in-person interaction, and attendance could not be captured. These could also afect SA scores. For any correlation study it is important to remember that it is not causation, so these results will have to be considered carefully. The lfexible course design is heavily dependent on both the teacher’s execution of the material provided and the dificulty of the FA quizzes and exercises used in these courses. The FAs need to be carefully chosen with the right alignment of learning goals. Also, this flexibility in course design has been tested on only one module of three diferent courses. Other than BPMN, this lfexible course design could possibly be expanded to other CM modules. In the future, this should be extended to complete courses of diferent mastery levels. Also, a higher sample size will allow for more in-depth analysis leveraging machine learning techniques. Further analysis is needed to dig deeper into the FA exercises that were used in the course. The feedback given in videos explaining diferent solutions and the most common errors can be improved/updated by correcting the BPMN diagrams submitted by the students.
A PM teaching module containing a fixed (lessons and exercise sessions) and a flexible part (number of FA and extra exercises) was designed to pedagogically align the learning goals, FAs, SAs and learning items with Blooms Taxonomy levels. This module was incorporated into courses with three diferent expertise levels to be attained by cohorts following diferent programs at a university. This exploratory work is aimed at bridging the gap that exists in connecting PM modeling and instructional design. A correlation study was conducted to check whether the diferent components of the course design had any efects on the summative scores. It can be seen from the results that both formative quizzes and exercises can be efective learning tools to teach PM. Specifically, in two of the three courses, we notice that when the Bloom level on an FA matches that of the SA, the correlation is higher. The BIS cohort breaks this pattern which could be possibly explained by a cohort’s preferences in learning material, or by teacher efects. Further research is needed to ascertain if these trends can be observed across cohorts, courses, or disciplines. From this study, we realize an incremental course design aligned with Bloom’s taxonomy formative tests (exercises and quizzes) paired with feedback that makes for a lfexible solution for teaching modeling at diferent levels of expertise. Useful insights are gained on aligning FAs paired with feedback for educators helping them align their teaching resources and assessments better. In future research, we intend to focus on improving the feedback, by analyzing frequent errors in student models, as well as considering student perceptions and preferences in learning material.
[4] G. Sedrakyan, M. Snoeck, S. Poelmans, Assessing the efectiveness of feedback enabled simulation in teaching cm, Computers & Education 78 (2014) 367–382. [5] J. Claes, I. Vanderfeesten, F. Gailly, P. Grefen, G. Poels, The structured process modeling method (spmm) what is the best way for me to construct a process model?, Decision Support Systems 100 (2017) 57–76. [6] D. Bogdanova, M. Snoeck, Domain modelling in bloom: Deciphering how we teach it, in: G. Poels, F. Gailly, E. Serral Asensio, M. Snoeck (Eds.), The Practice of Enterprise Modeling, Springer International Publishing, Cham, 2017. [7] D. Bogdanova, M. Snoeck, Use of personalized feedback reports in a blended conceptual modelling course, IEEE/ACM, 2019. [8] N. Bolloju, F. Leung, Assisting novice analysts in developing quality conceptual models with uml, Communications of the ACM 49 (2006) 108–112. [9] D. Bogdanova, M. Snoeck, Let’s automate! making use of a learning ontology for conceptual data modelling, volume 2469, CEUR workshop proceedings, 2019. [10] C. Verbruggen, M. Snoeck, Exploratory study on students’ understanding of multiperspective modelling, Springer, 2022. [11] J. F. George, S. Paul, R. De’, E. Karahanna, S. Sarker, G. Oestreicher-Singer (Eds.), Building ‘holistic’ Business process modelling skills for IS graduates, Association for Information Systems, 2020. [12] B. Rosenthal, Kristina; Ternes, S. Strecker, Learning conceptual modeling: Structuring overview, research themes and paths for future research, in: Proceedings of the 27th ECIS, Stockholm Uppsala, Sweden, 2019. [13] D. R. Krathwohl, A revision of bloom’s taxonomy, Theory Into Practice 41 (2002). [14] J. Gikandi, D. Morrow, N. Davisa, Online formative assessment in higher education: A review of the literature, Computers & Education 57 (2011) 2333–2351. [15] P. Marriott, A. Lau, The use of on-line summative assessment in an undergraduate financial accounting course, Journal of Accounting Education 26 (2008) 73–90. [16] J. Hattie, H. Timperley, The power of feedback, Review of Ed. research 77 (2007). [17] G. Deeva, D. Bogdanova, E. Serral, M. Snoeck, J. De Weerdt, A review of automated feedback systems for learners: Classification framework, challenges and opportunities, Computers and education 162 (2021). [18] S. M. Ismail, D. R. Rahul, I. Patra, E. Rezvani, Formative vs. summative assessment: impacts on academic motivation, attitude toward learning, test anxiety, and self-regulation skill, Language Testing in Asia 12 (2022) 1–23. [19] F. M. Rizo, An empirical study on the impact of formative assessment: A literature review,
Revista electrónica de investigación educativa 14 (2012). [20] K. Daniel, Assessing the benefits of online formative assessments on student performance,
Journal of learning development in higher education (2023). [21] J. A. McNulty, B. R. Espiritu, A. E. Hoyt, D. C. Ensminger, A. J. Chandrasekhar, Associations between formative practice quizzes and summative examination outcomes in a medical anatomy course, Anatomical sciences education 8 (2015) 37–44. [22] N. K. Mitra, A. Barua, Efect of online formative on summative performance in integrated musculoskeletal system module, BMC medical education. 15 (2015).