Teaching conceptual modeling leveraging formative assessments and adaptive release paths Pavani Vemuri1,∗ , Stephan Poelmans1 , Estefania Serral1 and Monique Snoeck1 1 KU Leuven, 3000 Leuven, Belgium Abstract Teaching and learning conceptual modeling have been reported to be difficult tasks. In the past few decades, there have been a myriad of studies and resources on how to simplify teaching, different teaching methods, and frameworks proposed for conceptual modeling education. Nevertheless, there exist gaps in helping educators construct facile study modules to help activate students toward achieving their learning goals. In order to make the choices on pedagogy, method, and assessment for an educator easier, we present three courses containing a business process management teaching module at different levels of mastery (introductory, familiar, and advanced) in blended contexts at a higher education institute. We examine and deliberate on several aspects of the course design. Through correlation analysis, this study explores whether (1) there are any discernible effects of the design on outcomes by analyzing the association between formative and summative scores; (2) the course design employed in all three study modules possesses the necessary flexibility to accommodate various levels of mastery that are intended to be achieved; and (3) the utilization of adaptive release of learning material effectively stimulates student engagement and participation. Keywords Conceptual modeling education, formative assessment, formative feedback, learning report 1. Introduction - Importance of teaching conceptual modeling Conceptual modeling (CM) plays a vital role in modern businesses, specifically in software development lifecycles and in process (re)design and (re)engineering. Especially when using poor quality models in the early stages of design, harmful effects are caused by the use and application of these models [1] which can prove to be very expensive in later stages of design. In describing several systems the field and role of CM have expanded dramatically including the modeling of workflows, data flows, workforces, processes etc, utilizing the most commonly used techniques and methods such as Unified Modeling Language (UML) or specific data models like ERD [2]. In this study, we focus on process modeling (PM), which belongs to the realm of CM, particularly in the area of information systems design, where it deals with dynamic organizational processes that concern diverse audiences (domain experts, managers, employees, ER2023: Companion Proceedings of the 42nd International Conference on Conceptual Modeling: ER Forum, 7th SCME, Project Exhibitions, Posters and Demos, and Doctoral Consortium, November 06-09, 2023, Lisbon, Portugal ∗ Corresponding author. Envelope-Open pavani.vemurii@kuleuven.be (P. Vemuri); stephan.poelmans@kuleuven.be (S. Poelmans); estefania.serralasensio@kuleuven.be (E. Serral); monique.snoeck@kuleuven.be (M. Snoeck) Orcid 0000-0001-8668-2011 (P. Vemuri); 0000-0002-4208-9723 (S. Poelmans); 0000-0001-7579-910X (E. Serral); 0000-0002-3824-3214 (M. Snoeck) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings ICT staff, etc) and purposes in the clarification, redesign or automation of existing processes. Novice designers and business analysts will be placed at an advantage in organizations if they have developed strong modeling skill sets during their university education. However, teaching requirements formalization through CM comes with its own set of challenges[3]. Acquiring proficiency in PM is essential for effectively managing or automating business processes, but learning PM poses challenges due to its inherent complexity, requiring specific cognitive schemata and practical experience [4]. Enhancing the training of process modelers contributes to the acquisition of necessary skills for PM, potentially addressing the prevalent problem of subpar process model quality within organizations [5]. There have been several publications researching CM education and on the usefulness of feedback in teaching CM in Higher education [6, 7, 8, 4]. But thus far, no standardized approach has been established for teaching CM or incorporating feedback into the teaching process. While there are myriad of educational materials available such as MOOCs, textbooks, and traditional classroom courses for/on teaching CM as listed in [9], there is no one-size-fits-all approach. Neither is there an understanding of how to support CM learning [10]. When offering/designing a course with PM, a teaching team needs to address (1) at what level (basic introduction to advanced knowledge/skill); (2) how (choose the blended scale, tools used, pedagogy); and (3) how much (should be influenced by learning goals and assessments) to teach. The teaching of PM occupies a prominent position on research agendas within the PM discipline [11, 12]. Nevertheless, in PM there are very few works connecting teaching PM with pedagogy or instructional design. In the presentation of this work, we try to provide direction to educators of PM in order to simplify these choices and disentangle the complexities of teaching PM using Business Process Modeling Notation (BPMN). To this end, we present a PM module in three courses at different knowledge/skill levels. While designing the courses discussed in this work, a few major decision points were kept in mind to align them pedagogically, with existing research and resources based on CM education. 2. Course design 2.1. Audience The BPMN module was presented to business program cohorts enrolled in the academic year 2022-2023 at three different academic levels: Bridging program (preparatory for an Academic Master for Non-Academic Bachelor), Academic Bachelor , and Academic Master. The cohort following the Bridging program completed a Non-Academic Bachelor. The Academic Bachelor cohort is in their third year of Bachelor in Business Engineering. The Masters’ cohort is following a 1-year MBA. The courses are ICT Management (ICTM) in the Bridging program, Business Information Systems (BIS) in the Bachelor program, and Business Process Modelling and Information Systems Design (BPM & ISD) in the Masters program. All cohorts are considered novice process modelers with none or very little modeling experience. 2.2. Learning objectives & evaluation The details on learning aims, ECTS, and summative evaluation of the three courses ICTM1 , BIS2 and BPM&ISD3 can be found in the links below in the footnotes. The three courses have distinct levels of learning goal complexities which we encapsulate pedagogically using Bloom’s Taxonomy[13]. [6] applied the revised Bloom’s Taxonomy of educational objectives[13] for domain modeling, identifying and defining 6 cognitive process levels and 4 knowledge levels with examples in the context of learning CM. In designing the module, we enforce these scaffolding levels and apply them to PM. We classified the contents starting from the evaluation (summative assessment:SA), the learning objectives, learning items, formative assessments (FAs) of the BPMN module of each course into different Bloom categories as defined in [6]. For example, the SA for the ICTM course only contains Multiple Choice questions (MCQs) which classifies it into the Understand, Analyse, and Evaluate levels. For the BIS and BPM&ISD courses, students have to model a given textual description using BPMN, which classifies these into the Create level. Additionally, BPM&ISD students have to solve a real case study. Table 1 lists the course characteristics and how the Bloom’s taxonomy levels map to course components and assessments. After following the courses ICTM, BIS, and BPM&ISD respectively, students are expected to have an introductory, moderate, and expert level of familiarity using BPMN. 2.3. Learning modules (content) & mode of delivery At first, the complete BPMN material was designed with some fixed elements (lessons and practice exercise sessions) partitioned into BPMN Basics and Advanced subsections. In lieu of the university’s response to post-pandemic circumstances and culture, where a reduced number of students attended in-person classes but took the exam, most of the learning material was required to be online. Lessons were planned as videos with each video having a central focus on specific BPMN concepts. Interspersed between the 4 BPMN Basics and 7 BPMN advanced videos, the FAs (6 quizzes; 2-8 exercises) were planned to help the student with self-evaluation and self-regulation. These learning items were also offered through an online component, thus giving autonomy to the learners to pace themselves and to work on it at home in case they cannot come to class. In-person sessions were set up for solving the exercises. During these in-person sessions, after a quick review of the lectures, a few exercises are solved step-by-step to introduce how a textual description can be interpreted and a process diagram be realized. Then, students are given time to solve exercises by themselves and the instructor and teaching assistant are available to answer any individual difficulties. A few possible solutions, common difficulties, and errors are discussed for each of the exercises at the end. This is done to facilitate discussion and improve on the quality of models. Exercises discussed in the exercise sessions for the three module levels were chosen carefully giving particular attention to the difficulty level of the exercises. In ICTM, more basic exercises were seen, and quiz MCQs were discussed as to how models had to be read and comprehended. In the BIS course, a few quiz items were discussed with more emphasis on 1 ICTM: https://onderwijsaanbod.kuleuven.be//2022/syllabi/e/HSA17AE.htm 2 BIS: https://onderwijsaanbod.kuleuven.be//2022/syllabi/e/HBN63BE.htm 3 BPM&ISD: https://onderwijsaanbod.kuleuven.be//2022/syllabi/e/HMH28FE.htm Table 1 Course and assessment characteristics Course Characteristics ICTM BIS BPM & ISD Academic Level Program Bridging Bachelors Masters Semester Sem 2 Sem 1 Sem 2 OPO ID HSA17a HBN63 B HMH28F Number of Students 75 37 44 ECTS Total course 3 6 6 ECTS BPMN Module 1.3 2 2.3 No. of teaching hours 8 hours 12 hours 14 hours BPMN Tool used Signavio Signavio Visual Paradigm No. of FA quizzes 6 6 6 No. of FA Exercises 2 4 8 Extra exercises 8 11 15 Summative Assessment (SA): Possible SA points 20 20 20 SA Type Written exam (20) Written exam (20) Group Case (10) + Written exam (10) Questions types on SA MCQs Draw models Draw models (BPMN only) No. of points 6 (out of 20) 4 (out of 20) 5 (out of 10) Written on SA (BPMN only) 5 (out of 10) Case Bloom levels: SA questions Understand, Create Create Analyse, Evaluate Teaching module Understand, Create Create Analyse, Evaluate Formative quizzes Understand, Understand, Understand, Analyse, Evaluate Analyse, Evaluate Analyse, Evaluate Formative Exercises Create Create Create Exercise Sessions Create Create Create modeling and understanding common errors for both basic and advanced BPMN. In the masters level course BPM&ISD, the exercise sessions were intense with a strong emphasis on advanced exercises. From table 1, one can notice that the total number of FA exercises varies per course according to the expertise level of PM needed to be mastered in that course. The higher the expertise, the FA exercises offered are more advanced. This flexibility in the learning material is also reflected in the total number of extra exercises (quantity) and difficulty levels offered during the in-person exercise sessions. 2.4. Formative assessments, feedback, and adaptive release Effective learning is described as the process of appraising knowledge, understanding, and skills [14]. In higher education, assessments are used mainly for measuring effectiveness. Beyond evaluation, assessments can also influence several factors and purposes like feedback, motivation, self-regulation, helping promote learning [15] and pushing the learners towards achieving their learning goals. In the study modules at all three levels, to help with self-regulation, and for the students to benefit from a structured study to progress towards the learning goals of the BPMN module, FAs have been introduced in the courses as quizzes and modeling assignments (exercises). Aligning with the SA and learning goals of the three levels of BPMN modules, the number of FA exercises is reduced in ICTM (only 2 basic ones are offered) and BIS (4 offered- 2 basic, 2 advanced) as compared to 8 FA exercises (2 basic and 6 advanced) in BPM&ISD. The Hattie and Timperley feedback model [16] advocates that feedback should be able to answer simple questions (the what and how) which direct the student in progressing towards the learning goals and what steps should be taken to reach them. Incorporating feedback into the teaching and learning processes and its influence on the learning outcomes in different learning contexts (traditional, blended, or online) have been well-established in the extant literature [17, 14]. With the increased use of technologies in education, feedback is also being automated by building it into the learning management systems (LMS) so that real-time, personalized feedback is available to learners [17]. In the case of the 6 FA quizzes, the feedback is directly built into the LMS quiz tool and video feedback via adaptive release for each FA exercise. In line with the Bloom’s taxonomy levels on SA and the learning objectives of the ICTM course, 4 videos (solution to FA exercises) with feedback on common errors are available to students (without adaptive release elements) to learn how to discern if a model is correct or not. Specifically, in teaching how to make conceptual models and giving feedback to novice modelers, [8] suggests that knowing and learning about the kinds of modeling errors that are most likely produced, helps novice modelers in developing conceptual models of higher quality. Bogdanova [9] takes it a step further by classifying errors into an ontology and aligning it pedagogically with Bloom’s taxonomy and providing personalized feedback reports using a semi-automatic tool. Following this prescription, offering personalized, individual feedback for every modeling exercise is not always feasible for instructors in Higher education when student cohort sizes are large. Offering group, online, or just-in-time teaching feedback sessions have also been extensively proven to be great alternatives. To this end, during the in-person exercise sessions, individual and group feedback with an emphasis on common errors is provided. Similarly, for the FA exercises, feedback on the common mistakes and errors in the solutions has been offered in feedback videos which become available via adaptive release when a formative modeling assignment has been submitted. The adaptive release of learning material was chosen in order to motivate students to solve the FA exercises before consulting the solutions directly. 3. Methodology 3.1. Data and context The log data of the BPMN modules in the three courses was extracted from the institutional LMS and the breakdown of summative scores for the BPMN questions on the exam was received from the course instructors. The courses ran in the academic year 2022-2023 and were offered in programs at the Faculty of Economics and Business at KU Leuven. Since the BPMN module was only a part of larger courses, for the summative scores we consider only the BPMN questions on the final exam. The percentage of students completing the FAs per course is shown in Table 2. Table 2 Percentages of students accessing FAs: Quizzes (FA Qz) and Exercises (FA Ex) FAs ICTM BIS BPM & ISD FA Qz_1 80.0 62.16 59.09 FA Qz_2 82.67 54.05 54.55 FA Qz_3 73.33 51.35 52.27 FA Qz_4 74.67 48.65 56.82 FA Qz_5 62.67 37.84 52.27 FA Qz_6 56.0 29.73 50.0 FA Ex_1 13.33 37.84 70.45 FA Ex_2 13.33 37.84 52.27 FA Ex_3 - 29.73 45.45 FA Ex_4 - 27.03 43.18 FA Ex_5 - - 38.64 FA Ex_6 - - 34.09 FA Ex_7 - - 29.55 FA Ex_8 - - 38.64 3.2. Correlation study Many previous correlation studies searching for associations between FAs and SAs as listed in reviews [18, 19]. Usually, these studies present regression controlling for variables like gender, previous academic performance, and attendance [20]; reflect on whether frequency of FA attempts [21] influences summative scores; or by statistical test of the Pearson’s correlation coefficient[22]. In our dataset, we do not collect any personal data due to strict GDPR guidelines and to a large extent since such data as gender or prior achievement do not influence the teacher and cannot be influenced by the teacher. Attending a class in-person is also left to the student as a matter of choice or as a situational choice post-pandemic and hence this data was not available to collect as well. In the BPM&ISD course, since the scores on the case study can be influenced by group effects and not individual performance, we only include the written exam scores and do not include the score for the Case study in the correlation comparisons. Pearson’s correlation coefficient is used to indicate the relationship between the scores on the quizzes, the number of quiz attempts, the exercise attempts, the number of views on the feedback video, versus the summative scores on the BPMN questions. 4. Results and discussion The results of the correlation study are presented in Figure 1. The figure shows the correlations between the number of attempts, scores, and feedback video views associated with the FAs (quizzes and exercises) versus the SA score. The top four rows correspond to the average values of exercise attempts, quiz scores, quiz attempts, and (FA exercise) feedback video views. The pattern in the bars indicates whether the adaptive release of the material was implemented or not. In places where there are no bars, those FA exercises and/or feedback videos were not offered in that particular course. A correlation of zero is clearly indicated. The color bar indicates the p-value giving the significance of Pearson’s tests; with darker color representing a higher p-value. When p<0.05 the correlation scores are significant. The following are the key observations from the correlation study: (1) The ICTM and BIS courses have the highest correlation between FA quiz scores with the SA score. (2) For the BPM & ISD course, the highest correlation is for the number of FA exercise attempts. (3) The feedback video views available through adaptive release do not correlate to SA scores in the ICTM and BIS courses but are highly correlated (p<=0.05), especially for the advanced FA exercises in the BPM&ISD course. (4) In the ICTM course, the feedback solution videos for advanced exercises without adaptive release also have a high correlation with exam scores. In Table 2 we can see that in the ICTM course, very few students have attempted the two FA exercises. This is reflected in the insignificant and low correlation score for the FA exercises in Figure 1. Moreover, the high correlation between the FA quizzes and the SA scores could be a reflection of the course design as the ICTM SA contains MCQs (Bloom levels - Understand, Analyse, Evaluate) and the FA Quizzes are on the same Bloom levels. In BPM&ISD, while more than 50 percent of the students attempt all the quizzes (see Table 2), the correlation is not significant either for the number of attempts or scores of FA quizzes. The correlation between FA (advanced) exercise attempts and SA scores is high. This is expected (similar in ICTM) as the SA is composed of modeling questions (Bloom level - Create) and FA exercises are on the same Bloom level as well. In Table 2, across courses there is a tendency of students to not access later FAs. This could be due to a drop in motivation towards the end of the chapter or giving up on FAs which are more complex or simply due to content fatigue. This can be further investigated by surveying students and collecting their perspectives and preferences. In the BIS course, the pattern of Bloom-level correlation between FA and SA seemingly breaks down. The FA quiz scores and attempts show a higher correlation with SA scores as opposed to the FA exercise attempts. The SA of BIS course is at Bloom level - Create, whereas the high correlation occurs with the FA Quizzes which are at lower Bloom levels. It is possible that Bachelor level students are more focused on or have preferences for quizzes as opposed to Exercises as learning items or for self-regulation. A second possibility is this choice could be SA dependent - the SA of the complete course also included MCQs, so this cohort could have anticipated MCQs for the BPMN questions on the SA. The differences in correlations (Figure 1) and the percentage of students accessing formative tests (Table 2) could also be an effect of the course design. At the introductory or moderate knowledge/skill levels, only a few FA (basic) exercises (2 in ICTM and 4 in BIS) are offered. Perhaps offering a higher number of FA exercises (8 in BPM&ISD) and a higher number of extra exercises during the exercise sessions (see Table 1) would have shifted the students’ perspective towards solving more FA exercises. Moreover, notice that the views on the feedback videos for advanced FA exercises (Ex 5 onward) whether they are offered via adaptive release (in BPM&ISD) or not (in ICTM) show a positive correlation with the SA scores. These observations can help instructors improve the next run of these courses: (1) very simple exercises (specifically Ex1 and 2) can be dropped as FAs; (2) FA quizzes can be dropped from BPM&ISD, (3) offer the feedback videos for advanced FA exercises in the BIS course (similar to that of ICTM) without adaptive release; (4) adaptive release is more relevant in BPM&ISD and hence this restriction on feedback videos can be dropped in other Figure 1: Correlation between the formative and summative assessments courses. There are several limitations to this study. While most of the learning material was online and the activity was collected through log data, in-person interaction, and attendance could not be captured. These could also affect SA scores. For any correlation study it is important to remember that it is not causation, so these results will have to be considered carefully. The flexible course design is heavily dependent on both the teacher’s execution of the material provided and the difficulty of the FA quizzes and exercises used in these courses. The FAs need to be carefully chosen with the right alignment of learning goals. Also, this flexibility in course design has been tested on only one module of three different courses. Other than BPMN, this flexible course design could possibly be expanded to other CM modules. In the future, this should be extended to complete courses of different mastery levels. Also, a higher sample size will allow for more in-depth analysis leveraging machine learning techniques. Further analysis is needed to dig deeper into the FA exercises that were used in the course. The feedback given in videos explaining different solutions and the most common errors can be improved/updated by correcting the BPMN diagrams submitted by the students. 5. Conclusion A PM teaching module containing a fixed (lessons and exercise sessions) and a flexible part (number of FA and extra exercises) was designed to pedagogically align the learning goals, FAs, SAs and learning items with Blooms Taxonomy levels. This module was incorporated into courses with three different expertise levels to be attained by cohorts following different programs at a university. This exploratory work is aimed at bridging the gap that exists in connecting PM modeling and instructional design. A correlation study was conducted to check whether the different components of the course design had any effects on the summative scores. It can be seen from the results that both formative quizzes and exercises can be effective learning tools to teach PM. Specifically, in two of the three courses, we notice that when the Bloom level on an FA matches that of the SA, the correlation is higher. The BIS cohort breaks this pattern which could be possibly explained by a cohort’s preferences in learning material, or by teacher effects. Further research is needed to ascertain if these trends can be observed across cohorts, courses, or disciplines. From this study, we realize an incremental course design aligned with Bloom’s taxonomy formative tests (exercises and quizzes) paired with feedback that makes for a flexible solution for teaching modeling at different levels of expertise. Useful insights are gained on aligning FAs paired with feedback for educators helping them align their teaching resources and assessments better. In future research, we intend to focus on improving the feedback, by analyzing frequent errors in student models, as well as considering student perceptions and preferences in learning material. Acknowledgments Thanks to Bert Coenen, ICT, KU Leuven for providing the data for this study. References [1] H. A. Reijers, J. Mendling, J. Recker, Business process quality management, in: Handbook on Business Process Management 1, Springer, 2010, pp. 167–185. [2] R. Buchmann, A.-M. Ghiran, V. Döller, D. Karagiannis, Conceptual modelling in education: a position paper, in: R. Matuleviius, R. A. Buchmann, V. epa, M. Kirikova, K. Sandkuhl, M. Pańkowska (Eds.), 4th Workshop on Managed Complexity, 2019. [3] G. Sedrakyan, M. Snoeck, Effects of simulation on novices’ understanding of the concept of inheritance in conceptual modeling, in: M. A. Jeusfeld, K. Karlapalem (Eds.), Advances in CM, Springer International Publishing, Cham, 2015, pp. 327–336. [4] G. Sedrakyan, M. Snoeck, S. Poelmans, Assessing the effectiveness of feedback enabled simulation in teaching cm, Computers & Education 78 (2014) 367–382. [5] J. Claes, I. Vanderfeesten, F. Gailly, P. Grefen, G. Poels, The structured process modeling method (spmm) what is the best way for me to construct a process model?, Decision Support Systems 100 (2017) 57–76. [6] D. Bogdanova, M. Snoeck, Domain modelling in bloom: Deciphering how we teach it, in: G. Poels, F. Gailly, E. Serral Asensio, M. Snoeck (Eds.), The Practice of Enterprise Modeling, Springer International Publishing, Cham, 2017. [7] D. Bogdanova, M. Snoeck, Use of personalized feedback reports in a blended conceptual modelling course, IEEE/ACM, 2019. [8] N. Bolloju, F. Leung, Assisting novice analysts in developing quality conceptual models with uml, Communications of the ACM 49 (2006) 108–112. [9] D. Bogdanova, M. Snoeck, Let’s automate! making use of a learning ontology for conceptual data modelling, volume 2469, CEUR workshop proceedings, 2019. [10] C. Verbruggen, M. Snoeck, Exploratory study on students’ understanding of multi- perspective modelling, Springer, 2022. [11] J. F. George, S. Paul, R. De’, E. Karahanna, S. Sarker, G. Oestreicher-Singer (Eds.), Building ‘holistic’ Business process modelling skills for IS graduates, Association for Information Systems, 2020. [12] B. Rosenthal, Kristina; Ternes, S. Strecker, Learning conceptual modeling: Structuring overview, research themes and paths for future research, in: Proceedings of the 27th ECIS, Stockholm Uppsala, Sweden, 2019. [13] D. R. Krathwohl, A revision of bloom’s taxonomy, Theory Into Practice 41 (2002). [14] J. Gikandi, D. Morrow, N. Davisa, Online formative assessment in higher education: A review of the literature, Computers & Education 57 (2011) 2333–2351. [15] P. Marriott, A. Lau, The use of on-line summative assessment in an undergraduate financial accounting course, Journal of Accounting Education 26 (2008) 73–90. [16] J. Hattie, H. Timperley, The power of feedback, Review of Ed. research 77 (2007). [17] G. Deeva, D. Bogdanova, E. Serral, M. Snoeck, J. De Weerdt, A review of automated feedback systems for learners: Classification framework, challenges and opportunities, Computers and education 162 (2021). [18] S. M. Ismail, D. R. Rahul, I. Patra, E. Rezvani, Formative vs. summative assessment: impacts on academic motivation, attitude toward learning, test anxiety, and self-regulation skill, Language Testing in Asia 12 (2022) 1–23. [19] F. M. Rizo, An empirical study on the impact of formative assessment: A literature review, Revista electrónica de investigación educativa 14 (2012). [20] K. Daniel, Assessing the benefits of online formative assessments on student performance, Journal of learning development in higher education (2023). [21] J. A. McNulty, B. R. Espiritu, A. E. Hoyt, D. C. Ensminger, A. J. Chandrasekhar, Associations between formative practice quizzes and summative examination outcomes in a medical anatomy course, Anatomical sciences education 8 (2015) 37–44. [22] N. K. Mitra, A. Barua, Effect of online formative on summative performance in integrated musculoskeletal system module, BMC medical education. 15 (2015).