Conceptual modelling languages are typically taught in graduate and postgraduate information systems programs. In the specific case of postgraduate information systems students, the lecturer has the opportunity to exploit conceptual modelling for educational purposes that go beyond learning how to apply the modelling paradigm or specific languages. In this paper, we report on our employment of the recently proposed iStar 2.0 in the master-level course Advanced Research Methods at Utrecht University in the Netherlands. During this course, the students conducted a design science project where iStar 2.0 artefacts are evaluated in an experimental setting. We present the course (intended learning objectives, learning materials, etc.), we explain how we employed iStar 2.0 therein, and we discuss our teaching experience including lessons learnt.
Teaching conceptual modelling poses major challenges for the current educational
programs in information and computer sciences. In this context, the lecture room lets the
students learn novel conceptual modelling languages, use tools and techniques for the
specification of conceptual models, apprehend guidelines and algorithms for
transforming conceptual models into software systems, and come across methods that facilitate
the practical application and evolution of conceptual modelling languages [
Paper organisation. Section 2 presents the intended learning outcomes and activities for the advanced research methods course; Section 3 describes the employment of iStar 2.0 during the ARM course; and Section 4 discusses lessons learnt and teaching directions for adopting iStar in teaching environments. 2
Intended learning outcomes and activities for the ARM course The Master’s Programme in Business Informatics (MBI) at Utrecht University, Department of Information and Computing Sciences, is a research master with an integrative and multidisciplinary approach that trains future ICT Researchers, Analysts, and Entrepreneurs. ARM1 is a compulsory course and one of the pillars of the MBI program. The main objective of the course is to train the students to apply research methods, and give the students the opportunity to play the role of researchers in information sciences.
2.1
During the lectures and laboratory sessions, students gain skills to conduct research projects, develop a researcher attitude, and acquire knowledge about research methods.
To contribute to the successful achievement of ILOs 1 and 2, we provided students with conceptual modelling artefacts that can be evaluated in a certain context. We selected these among the iStar 2.0 artefacts and defined the following learning outcomes: a) Knows how iStar 2.0, and social modelling languages in general, relate to other conceptual and enterprise modelling languages; b) Is able to comprehend existing iStar 2.0 models by having learned the language’s syntax and semantics;
To achieve the learning outcomes a), b) and c), in the same spirit as [
For the ARM course, we designed different activities for one academic period of 10 weeks. Fig. 1 illustrates the main activities that took place during the 2016-2017 instance of the course. We kick-started the course with an introductory lecture, in which we explained the main objectives and activities of the course. The same day, we asked the students to fill in a demographic questionnaire with the purpose to get more insights about their background and previous experience with research methods, requirements engineering, conceptual modelling, goal modelling, business process modelling, and design science. As a result, we found out that 53.7% of the students (29 out of 54 students) had experience with conceptual modelling thanks to bachelor courses like information sciences and data modelling. One student reported to have industrial experience with the application of conceptual modelling for software development.
LECTURE: EXPERIMENTAL PLANNING &
OPERATION LECTURE: EXPERIMENTAL i* DESIGN
DISTRIBUTION OF TEAMS
DRAFT OF EXPERIMENTAL EXECUTION
RAW DATA PER TEAM
RESULTS OF STATISTICAL
TESTS RESEARCH QUESTIONS
LECTURE: STATISTICS
DRAFT DATA ANALYSIS POSTER
LECTURES: REPORTING & PACKAGING + ELEVATOR PITCH+POSTER LECTURE: DESIGN SCIENCE
FILLED DEMOGRAPHIC DEMOGRAPHIC
QUEST
QUEST 1. KICK
OFF 2. CREATE TEAMS
DRAFT: TITTLE + AUTHORS +
ABSTRACT 4. SET UP THE
DESIGN SCIENCE
PROJECT S M A E T L L A S M A E T L A U D I V I D N I 9. COMPILE RESULTS FOR THE PROJECT COMPILED DATA PER PROJECT notions about it. Regarding their professional experience, 42.6% of the students have some professional experience; some roles to highlight are analyst, entrepreneur, programmer, and junior consultant.
For the next session (task 4 in Fig. 1), the students received the training to set up design science projects and started to study the selected artefact for each team. For the iStar lecture (task 5 in Fig. 1), the third author trained them to use iStar 2.0. For this, the students received an interactive lecture of 2 hours plus 2 hours of practice. For the practical assignment, the students took a scenario and modelled by using iStar. They were in charge of identifying the main actors, define their goals, find their dependencies, use intentional element links, analyse and evaluate alternative ways of fulfilling goals, create the models pen-on-paper, and scan and send the models the same day. We checked the models and the students received feedback to improve their models. The students were committed to the activity for two main reasons: 1) they wanted to learn the details about the artefact to evaluate, and 2) the models they prepared would serve as experimental objects for the comparative experiments (the main project of the course).
When the students finalised the experimental design and improved the iStar models, they conducted the experimental tasks. Finally, they collected data, performed data analysis, and reported the results via a scientific paper, a poster, and an elevator pitch. 3
Using iStar 2.0: the ARM course projects During the ARM course, the students conducted a design science project with a basic design cycle and a full empirical cycle. For this, the students were distributed in teams of 3-4 people; each team elected a team leader (see Fig. 2). Since it was not the objective of the course to build information systems artefact from scratch, the students studied existent iStar 2.0 artefacts in the context of a design science project.
For the design cycle, each team selected 1 out of 4 iStar 2.0 artefacts. For this year, we have selected two artefacts that prescribe guidelines and techniques that use iStar 2.0 models (A1 and A2), and two artefacts that support the specification and syntactical notation of iStar 2.0 models (A3 and A4).
For the empirical cycle, each team conducted a comparative experiment for an iStar 2.0 artefact. During the empirical cycle, students analysed a given a problem with the artefact, designed an experiment, analysed the threats on the validity of the experiment, executed the experimental tasks, and analysed the experimental results. For the experimental tasks, the students acted as researchers of their own experiment, also as experimental subjects of the experiments of their researcher fellows. For example, in the case of the team A1.1, they took the role of researchers for the artefact P1 whereas they were the experimental subjects of the teams A2.1, A3.1, and A4.1 (see Fig. 2). A1 SUBJECTS Teams: A3.2 A3.3 A3.4 DATA A1
A3 SUBJECTS Teams: A4.2 A4.3 A4.4
DATA A3 TEAM A1.1 TEAM A1.2 TEAM A1.3 TEAM A1.4 TEAM A1.5 TEAM A2.1 TEAM A2.2 TEAM A2.3 TEAM A2.4 TEAM A2.5 SUBJECTS Teams: A2.1 A2.2 A2.3
SUBJECTS Teams: A2.4 A2.5 A3.1
SUBJECTS SUBJECTS Teams: Teams: A3.5 A4.3 A4.1 A4.4 A4.2 A4.5
SUBJECTS Teams: A1.1 A1.2 A1.3
SUBJECTS Teams: A1.4 A1.5 A3.1 SUBJECTS Teams: A1.1 A1.2 A1.3
SUBJECTS Teams: A1.4 A1.5
A4.1 Legend PROJECT SUBJECTS MEMBER LTEEAADMER ASSOCIATION
TEAM/ TEAM A3.1 TEAM A3.2 TEAM A3.3 TEAM A3.4 TEAM A3.5 TEAM A4.1 TEAM A4.2 TEAM A4.3 TEAM A4.4 TEAM A4.5 SUBJECTS SUBJECTS SUBJECTS Teams: Teams: Teams: A4.5 A2.3 A1.1 A2.1 A2.4 A1.2 A2.2 A2.5 A1.3
SUBJECTS Teams: A1.4 A1.5 A2.1
A2 SUBJECTS Teams: A3.2 A3.3 A3.4 DATA A2
A4 SUBJECTS Teams: A2.2 A2.3 A2.4 DATA A4
SUBJECTS Teams: A3.5 A4.1 A4.2
SUBJECTS Teams: A4.3 A4.4
A4.5 SUBJECTS Teams: A2.5 A3.1 A3.2
SUBJECTS
Teams:
A3.3
A3.4
A3.5
For the design of the comparative experiments, the students applied the experimental
protocol for experimentation in software engineering prescribed by Wohlin et al. [
Experimental objects: iStar 2.0 artefacts We provided the students with iStar 2.0 artefacts for their evaluation. Table 2 presents a summary of the artefacts2. 2 Further details about the given artefacts are available as part of the course material:
[
and information gathering of two information systems. The Delta Analysis technique serves on the purpose to analyse the delta of two information systems. Delta Analysis is model-based; thus, the comparison or delta is performed between pair of models that specify information systems. The
make errors and create models that are not compliant with the syntax. To such extent, iStar 2.0 comes with meta-model enriched with additional constraints about syntactic well formedness. iStar 2.0 models can be drawn by hand on paper, digitally using a general purpose drawing tool such as Microsoft Visio, or using a dedicated application for iStar 2.0 such as piStar.
the original version of iStar) to represent concepts. For example, circles represent actors, stadium shaped nodes represent goals, and hexagons represent tasks. The relationships between these constructs are captured by links between those shapes, sometimes labelled by text. The arrow heads used to connect the directed edges may differ for different relationships.
To illustrate the experimental setups used in the course, we describe the project of the students that have selected the artefact A4. In this project, the students (5 teams) have designed a comparative expeiment where the standard iStar2.0 notation is compared against the Moody’s visual notation in terms of practioners‘ performance and perceptions (see Fig. 3). Each team had between 9 to 11 experimental subjects. For this experiment, we defined dependent and independent variables, which were measured by means of two experimental tasks in two weeks (see Fig. 3, week 1 and week 2). Each team of artefact A4 elected one context and specified models using the standard iStar 2.0 and Moody’s notations. In the first experimental task the experimental subjects were divided in two groups: the group A analysed a goal model specified by means of the iStar 2.0 notation, and the group B analysed a goal model based on Moody’s notation. In the second experimental task, each team chose a different context from the one used for the experimental task 1 and defined the iStar 2.0 and Moody‘s models. In week 2, the teams switched the experimental subjects as described in Fig. 3; and kept the same complexity of the experimental objects in terms of the amount of modelling elements and type of elements that were used in the models for the experimental task 1. TREATMENT
INPUT MODEL
COMPRENHENSIBILITY READABILITY
USEFULNESS EFFICIENCY
EASE OF USE INDEPENDENT VARIABLES
PERFORMANCE
PERCEPTIONS DEPENDENT VARIABLES
INTENTION TO USE
Week 1 1 iStar 2.0 A
Week 2 2 iStar 2.0 B
To measure comprehensibility and readability, the teams have designed questionnaires
with context dependent questions about the iStar 2.0 and Moody’s models. In this case,
10 questions were dedicated to readability and 5 to comprehensibility. For the
efficiency, each team registered the amount of time that each subject took to finish the
questionnaire. Finally, the teams measured usefulness, ease of use and intention to use
according to a quality framework [
Discussion and lessons learnt In this paper, we reported our experience in using conceptual modelling as artefact for teaching non-conventional information systems courses like research methods. A key aim of our attempt is to make students understand that conceptual modelling research goes well beyond reading and creating models, but rather involves thorough theoretical and empirical studies concerning conceptual modelling artefacts.
In our case, we describe the design of the Advanced Research Methods course (ARM), which has as a main component the Design Science Method. As part of the main project of the course, students conducted a design science project in order to evaluate artefacts in context. For the academic year 2016-2017, students evaluated conceptual model artefacts by means of a comparative experiment, wrote a research paper, and presented the results by means of a poster and elevator pitch. As a proof of concept, the students evaluated iStar 2.0 artefacts.
Reflecting on the intended learning outcomes and our experience, we highlight the following positive findings and opportunities for improvement. 4.1
Positive findings The existence of conceptual modelling language variants can be beneficial! One of the main criticisms made to the conceptual modelling community is that many variants of a given language (and related artefacts) are proposed. However, this drawback turned into an advantage for the ARM course, for we could easily select different iStar 2.0 artefacts to evaluate. Other conceptual modelling languages, such as for business processes, offer a comparable artefact selection space to be exploited. Easily customizable materials and artefacts. Conceptual modelling artefacts are adaptable for classroom settings without investing considerable resources. It is relatively simple and nonintrusive to modify the graphical notation, alter the syntax, or build a new analysis algorithm. Compare this to the effort required to modify a software product (e.g., user interface, algorithm, input devices). Thus, conceptual models are excellent artefacts for use in a research methods course. Experimental outcomes trigger new design cycles for conceptual model artefacts.
The students gained first-hand experience about the limitations of existing conceptual modeling artifacts. This is a much more powerful learning experience compared to a teacher telling them that a modelling language suffers from a certain problem. We are confident that the obtained findings will lead some students to follow a new design cycle during their master’s thesis. Comparable complexity of the experiments. Despite the differences of the four evaluated artefacts, it was possible for the instructor to keep a balance in workload of each team in terms of artefacts to build and variables to measure. For example, the teams measured subject performance and perceptions for each artefact, which required them to build experimental objects (iStar 2.0 models plus other objects according to the type of artefact, like Moody’s models), create questionnaires to evaluate subjects’ perceptions, and record the time spent per subject during the execution of the experimental task. The balance in the design and workload of the projects was a key point to avoid frustration and competition among the teams. 4.2
Opportunities for improvement Managing multiple experiments at the same time. It was difficult to assist four different research projects and schedule the parallel execution of 40 experimental task (20 teams, two experimental tasks each) in two weeks. These aspects need to be readjusted by, e.g., offering only a couple projects and fewer experimental objects. Testing the achievement of the iStar ILOs. Although we supported students by giving them feedback about their models until they reached a solid version, we did not evaluated the actual knowledge in iStar modelling. For the next edition of the ARM course, it is necessary to evaluate the extent to which each student achieved the iStarrelated ILOs; this will reduce the threats to validity of the experiments given the analysis of iStar models that was required for all the experimental tasks. On the difficulty of establishing scientific rigor. Students learnt how to identify threats to the validity of the experiments, but we did not have sufficient time or experience to avoid many of them. Some threats were related to the difficulty to manage four experiments and to ensure the participation of the students in the experimental tasks. We noticed that the students tend to feel frustrated if they need to confront a threat to the validity of their experiments. It is important to allocate sufficient time for the students to prevent the most important threats and to teach them how to mitigate the threats when they occur. For example, testing the subjects’ knowledge of iStar would help ensure an appropriate execution of the experimental tasks. Looking at the intended learning objectives for ARM, we find that the experimental setting and the employment of iStar 2.0 are appropriate for a master-level course, especially thanks to the existence of many artefacts and the possibility to follow a full experimental protocol. In the upcoming years, we plan to improve the course design by establishing a better distribution of teams and less artefacts to evaluate. Also, we plan to reduce the amount of experimental objects (one per artefact, but not one per team), and to help the students manage the threats to the validity of the experimental results. 5