In our previous publications we introduced the concept of Didactical Structural Templates (DST) which are defined as a possibility to describe the didactical structure of a course, a study program or an applied game in an abstract way. The idea of DSTs is based on the structure of IMS Learning Design (IMS-LD), which is a quasi-standard for modelling learning structures. An extension of the concept to support learning analytics, so-called Learning Assessment Goals (LAG) are defined, ofers a way to determine the state execution respectively it is ended successful or not. Furthermore, we will extend the DSTs the way, that they can support LAGs in order to provide every consumer of DSTs the same information. Finally, this paper presents the relevant state of the art, the conceptual modeling, and the relevant implementations. The paper comes to a close with a summary and a list of the remaining challenges.
A central content focus of the module AF A, "Industrial and Organizational Psychology,"
in the bachelor’s degree program in Psychology at the University of Hagen (FUH) [
The critical teaching of theoretical basics of psychological work design, which is mainly
done by reading and discussing relevant theories and research results, is unfortunately
mostly lacking in the experience of actual work design during studies. This can only be
achieved by experiencing and trying out diferent forms of work design. However, direct
confrontation, one’s own experience and trying out, as well as intensive reflection on
what is experienced is an essential prerequisite for the acquisition of action competencies
[
The primary learning objective of the planned didactic innovation is the acquiring
of competencies in occupational psychology in the sense of analyzing, evaluating, and
designing work tasks according to defined human criteria [
The project Gaming-and-Learning-Analytics4QBLM (GALA4QBLM) has the aim to
provide an AG within the so-called Knowledge-Management Ecosystem Portal (KM-EP)
[
"Good assessment planning begins by identifying learning outcomes for students. Planning then involves building programs and courses that provide students opportunities to achieve these learning outcomes. [...] Outcome statements must also be measurable and must target various skill levels within the cognitive domain." [18] In order to review and measure CQs, Learning Outcomes (LO) must be defined beforehand, as outlined in the ifrst step of the Assessment Lifecycle (ALC) [18] (see figure 1). These LOs should detail which skills, knowledge, abilities and values the learners have after the learning content. In a next step, the topics and contents to reach the LOs have to be taught to the learners [18]. "Learning activities must be designed to stimulate learning and to yield assessment data for the evaluation that follows" [18].
In a further step, these defined LOs have to be verified with the appropriate assessment methods [18]. "Assessment gathers data on what students do (what is learned) not on what instructors do (what is taught)" [18]. Here, direct measurement [18] methods as well as indirect methods [18] are to be used [18]. In this process, data, the so-called assessment outcomes, are collected to be analyzed by the teaching staf in the next step [ 18]. After the ALC, in the last step, it should be shown based on the assessment outcomes, how the students can improve their performance on the basis of these results.
Several Problem Statements (PS) can be derived from the objectives and motivation mentioned above. The first PS1 is that in the context of QBLM the term assessment goal is not defined in a way that they are machine-readable. In order to analyze the CQs to be obtained in the GALA4QBLM project, it must be possible, based on the ACL, for the objectives and outcomes from learning and assessments to be made digital and machine-readable. PS2 covers the issue that there is no structure within DSTs that defines the appropriate assessment goals and learning outcomes for each learning unit.
The PSs mentioned above result in the following Research Questions (RQs). RQ1: "How can assessment goals be formulated machine-readable for later automated analysis?" and RQ2: "How can a generic structure for describing assessment goals be implemented within DSTs for various modalities?"
As the basis of our research methodology, the multi-methodological framework of Nunamaker and Chen [19] is used for the structured research and development of information systems.
The framework is divided into four phases supporting diferent methodological strategies: Observation, Theory Building, System Development, and Experimentation. To achieve our research goal to answer our research questions, the methodological phases can be executed repeatedly in any order. It is also possible to return to previous phases.
Based on the research methodology of [19], the following Research Objectives (ROs) were derived from the RQs. RO1 is assigned to the Observation Phase (OP). This phase identifies a general approach for learning outcomes and assessment goals. In addition, suitable systems and tools are identified. RO2 is assigned to the Theory Building Phase (TBP). A concept is designed that shows what system components and interfaces are needed. The System Development Phase (SDP) moves the concept into a prototype and is assigned to RO3. The result of the SDP is evaluated in the Evaluation Phase (EP) in the context of a Cognitive Walkthrough (CW) [20]. Finally, the EP is assigned to RO4. In this phase, all RQs are evaluated. The remainder of this paper is structured according to the ROs. This means that in the State-of-the-Art section, the OP is described. In the Conceptual Design section, the TBP is described, and the SDP phase is presented in this paper in the Proof-of-Concept implementation section. Finally, in the Evaluation section, the EP is presented. Finally, the paper concludes with a summary and indications of future developments.
After describing the research methodology and the research questions the paper wants to address, we want to show in this section, which concepts already exist to provide some answers for our questions.
The so-called Didactical Structural Templates (DST) have been introduced in [21] and extended in [16]. As described, the DSTs are based on the IMS Learning Design (IMSLD) [22] and represent the didactical structure of a course and cannot only be used as didactical structure for creating courses. In fact, the DSTs can also be used as a didactical structure for a hybrid environment existing of a "classical" course with integrated applied gaming content just like a pedagogical structure for applied game which can be a webbased computer game or a VR/AR based game. Therefore, one DST can have diferent implementations.
The advantage of this approach is, that learners will be able to switch between diferent implementations of one DST whenever they want to and they have got the same learning progress as if they had used only one specific implementation of this DST. This means if learners like gaming, they can use the applied gaming implementation to work on the learning content. If it is easier for the learners to answer the self-tests or the final test – to stay in the exemplary stated pedagogical structure of a course – as e.g. multiple-choice quizzes, they can switch to a course within an LMS to answer the questions.
Therefore, the DSTs have the following hierarchical structure (figure 2 shows the structure in an exemplary DST):
Method: There are many diferent ways a person can learn or teach. Each learning method is a sequence of learning processes.
Play: is a key part of the learning design, which represents a teaching-learning process. Similar to a theatrical play with a sequence of acts. When an act is completed, the next act begins until the completion condition is met.
Act: An Act represents a series of simultaneous activities and activity structures.
Activity: is one of the core elements of learning design, which relates to many learning environments.
Activity Structure: Activities can be combined into an activity structure with sequence mechanisms or freely select-able structuring.
To give access to the DSTs, we have provided a RESTful API, which is described in [23].
With the DSTs we have a possibility to provide the didactical structure of learning content in a standard conform way. In the next section we will have a look into a de facto standard for exchanging learning content between applications.
The Bologna Reform aims to compare study programs [24]. For this reason, the Bologna Reform challenges Higher Educational Institutions (HEI) to align or build up their study programs, the study courses therein, the Learning Units (LU) therein, and the formats in which the courses are produced in a competence-oriented manner in the future [25]. To compare the resulting learning success and the acquired Competencies and Qualifications (CQ) in the courses, the corresponding learning data and results need to be analyzed and recorded digitally at the course level. In the context of this research, the term CQ will be used for Qualifications. CQs consist of competencies, skills, and Proficiency Levels (PL). To be able to measure and assess the CQs acquired by a learner in a course, a course needs to have clearly described course objective.
The course consists of learning units, each of which is completed by means of an assessment or verification of the CQ to be imparted. In order to be able to check the CQ to be imparted, a course objective must be defined first. A course objective, like a guideline learning objective, only specifies a learning field to be provided [ 26]. They simply indicate the area from which learners should draw their knowledge. At the next level down, the course is broken down into learning units. On the level of the learning units, certain CQs are conveyed with the help of individual learning units, which are checked with an assessment. For this purpose, so-called learning objectives are defined at the level of the learning unit. "The distinction between "learning goals" and "learning objectives" is actually pretty commonsensical: in this context goals generally refer to the higher-order ambitions you have for your students, while objectives are the specific, measurable competencies which you would assess in order to decide whether your goals had been met" [27]. "Learning objectives can then be broken down into small learning activities, or assessments" [28]. The learning goals are analogous to the rough learning goals [26].
At the assessment review level, each Learning Unit reviews the specific CQs to be taught. For this purpose, a learning unit can consist of any number of assessments. A precise assessment objective must be defined for each assessment. An assessment objective is a sub-objective of a learning objective, analogous to a precision learning objective [26]. On this level, the learning objective is divided into measurable sub-competencies and activities. Combined, diferent assessment objectives result in a learning objective (see ifgure 3).
A common possibility to cluster learning objectives is the taxonomy of Bloom (see ifgure 4). "Bloom’s Taxonomy [...] uses a multi-tiered scale to express the level of expertise required to achieve each measurable student outcome. Organizing measurable student outcomes in this way will allow us to select appropriate classroom assessment techniques for the course.[...] Bloom’s Taxonomy is a convenient way to describe the degree to which we want our students to understand and use concepts, to demonstrate particular skills, and to have their values, attitudes, and interests afected." [29] In terms of QBLM, CBL and this paper we refer to Bloom’s taxonomy of educational objectives for skill-based goals [29].
Before we can use the assessments for our purpose, we have to do some conceptual work, which will be done in the next section.
In this section, we will provide some conceptual work regarding to the open challenges from the previous section.
Bloom’s taxonomy divides the level of expertise into 7 categories (Perception, Set, Guided Response, Mechanism, Complex Overt Response, Adaptation, and Organization). Each of these levels represents a level of understanding. Linking the objectives for skills-based goals with the previously described assessment, a level of expertise for a learning unit can be stated out as an Learning Assessment Goal (LAG). The LAG describes the condition for a successful conclusion of a learning unit. A list of LAG is to be integrated into the DST to allow a formulation of finishing conditions for a learning unit respectively a Play, Act or Activity. It is to be assumed, that multiple LAGs (0-n) with diferent levels of expertise are part of a condition.
To determine an LAG is reached, actions of the learner are to be analyzed. The learner facing application for example a serious game captures the action of the learner and sends it to the Learning Record Store (LRS) [31]. Afterwards an analytics engine has to qualify the set of the executed actions of the learner and determine the level of expertise.
In following section, the concept of describing LAGs and the classification of learner actions to generate an assessment is described. Each action is to be classified for the assessment. A procedure for an assessment is inspired by of the software risk assessment method. ROAM (Resolved, Owned, Accepted, Mitigated) [32] classifies risks into four categories, which then leads to further actions. Transformed into context of learner action assessment, each item can be Resolved (executed positive), Avoided (not answered or skipped), Neutral (nor positive or negative answer) or Miscarried (Wrong answer or failed quest) (see figure 5).
The classification follows the quantification of the action (see figure 6). Each action afects the assessment(s). This means that each action has a learning value behind for each of the four assessment classifications, which is expressed by achievement points. For each level, each action defines achievement points to gain. This leads to tuple describing the points for an action: a-tuple (Resolved points, Avoided points, Neutral points, Miscarried points). An action has to refer 0-n a-tuple to describe the impact on 0-n assessments. On the other side, an assessment contains 0-n actions respectively for each action a classification and achievement points. Summarizing the points compared with the max amount of points, a performance percentage is calculated. This performance percentage is to be confronted with the levels of expertise. Depending on the assessment, 100% and 0% represents diferent levels of expertise. The level of expertise is then compared with the DSTs’ LAG, which is identified by the equal name. If the achieved level of expertise is equal or higher than the LAG, the learning unit is stated as successfully concluded.
The DSTs represent the didactical structure of a learning journey, which is has representation as for example a Moodle course. We can already preset condition- and goal CQPs for each IMS-LD element. These presets are transfered to the modality (for example Moodle), which is uses the information to generate a usable course for the learner. Like the LMS Moodle, a modality AG has access to this presets and can act in its specific way the provided information.
In order to deliver a learning analytics tool measurements about the games state, like the successful conclusions, it is required to transfer this information to all consumers of the DSTs. Therefore, it is necessary, to extend the DSTs for LAGs. This extension should also take place for every IMS-LD element. The possible values for the LAGs are described in the section before in figure 6, assessment classification.
In [17] it is shown that extention of the IMS-LD via the DST concept is able to cover the features to support the QBLM approach. In order provide the just described extension, we have to extend the IMS-LD specification by additional LAGs to cover not only the presets, but also the evaluation and measurement of learner interaction (changes are marked in green) shown in figure 7.
In the context of this paper, a new approach to adding Learning Assessment Goals to the DST was presented. For this purpose, RQ1 presented in this paper what how Learning Assessment Goals are to be defined in a learning objective context are and how they are defined in the context of QBLM. Furthermore, the concepts for RQ2 were presented. In the future, the concept will be implemented and evaluated and further specified within the Gala4QBLM research project. [14] B. Wallenborn, Entwicklung einer innovativen Autorenumgebung für die universitäre Fernlehre, Ph.D. thesis, FernUniversität Hagen, 2018. URL: https://ubdeposit.fernuni-hagen.de/receive/mir_mods_00001428. [15] R. Srbecky, M. Krapf, B. Wallenborn, M. Then, M. Hemmje, Realization of a framework for game-based learning units using a unity environment, in: Proceedings of the 15th European Conference on Game Based Learning, 2021. doi:10.34190/ GBL.21.055. [16] M. Winterhagen, M. D. Hoang, H. Lersch, F. Fischman, M. Then, B. Wallenborn, D. Heutelbeck, M. Fuchs, M. Hemmje, Supporting structural templates for multiple learning systems with standardized qualifications, in: EDULEARN20 Proceedings, 12th International Conference on Education and New Learning Technologies, IATED, 2020, pp. 2280–2289. URL: http://dx.doi.org/10.21125/edulearn.2020.0705. doi:10.21125/edulearn.2020.0705. [17] M. Winterhagen, M. D. Hoang, B. Wallenborn, D. Heutelbeck, M. Hemmje, Supporting qualification based didactical structural templates for multiple learning platforms, in: Advances in Software Engineering, Education, and e-Learning - Proceedings from FECS’20, FCS’20, SERP’20, and EEE’20, Springer, 2020, pp. 793–808. [18] W. College, The assessment cycle, 2022. URL: https://www.westminster.edu/about/ accreditation-assessment/cycle.cfm. [19] J. F. Nunamaker, M. Chen, T. M. D. Purdin, System development in information systems researchs, 1990. URL: http://gkmc.utah.edu/7910F/papers/JMIS% 20systems%20development%20in%20IS%20research.pdf. [20] C. Wilson, User Interface Inspection Methods: A User-Centered Design Method,
Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2013. [21] M. Winterhagen, M. Then, B. Wallenbrn, M. Hemmje, Towards Structural Templates for Learning Management Systems taking into account Standardized Qualifications, Technical Report, FernUniversität Hagen, 2019. URL: https : / / www.researchgate.net / publication / 340601087_Towards_Structural_Templates_for_Learning_Management_Systems_taking_into_acc [22] I.G.L.-Consortium, Learning design specification, 2003. URL: https : / / www.imsglobal.org/learningdesign/index.html. [23] M. Winterhagen, D. Heutelbeck, B. Wallenborn, M. Hemmje, Integration of pattern-based learning management systems with immersive learning content, in: CERC 2020 Proceedings, 2020, pp. 211–221. URL: http://ceur-ws.org/Vol-2815/ CERC2020_paper13.pdf. [24] C. Baumann, T. Benzing, Output-orientierung und kompetenzformulierung im bologna- prozess, 2013. URL: https://bit.ly/3JTVtKc. [25] N. Hericks, M. Rieckmann, Einfluss der Kompetenzorientierung auf die Tätigkeit von Hochschuldozent/ -innen, Springer Fachmedien Wiesbaden, 2018. [26] R.-U. Bochum, Typen und stufen von lernzielen, 2022. URL: https://dbs-lin.ruhruni-bochum.de/lehreladen/planung-durchfuehrung-kompetenzorientierter-lehre/ lehr-und-lernziele/typen-und-stufen/. [27] H. University, On learning goals and learning objectives, 2022. URL: https: //bokcenter.harvard.edu/learning-goals-and-learning-objectives#:~:text=The% 20distinction % 20between % 20 % 22learning % 20goals,to % 20decide % 20whether % 20your%20goals. [28] LearnWorlds, Learning goals & objectives in course design: How to prepare a great lesson plan, 2022. URL: https://www.learnworlds.com/learning-goals-objectives/. [29] T. C. for Teaching, Learning, Bloom’s taxonomy of educational objectives, 2022.
URL: https://teaching.charlotte.edu/services-programs/teaching-guides/coursedesign/blooms-educational-objectives. [30] U. of Florida, Bloom’s taxonomy, 2022. URL: https://citt.ufl .edu/resources/thelearning-process/designing-the-learning-experience/blooms-taxonomy/. [31] xAPI.com, What is the experience api?, 2022. URL: https : / / xapi.com / overview/?utm_source=google&utm_medium=natural_search. [32] I. Software, Roam & risk management under safe, 2022. URL: https : / / content.intland.com/blog/agile/safe/roam-risk-management-under-safe.