Assessment is a very relevant part of any instructional process. Its results, when adequately provided, can improve students' learning. Assessment results could also be used to guide teachers in order to align their course design with the students' results. However, the analysis of the assessment results and the extraction of useful conclusions for both teachers and students is not an easy task and computerized tools can be very helpful to provide automation of these processes. Moreover, the information regarding assessment of students can come from many sources and in the case of blended learning environments, the number of sources increases. In this paper, a system aimed at supporting Competencebased Learning in blended learning environments (COBLE) is presented. The paper presents the system's architecture and its main elements. It is specially described how information from different sources can be gathered together into the system in order to update its models and analyze the assessment data as a whole.
Assessment is a key element of the instructional process [
Current educational approaches in Higher education are focusing on the one hand on
blended learning [
Copyright © 2018 for this paper by its authors. Copying permitted for private and academic purposes
approaches and technologies [
In this kind of educational environment, the information regarding assessment in a
course can come from many sources and it should be integrated in order to analyze it
and provide adequate feedback to users [
This paper describes the main components of COBLE and how its models can be updated with information from other learning environments or tools. This paper first presents the architecture of COBLE and then it describes its domain and learner models. After that, the paper centers on the aspects related to the data extraction from systems the students have interacted with, on the loading of that information into COBLE and on how to update the learner model using that information. Next, some examples of visualizations provided by COBLE are presented. The paper finishes with some conclusions and future work. 2
COBLE, like any learning support system, represents the domain and learner models (see Fig. 1) which structure is detailed in the following section. It has also two main modules (see Fig. 1) that interact with the information stored regarding the domain and the learners. These two modules interact with those models in order to support competence-based learning environments.
……
In a blended learning environment, students can interact with different tools and their
marks can come from many different sources. For example, the students can use online
learning tools such as the Moodle Learning Management System (LMS) or
e-assessment systems such as AdESMuS [
COBLE is able to integrate all this information in its learner and domain models as it will be described in the following section. In order to do that, COBLE incorporates a Data Management module.
Finally, a Visualization module is in charge of applying visual learning analytic techniques in order to provide visualizations for both students and lecturers. The particular aspects of the provided visualizations are described in the following sections. 3
The domain model of COBLE represents the course competences taking as a basis the
Competence-based knowledge space theory (CbKST) [
The competences are also related to the course topics, resources and to a set of assessable items or exercises that will allow to determine the mastery level on each competence. The system also includes the possibility of using evaluation rubrics for the assessment of the assessable items. 4
The Data Management module allows lecturers to interact with the domain and learner models in order to load or update the information stored in them. The module has an interface that allows lecturers to directly define or update both the domain and the learner model. COBLE also allows lecturers to import the information related to the interactions of the students with the domain extracted from an external system.
In order to do that, the user can interact with the interface to introduce the information individually but it can also load information provided in a text file with commaseparated values. 4.1
Several systems allow the extraction of information related to the interactions and performance of the students related to different resources. For example, Moodle allows lecturers to export information related to grades obtained by the students in the course’s assignments and quizzes among other items. Fig. 3 shows the lecturer selection of items to be exported from Moodle’s gradebook (on the left) and the generated text file with comma-separated values loaded in a spreadsheet (on the right). Even when lecturers do not use any management tool to evaluate the student tasks, they usually store this information in some document, frequently in a spreadsheet.
In order to be able to load into COBLE the extracted data, it must be stored in a comma-separated values text file, which is one of the available formats in Moodle and spreadsheets among other platforms. The only compulsory field in the file is the student’s e-mail because it is used as the student’s identifier in COBLE. This makes it easier to import and integrate data from different systems as it does not compel the student to have the same user name in the different systems being used.
Currently, COBLE supports the loading of comma-separated values text files where each line represents the information of a particular student. This format was established because it is the same structure in which Moodle data can be exported and because it also matches the structure of the spreadsheets used by many teachers. However, it can be easily extended to incorporate other file structures. 4.2
Once the information to integrate in the system has been extracted, the Data Management module is in charge of incorporating the information into the domain and learner models.
As it can be seen in Fig. 4, the first thing the user must do is selecting the text file (Fig. 4, a) that contains the information to be loaded into COBLE.
When the file has been selected, the user is provided with a preview of the information of the file (Fig. 4, b) to assure that the information and its structure are correct.
IMPORTACIÓN DE DATOS e The file can have more information than the one the lecturer wants to load into the system. For example, when exporting from the Moodle’s gradebook, the whole information about the student is incorporated, including the system’s internal identification for the student or the institution’s identifier. Thus, the lecturer has to select from all the fields that appear in the file, which ones should be incorporated into COBLE (Fig. 4, c). The user must also identify which is the field that contains the student’s email account as that will be the identifier in COBLE (Fig. 4, d).
Sometimes lecturers might have defined completely in COBLE the tasks whose marks are going to be loaded, but sometimes it will not be the case. In this last case, COBLE warns the user about the tasks that are not defined in the domain model and allows the updating of the domain model defining the tasks and their relationships in the system (Fig. 5). The information related to the assessment of the different tasks, is stored in the historical data part of the learner model. Taking into account this information, COBLE must update the knowledge and competence mastery levels of the learner model.
Literature reports different means to update knowledge and competence models [
BKT represents the changing knowledge state during skill acquisition based on the results of the student opportunities to apply the skill. In this model, the student knowledge is represented by a set of variables, one per topic, that describe the probability that the topic is either mastered by the student or not. This model considers the student’s performance on an activity related to a topic to update the probability of the topic being mastered. The model does not take into account the possibility of the student forgetting a topic, but it considers that the student might succeed by chance or fail even when mastering the topic. The model entails four parameters: • p(Lo) or p-init, which represents the probability of the student knowing the topic beforehand. • p(T) or p-transit, which represents the probability of the student having learnt the topic. • p(S) or p-slip, the probability the student makes a mistake when working on the topic. • p(G) or p-guess, the probability that the student has a lucky guess on an activity related to an unknown topic.
These parameters allow updating the students’ mastery for each topic. The initial probability of a student u mastering topic k is set to the p-init parameter for that topic (1). (1) (2) (3) ($)'& = () )' After that, computing the conditional probability depends on whether or not the student succeeded on the task related to that topic.
If the student correctly solved the task, the conditional probability is computed using (2). In this equation, the probability of knowing the topic before the assignment and not slipping is divided by the probability of correctly solving the activity (which also considers succeeding by chance). If the student does not pass the task, the conditional probability considers the probability of knowing the topic before the assignment and slipping divided by the probability of failing (3). The conditional probability obtained using (2) or (3), depending on whether the student succeeded or not, is used in (4) to update the probability of the student knowledge on the topic. Equation (4) considers the probability of knowing the topic before the activity plus the probability of not knowing it and having learnt the topic.
(*)'& = (* − 1|)'& + (1 − (*|)'&)() (4) The described model is used in COBLE to update the students’ knowledge level and the competence model in the Learner Model. 5
The main objective of COBLE is to support the main user types in competence-based learning environments. For that, it currently incorporates a Visualization module that provides different visualizations for students and lecturers. The visualizations provided by this module are intended to promote students’ self-reflection and to help lecturers to adapt their course plan.
COBLE provides diverse simple visualization of the knowledge level of the students,
but it also provides richer visualizations of the competence acquisition level. This
information is always provided in context, providing what is called an Open Social
Student Model [
Next, some of the visualizations COBLE provides are described.
For example, lecturers can consult the mastery level of a particular student on each competence contextualized as a bullet in a boxplot chart that represents the mastery level of the whole class.
Lecturers can also analyze the acquisition level of different competences for a group of students. For example, the system generates a violin-plot chart where the distribution of the group’s students having each level of mastery for each of the competences is shown (see Fig. 6). Competences’ acquisition level for the group Lecturers have also the possibility to see the information related to a particular competence, observing the evolution of the mastery level on that competence. For example, Fig. 7 shows the evolution of a student in a competence and compares it to the class or group the student belongs to.
COMPETENCE ACQUISITION Competence Competence1
From Week 1
To
Week 30 COBLE also provides some customizable visualizations for lecturers, which can help them detecting interesting information in order to organize their lectures. For example, a lecturer interested in determining which competence to reinforce in the following lecture, could be interested in detecting competences with lower mastery levels than expected. Fig. 8 shows the visualization that summarizes the percentage of students that have mastery levels above the threshold established by the lecturer (0.8 in this case) for each competence. Show competences’ acquisition percentages according to the threshold: 0.8 Competence 3
9% Competence 2 Competence 1 96%
91% 40% 50%
YES NO 62% 38% 4% 0% 10% 20% 30% 60% 70% 80% 90% 100%
Fig. 8. Customizable interface to obtain group information 6
This paper has presented the proposal of the system COBLE, which has been designed to support competence-based learning. The system capabilities have been improved providing a module that allows integrating domain and student information coming from different systems.
The import characteristics have been evaluated, importing student interactions with Moodle for several courses, and also from files created by lecturers to store information related to face to face interactions with the students.
Future work will be devoted to two different aspects. On the one hand, it will be analyzed the possibility of importing the whole domain data from files extracted from other systems. On the other hand, we plan to design and implement a new module that will use recommendation techniques in order to provide customizable automatic alerts and recommendations both for students and lecturers.
This work is supported by the Basque Government (IT980-16), the UPV/EHU (EHUA 16/22) and the Office of the Vice-Chancellor for Innovation, Social Engagement and Cultural action of the UPV/EHU through the SAE-HELAZ (HBT-Adituak 2018-19/6).