The goal of research is to determine whether modern analytical tools for Moodle courses data can be useful for ordinary educators. A considerable amount of literature is examined. Possible analytical tools were classified by its suitability from the teachers' point of view. Presented case study demonstrates the potential of analytical tools can be used by teachers. By authors' approach, it is shown that even simplest analytic tools allow to learn a lot about the online activity of students in the online course and get interesting results.
Today, the main features of modern students’ requests are becoming participants in the organization of their training, creating an individual educational route, personification of the educational process, providing opportunities for initiative, taking into account the social mechanisms of interaction: competition, cooperation, mutual learning and assessment, as well as building a creative space for the student.
Modern students’ activity is organizing by the digital educational environment as learning units, that should be provided to the teacher at the same time in a convenient, compact and demonstrative forms, such as mind maps, graphs and diagrams that should reflect the relationships and patterns between identified learning activities. Huge amount of information about students “digital traces” are storing in open education platforms (Coursera, edX, NPTEL, FutureLearn, Open Education, Universarium, Lectorium, etc.) and learning management systems (Moodle, iSpring, Mirapolis, ShareKnowledge, TeachingBase, WebTutor, etc.).
Nowadays an actual task become studying the capabilities of the educational analytics to build a high-quality educational process in a digital learning environment. The information contained in system log files is very useful in making predictions of learning processes. Mostly it should help in marking about the success of training (predictive analytics ), in describing what has it happened (descriptive analytics ) or in making conclusions about why have it happened (diagnostic analytics ). USA Spain Mexico Germany
Brazil France Russia
Italy Great Britain
India 11075 10237 8763 6165 5956 5586 4573 4088 4057 3392
Percentage of educational institutions sites (%) 23,7 26,1 23,9 19 35,1 41 31,3 26 24,6 24,7 The main analytics’ aim is to get informed conclusions about what needs to be done to improve the educational process.
LMS Moodle system is one of the most powerful, popular and, that’s why, widely used systems in the world for organizing distance learning scaffolding for courses. In the Russian Federation, as well as in some developed countries, the Moodle platform is actively used in official educational institutions (Table 1). The popularity of Moodle is because it is free and has detailed technical documentation, widest possibilities of customizing the interface and functions, as well as a flexible system of statistics and reports in comparison with other free LMS (such as ATutor, Ilias, Diskurs).
Further in the paper, the world experience of data analysis from Moodle is given in Section 1.2, a different groups of tools for Moodle data analytics are identified in Section 1.3, and a case study on the analytics of the digital behaviour of the first-year students in Moodle-course “History” is presented in Section 2. An experimental research has made in the Herzen State University of Russia in collaboration with Peter the Great St.Petersburg Polytechnic University in 2019. The foregoing indicates the relevance of acquiring and transferring experience in analyzing the data accumulated in the LMS Moodle. 1 1.1
We conducted some experimental study on the analytics of data collected in the distance course “History” in LMS Moodle in the period of investigations from February 1 to June 30, 2019 (the 2nd semester of the 2018-2019 academic year). Number of respondents was more than 3 thousand people. The course is intended for all the 1st year students and lasts for one semester. The course contained 8 video lectures, with presentation and self-control test for each one, description of tasks for seminars, 3 control tests and 1 final test. 1.2
Educational analytics implies the application of knowledge from various fields, such as statistics, recommendation systems, data mining, psychometry, learning technologies, etc. The main purpose it applied is to analyze data from the educational environment: in [Nilgu¨n, 2018] and [Noskova et al., 2018], by using the correlation and cluster analysis, there were studied the strategies of educational activities of students and security in social networks, in [Almutairi et al., 2019] and [Nesterov et al., 2019] predicting students’ academic performance and main behavioural features of learning outcomes were evaluated from distance courses by using data mining methods, in [Spatiotis et al., 2019] and [Fomin et al., 2019], text-mining technologies were used to evaluate educational information resources.
Nowadays special actively developing area is highlighted by science - the educational data mining / EDM, which is aimed at developing data mining methods for making decisions in the field of education. The possibilities of EDM are broadly considered in the science community [Bakhshinategh et al., 2018]. Approaches to training educators in data analysis are considered in [Piotrowska & Terbusheva, 2019]. As we are working with Moodle-courses we identified the main areas of the research in the field of EDM conducted in the scientific community directly with the data of the Moodle environment: study of the relationship between the activities of students in Moodle and their successes [Kadoi´c et al., 2018], [Mogus et al.], [Stiller et al., 2018]; identification of behavioral strategies in online learning [Akcapınar et al., 2015], [Bogar´ın et al., 2015]; analysis of gender differences in education in LMS [Kadoi´c et al., 2018]; visualization of the extracted data [Rybanov et al., 2013], [Aguilar et al., 2008]; instructions for preprocessing Moodle data and examples of the application of mining techniques (visualization, clustering, classification, associative rules, etc.) to such data [Romero et al., 2008].
In these papers, the analysis was carried out mainly on the data from questionnaires and a system log files. The following tools were used for analysis: Excel, Statistica (for calculating statistical indicators and correlation analysis), RapidMiner, Weka, Deductor (for clustering and other data mining methods), self-developed tools that include Moodle data processing and some analysis and visualization capabilities). Many works use several kinds of different tools and methods to perform the study [Lahbi & Sabbane, 2020].
A more complete overview of the possible tools for Moodle data analysis is given in the next section. 1.3
Standard Moodle tools. These tools are reports (competency breakdown report, logs, participation report, activity report, grade report, statistics) and simple analytical models (such as students at risk of dropping out, upcoming activities due) that are included in the basic version of Moodle and available to all teachers. Standard Moodle tools provide some statistics on student activity in the course (for example, students viewing various elements of the course). Reports are easy to get, but they often don’t have sufficient functionality and visibility to make decisions. Detailed information on the capabilities of these tools is available in the Moodle documentation [Moodle Reports].
Moodle plugins. Such plugins are additional functional extensions of Moodle, which can be installed only by the site administrator. There are paid solutions, but at the same time a large number of free solutions are available in the Moodle plugins directory. Examples of various analytic plugins: for additional statistics (IntelliBoard, Courses Usage Statistics), for visualization (Heatmap, Analytics graphs, Daily usage), for pattern discovery or student at risk predictions (SmartKlass, Inspire) and others. But individual teachers are not able to select, install and research plugins, which has grate limitations in their use and usefulness.
Specialized external tools. These tools are designed specifically for analyzing data from
Moodle courses. Despite the number of descriptions of such tools in the literature
Multifunctional external tools. These are various data analysis environments that provide wide functionality and capabilities for high-quality and multi-aspect data analysis. But to use them, it is often necessary to have specialized knowledge in the field of IT and Math (statistical methods, data mining, programming languages, etc.). In this category we distinguish 5 subgroups: computing environments, data mining programs, development environments, database management systems and web-site analytic tools. Among these tools, there are freely distributed ones with a fairly simple interface, for example, the Weka program that implements data mining methods.
In Table 2, we tried to compare the different groups of tools of data analysis by various parameters from the point of view of the teacher’s convenience. All these solutions fulfill their role of big educational data transform into some more understandable information. However, they are limited in terms of customization and do not provide sufficient resources for various situations. Our approach described below in this paper expands opportunities through an open solution that enables the development of resources to meet needs identified by specific educational contexts. 2
To show how ordinary educator can study students digital behavior with the help of simplest tools we provide the further case study. The general scheme for solving the paper problem is shown in figure 1.
It’s possible to understand the intensity of the course usage using the standard Moodle report “Activity report”. To get data on views of course elements for a certain time, one can set a filter. In this purpose we set the period from February 1 to June 30, 2019, that corresponds to the spring semester of the 2018-2019 academic year. Already at this stage of the study, a significant shortcoming in the design of the course regarding further analytics was revealed: the activity report does not contain data on viewing video lectures, because the videos were not included as Moodle elements of the course.
Visualization (using MS Excel program) of the activity report information allows us to see that the educational activity of students decreases throughout the semester (Fig. 2). The number of users viewing presentations for video lectures and doing tests for self-control decreased by the end of the semester by 3,3-3,8 times (by 70-74%). At the same time, control tests are performed by students. The number of viewers of control tests also decreases, but not as significant as the number of viewers of the rest of the digital content. The third test out of four was viewed by 7% less students than the first test, and the final test – by 24% less students than the first.
Analysis of the time activities on the course of 36 students of one group with KEATS Analytics tool [Konstantinidis et al., 2013] showed that mainly students entered the course in the middle of the week; the time from Tuesday to Friday was the most active (Fig. 3). On Saturday and Monday, there was a decline in activity, on Sunday it was average. At the same time, students most often worked in the course during the morning hours from 8 to 12 and in the evening between 17 and 23 hours. This information can be used in planning educational activities. For example, it is better to plan online testing or discussion during hours of high students’ activity. At this time most students are likely to be able to take part in online activities.
We also carried out a correlation analysis between some statistical indicators of activity of this group of students on the course and their final scores (Table 3). To do this, we first combined the values of students’ activity indicators obtained in KEATS Analytics tool with their total score in one file and then we used Excel to calculate the correlations. Among such indicators: number of total views of the course (i.e. any its element); number of various actions; the number of different course pages that have been viewed; the number of different dates a student was active on the course. The results of the analysis allowed us to conclude that the final score depends more on the number of different actions on the course (correlation coefficient 0.77) than on the number of total course views (0.51), the number of different pages visited (0.54) or the number of active days (0.58). For further analysis, we combined information on the activities and performance of student by manually copying the data from the “Participation report” and “Grade Report” reports into the Excel spreadsheet. Since the report “Participation in the course” is built in Moodle for each course element separately, it is necessary to make this report as many times as there are elements in the course for which we would like to take into account the information. Data from all built reports were copied to one spreadsheet. Further, data on students’ grades from another report were added to the same table.
Manually collecting data for many students is laborious, so it is important to develop specialized software that automate this process.
In order to identify groups of students based on the collected data, we performed a cluster analysis in the Weka program. In order to perform clustering in Weka, it’s only needed to load the data on the Preprocess tab and select the clustering algorithm on the Cluster tab. Since we hadn’t elaborated an assumption about the number of clusters, we chose an SelfOrganizingMap algorithm that didn’t not require setting the number of clusters for execution. Fragment of clusterer’s output (for 6 of 36 attributes) is shown in fig. 4.
As you can see, the algorithm divided the students into 3 clusters. The analysis of these clusters allowed to distinguish three groups of students.
1. Students who did not access the course (or accessed only a few times) and did not perform control tests (cluster 3). 2. Students who performed only control tests and showed no other activity in the course (cluster 1). 3. Students who performed some educational activities (such as viewing presentations for video lectures, passing tests after lectures for self-control) and did control tests (cluster 2).
not identical, but similar results. This requires the development of recommendations for working with each group, and it is additionally necessary Figure 4: to study students’ motivation and approaches to teaching students in each group.
In the study group, the first group includes 5 control tests. In the corresponding cluster 1 on out of 36 students. To reduce the number of such Fig. 4 min value for the scoreTest2 is zero. It students, you can send them reminders about the means that this group also includes students who course and the upcoming first assessment, if an missed some of the control tests and by default, inactivity was detected. The Moodle functional- they have 0 points for it. Other students of the ity allows to select these students in the list and group passed the tests quite successfully, no worse send them messages. The opportunity to send than students of the third active group. But we messages is available in the report “Participation do not have information on viewing video lectures, in the course”. But to identify the student’s pro- since these videos were not included as elements of longed inactivity, it needs to look at the reports the course (as opposed to presentations for video for a number of elements of the course, that is lectures, seminars, tests for self-control). difficult in cases of large groups and high employment of teachers.
As for the students of the second group, they do not view the course materials, they only pass The previous discussion suggests two variants: students of the second group did not watch the video lectures. In this case, several questions are arising: “How was the learning material studied?”, “Why was it learned outside the LMS?” students of the second group watched the video lectures. In this case, we could conclude that video lectures watching is the main criterion for a successful assessment.
Our review of research in the field of Moodle data analysis and possible tools for this domain shows that there is no universal tool that provides extensive functionality. To understand the learning and draw more meaningful conclusions, researchers are forced to use several tools process. It allows the user to conduct a high-quality experiment on the analysis of data accumulated in LMS Moodle. In addition, research is usually carried out by IT professionals, because it is too complex for ordinary non-IT educators.
But our experimental study demonstrates that even the use of the simplest tools, which teachers can understand, allows to learn much about the online activity of students in the online course and get interesting results.
The usefulness of learning analytics and educational data mining methods has been repeatedly shown. Further, it is necessary to ensure the widespread use of these opportunities by teachers. For this purpose, it is necessary on the one hand to train ordinary teachers to work with various methods and tools for data analysis [Piotrowska & Terbusheva, 2019], and on the other hand - to provide teachers with multifunctional tools with a simple interface and clear instructions.
The research was supported by the Russian Science Foundation (RSF), Project “Digitalisation of the high school professional training in the context of education foresight 2035” (No 19-18-00108). [KEATS analytics] KEATS https://clck.ru/My8h6.
analytics: program for analysis
logs.
URL: [Moodle Reports] Moodle documentation. https://docs.moodle.org/38/en/Coursereports
reports.
URL: [Mogus et al.] Mogus A. M., Djurdjevic I., and Suvak N., The impact of student activity in a virtual learning environment on their final mark // Active Learning in Higher Education. 2012. Vol. 13.
No 3. Pp. 177–189. [Nesterov et al., 2019] Nesterov S.A., Smolina E.M. Metody intellektual’nogo analiza dannyh v zadachah ocenki rezul’tatov distancionnogo obucheniya //Sistemnyj analiz v proektirovanii i upravlenii. 2019. No 3. S. 406-412. (In Rus) == Нестеров С.А., Смолина Е.М. Методы интеллектуального анализа данных в задачах оценки результатов дистанционного обучения //Системный анализ в проектировании и управлении. 2019. No 3. С. 406-412.