=Paper=
{{Paper
|id=Vol-3041/180-184-paper-33
|storemode=property
|title=The Analysis of the Educational Measurement Results and Its Providing as “Software-As-A-Service” Solution in Elearning
|pdfUrl=https://ceur-ws.org/Vol-3041/180-184-paper-33.pdf
|volume=Vol-3041
|authors=Yulia Lavdina,Oleg Gustun,Nikolay Budaragin,Nataliya Leonova,Alexey Modyaev
}}
==The Analysis of the Educational Measurement Results and Its Providing as “Software-As-A-Service” Solution in Elearning==
https://ceur-ws.org/Vol-3041/180-184-paper-33.pdf
Proceedings of the 9th International Conference "Distributed Computing and Grid Technologies in Science and
Education" (GRID'2021), Dubna, Russia, July 5-9, 2021
THE ANALYSIS OF THE EDUCATIONAL MEASUREMENT
RESULTS, AND ITS PROVIDING AS “SOFTWARE-AS-A-
SERVICE” SOLUTION IN ELEARNING
Y.K. Lavdina1,a, O.N. Gustun1,b, N.V. Budaragin1,c,
N.M. Leonova1,d, A.D. Modyaev1,e
1
National Research Nuclear University MEPhI (Moscow Engineering Physics Institute),
31 Kashirskoe Shosse, Moscow, 115409, Russian Federation
E-mail: a julia_lavdina@mail.ru
In modern eLearning systems, educational measurements are used both to evaluate the students’
achievements and to control the learning process. However, eLearning systems usually have
comparatively trivial embedded features for analyzing measurement results, which are not of
considerable interest for sufficient statistical research of the assessment tools quality. To identify the
characteristics of assessment materials such as reliability, homogeneity, discriminatory power,
validity, and others, researchers are forced to get dump from the database of eLearning system. And
then they use third-party software to perform required data processing operations and calculations.
This makes it difficult to analyze the measurement results during the measuring itself, for example, in
adaptive testing. We propose the approach to organizing and performing measurement results analysis
by using the software-as-a-service (SaaS) model for cloud computing. The SaaS user is provided with
the set of necessary tools for conducting full-fledged statistical analysis in real time. They also get the
access to customizable applications for implementing their own measurement procedures (including
adaptive ones).
Keywords: educational measurement, eLearning, SaaS, assessment tool, statistical analysis
Yulia Lavdina, Oleg Gustun, Nikolay Budaragin, Nataliya Leonova, Alexey Modyaev
Copyright © 2021 for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
180
Proceedings of the 9th International Conference "Distributed Computing and Grid Technologies in Science and
Education" (GRID'2021), Dubna, Russia, July 5-9, 2021
1. Adaptive control in eLearning
Modern eLearning systems often include a various learning assessment tools designed to
monitor the learning process and take educational measurements. In advanced eLearning systems, the
results of such measurements are used to control the learning process, the control aim of which is to
increase the effectiveness of learning by individualizing educational trajectories. In this case, the
functions of monitoring and control of the learning process are performed by a computer system that
builds the learning trajectory of educational course individually for each student.
A necessary condition for solving the control problem is the use of feedback in the dynamics
of the system. Moreover, the learning process should be organized so that regular measurements of the
students’ characteristics related to the success of their education at the current stage are taken. This
measurement information is transmitted through a feedback channel and is used to make a decision to
form and correct the current setting action, which leads to an increase in the success of learning
objectives.
The learning process control problem can be effectively solved by using methods of adaptive
control. The main feature of adaptive control is the use of information about the current state of the
object to adjust the control device which will form the controlling action to achieve the control aim
[1].
Figure 1 shows the chart of the adaptive control system of the learning process. The measuring
device estimates the students’ characteristics and transmits the results of measurements x(t) through
the feedback loop [2]. This data are used to adjust the adaptive control device, which forms the
controlling action u(t).
Figure 1. Adaptive control system in eLearning
2. Educational measurement
Educational measurements are a process of interaction with students, as a result of which the
latent parameters of students' models are set the numerical values. At various stages of this process,
the measuring device uses the appropriate measuring tools. The measurement accuracy is determined
by the quality indicators of the measuring materials used by the measuring device.
When creating new measuring instruments, it is necessary to ensure their compliance with the
measurement objectives and the conditions of the measuring process. The quality and accuracy of
measuring instruments are determined based on the analysis of the results of educational
measurements. During the measurement, data on the actions of students, their reaction to the presented
measuring materials are recorded and accumulated in databases. To analyze the collected information,
special statistical methods are used to determine the characteristics of control and measuring materials.
If we consider tests as a means of measurement, then the most important and frequently used
indicators of their effectiveness include difficulty, reliability, homogeneity, and discriminatory
power [3].
181
Proceedings of the 9th International Conference "Distributed Computing and Grid Technologies in Science and
Education" (GRID'2021), Dubna, Russia, July 5-9, 2021
Difficulty is determined by the proportion of students who give the correct answer. The
difficulty of a test containing several items is calculated as the total score mean, taking into account its
variance [4].
Reliability index shows the relationship between true and observed test scores and it is
determined by a correlation coefficient. Reliability also characterizes the internal consistency of the
test. It is calculated on the base of variance-covariance matrices by different methods depending on
conditions and aims of testing.
Discriminatory power is the index of how effectively the item discriminates between students
who are relatively high on the criterion of interest and those who are relatively low [4]. To calculate it
in the general sample, it is necessary to distinguish two corresponding groups of students and perform
a comparative analysis.
Homogeneity shows how closely the items are related to one another and is calculated based
on cross-correlations. Assignments with high homogeneity are aimed at measuring the same
phenomenon.
Homogeneity also determines the interchangeability of different test item variants. In this case,
the homogeneity criterion makes it possible to establish the similarity (or difference) in the parameters
of the test items and to give recommendations for correcting their variants.
To determine all the above indicators of test materials, it is necessary to go through several
stages:
1) preparation of measuring instruments and planning of measurements based on the available
information;
2) collection of measurement process data and organization of their storage;
3) data processing and preparation for analysis;
4) analysis and evaluation of the parameters of test materials.
Modern eLearning systems with varying degrees of success cope with the solution of the tasks
performed in the first three stages. But most of them do not have sufficiently developed tools to
conduct a full-fledged statistical analysis of the test materials quality. To research measuring materials
and evaluate their parameters, third-party tools have to be used.
To do this, you need to get dump from the eLearning database and independently process the
data. And in order to perform the necessary calculations using third-party software, the researcher
must have the necessary professional training. In addition, this method does not enable to analyze the
measurement results in the course of the measurements themselves, which imposes restrictions on the
choice of types of measurement procedures.
3. SaaS solution for measurement results analysis
Analysis of measurement results can be organized and conducted by using a software-as-a-
service (SaaS) service model for cloud computing. The eLearning system connects to the SaaS service,
which provides a set of necessary tools for statistical analysis of results, including in real time. In
addition, the SaaS user gains access to custom applications to produce their own measurement
procedures.
The scheme of using the service in an eLearning system is illustrated in Figure 2. To conduct
an analysis using a SaaS service, it is necessary to go through the preparatory stage, at which the
service is prepared and configured. Via the software interfaces, the SaaS application receives the
identification data of students, information about the structure of the course, the description of the
tasks that students will complete. The description of the tasks contains:
task type;
structure of task and it items;
parameters of items;
conditions of task administrating;
correct responses (or answers) to task items.
182
Proceedings of the 9th International Conference "Distributed Computing and Grid Technologies in Science and
Education" (GRID'2021), Dubna, Russia, July 5-9, 2021
As a result of preparation on a remote server, the structure of the data warehouse about the
actions of students in the measurement process and its results is created, and a place is reserved for
storing the specified data.
Figure 2. Using of SaaS for measurement results analysis
The next stage is getting educational data in the learning process, which is stored on the server
in the prepared structure. This data can come both in joined packages after the completion of the
measurement process, and in separate frames in real time. Each of such frames contains at least:
student ID;
task (or item) ID;
response (or answer) ID;
time.
At the final stage, the SaaS service generates a report on the analysis of the received data in
accordance with the settings that the user made at the first stage. In addition to numerical information,
this report may include tables, graphs, charts that enable to visualize the analysis results. If the
analysis is carried out in real time, then numerical and graphic information is displayed on the monitor
screen, and events are recorded in the log.
183
Proceedings of the 9th International Conference "Distributed Computing and Grid Technologies in Science and
Education" (GRID'2021), Dubna, Russia, July 5-9, 2021
4. Conclusion
The use of applications functioning according to SaaS model makes professional analysis of
educational data available to a wide range of users. The development of SaaS applications and
software and its frequent use to enhance the functionality of eLearning systems can take educational
measurement research to a new advanced level.
References
[1] Sastry S., Bodson M., Adaptive Control: Stability, Convergence, and Robustness, Prentice Hall,
New Jersey, 1989.
[2] N.M. Leonova, Sintez algoritmov adaptivnogo strukturno-parametricheskogo upravleniya
obrazovatelnoy deyatelnostyu: Pod red. A.D. Modyaeva, MEPhI, Moskva, 2006.
[3] Secolsky C., Denison D. B., Handbook on Measurement, Assessment, and Evaluation in Higher
Education, 2nd. ed., Routledge, 2018.
[4] Crocker L., Algina J., Introduction to Classical and Modern Test Theory, Harcourt Brace
Jovanovich, New York, 2006.
184