=Paper=
{{Paper
|id=None
|storemode=property
|title=Methods and Technologies for the Quality Monitoring of Electronic
Educational Resources
|pdfUrl=https://ceur-ws.org/Vol-1356/paper_109.pdf
|volume=Vol-1356
|dblpUrl=https://dblp.org/rec/conf/icteri/Kravtsov15
}}
==Methods and Technologies for the Quality Monitoring of Electronic
Educational Resources==
https://ceur-ws.org/Vol-1356/paper_109.pdf
Methods and Technologies for the Quality Monitoring of
Electronic Educational Resources
Hennadiy Kravtsov1
1
Kherson State University
27, 40 rokiv Zhovtnya St., 73000, Kherson, Ukraine
kgm@ksu.ks.ua
Abstract. Support of the quality of training is one of the main objectives of the
university system. The results of modeling the quality management system of
electronic educational resources (EER) on the basis of the analysis of its elements
are presented. The subject of the study is the EER quality monitoring. Technolo-
gies for EER quality monitoring are based on the method of expert evaluations.
The criterion of EER quality is considered as the weighted average value of qual-
ity indicators. The weights of EER types and indicators of EER quality for their
types are evaluated in pedagogical experiment. Results of experiment confirmed
the assumption that the method of expert evaluations can be the basis for the EER
quality monitoring. Concordance method is used to assess the degree of consen-
sus of experts on the factors: weights of EER types, parameterization of EER
quality indicators, and weighted average criterion of EER quality. The model of
quality management system is shown in the example of assessing the quality of
the distance learning system resources.
Keywords. quality management system, electronic educational resources, moni-
toring of quality, distance learning system «Kherson virtual university».
Key Terms. QualityAssuranceMethodology, StandardizationProcess, Knowl-
edgeManagementMethodology, KnowledgeManagementProcess, Teaching-
Methodology.
1 Introduction
Electronic educational resources (EER) is object of quality management system of
the educational process with the use of ICT [1, 2]. There are two main approaches to
the concept of quality EER: compliance with standards and customer requirements.
Therefore it is necessary to take into account two aspects: compliance with educational
standards and meeting the requirements of students and teachers of the university. The
compatibility with international standards IMS, SCORM can be chosen as a criterion
for EER quality.
Improving the EER quality is the main purpose of the quality management system
(QMS) [3]. Implementation of QMS in institutions can improve processes by establish-
ing the effective and efficient management systems. Thus, EER quality management
provides tools, methods and technologies for the continuous improvement of the edu-
cational process. This improves performance, reduces the costs and ultimately increases
the competitive advantages of the institution.
Standards ISO 9000/9001 and ISO 29990 represent one of the models of manage-
ment of the institution to ensure the quality of the educational process [4]. Monitoring
is an essential tool of evaluating the quality of the educational process, in particular the
quality of the EER. The EER quality monitoring is understand as continuous process
of observation and recording EER parameters and their subsequent evaluation. Quality
monitoring provides expert advice according to the estimating procedure of the EER.
Because EER are classified as electronic educational editions and at the same time
they are software products then EER quality monitoring should be multilevel taking
into account their classifications.
The basic types of electronic educational resources for EER quality monitoring
should be assigned. For each EER type the weight factors and quality indicators should
be offered. The general criterion of quality electronic resources should be used to assess
their quality. It is average weighted characteristic of quality and takes into account the
weights of resource types and their relative quality indicators. The assessment of EER
quality monitoring is given by a corresponding university commission of experts [1].
Task of the present work is the analysis, calculation and optimization of parameters
of EER quality management system with use of methods for the analysis of complex
systems [5].
2 Model of EER Quality Management System
The EER quality management system is a structural element of architecture of edu-
cation quality management system in the higher educational institution. It plays a feed-
back role in EER quality management system of educational process.
The structure of EER quality management system is presented on figure 1 [1].
Let's list the basic elements of quality management system of electronic resources of
learning.
Assessment of EER quality underlies a quality management system of electronic
resources of learning. For an assessment of EER quality it is necessary:
to carry out monitoring for control of EER quality on a fixed basis;
to have a feedback with users of EER for the account of wishes in their improve-
ment from positions methodical and program-technology requirements.
It is necessary to develop these criteria of EER quality for carrying out of monitoring
of quality. The university council of experts confirms the criteria of EER quality devel-
oped by the methodical commissions. The university council of experts also confirms
the recommendations about improvement qualities of EER received as a result of the
analysis of users’ responses in Feedback system.
Results of an assessment of EER quality should be used on the one hand for im-
provement of their substantial part and satisfaction to technology requirements, and on
the other hand for publication of a rating of electronic learning resources that also pro-
motes the increase of their quality.
EER Quality management system
University coun- Monitoring of Standards and
cil of experts EER quality certification ISO
9000/9001
Assessment of
Feedback system Rating of EER
EER quality
Training of teach- EER Support Designing and
ers and employees and upgrade purchase of new
EER
Educational process
Fig. 1. Structure of EER quality management system.
Monitoring of EER quality has a leading role at their assessment of quality. The
analysis of electronic resources of learning shows, that they have the following classi-
fication: to a functional character they can be referred to learning editions, under the
form of representation they belong to a category of electronic editions, on the technol-
ogy of creation they represent software product [5]. Therefore the monitoring of quality
of electronic educational resources should be multi-criterion and multilevel according
to their classification. The satisfaction requirement to the universal international stand-
ards that are IMS, SCORM [6] is the uniting attribute of multilevel monitoring of EER
quality.
At monitoring of EER quality it is necessary to consider, the certain typological
model of system of educational editions for high schools which includes four groups of
the educational information resources differentiated to a functional sign, defining their
value and a place in educational process has affirmed [7]: learning-methodical, training,
auxiliary and supervising.
At monitoring of EER quality by criterion of compatibility with educational stand-
ards at definition of quality indicators it is possible to use specifications IMS which
describe information model of educational objects. These specifications define the
standardized set of information blocks which contains data about an educational re-
source. The IMS-package which contains educational object consists of two main ele-
ments [6]:
the IMS-manifesto – a special file which describes the base resources, the mainte-
nance and the organization of educational object (it is represented in language
XML);
the physical files which make educational object.
At monitoring of EER quality it is necessary to consider their typical classification:
electronic textbooks and methodical manuals, practical and virtual laboratory works,
tests and training simulators, etc.
Among all EER the special role is played by a distance learning course. It is the basic
educational object which is used in distance learning. It is compound training object
which unites various EER for the purpose of the organization of learning process with
use of special program environments – Distance Learning System (DLS). The example
of such program environment which allows to create, keep and use distance courses, is
DLS «Kherson Virtual University» [7].
The criterion of EER quality is considered as the average factor of quality K = (α1k1
+ α2k2 + … + αnkn)/n, where αi – average value of quality indicators, ki – value of weight
factor of i-type resource.
The general relative average criterion of EER quality can be calculated under the
formula [1]
K i1 aiti .
N
(1)
Here ai = ni∙γi – the quality metrics, i k /k – average factor of quality,
mi
iM
j 1 ij
ni – weight factor, mi – quantity of metric indicators of quality, kij – j-indicator of qual-
ity, kiM – the maximum value of an indicator of quality, ti – the generalized factor of
quality of i-type resource, N – quantity of EER.
The Feedback system serves as the tool for the organization of flexible and all-round
polls of opinions of students and teachers of university. Usually the system takes ques-
tioning in an automatic mode. The generalized assessment of EER quality was received
after statistical processing of results of questioning of users, it gives the opportunity to
consider the degree of their demand at quality monitoring.
Standards and certification ISO 9000/9001. Certification is a documentary
acknowledgement of conformity of production to certain requirements, concrete stand-
ards or specifications. It is necessary to notice, that conformity to standard ISO
9000/9001 does not guarantee high EER quality. However conformity to requirements
and recommendations of these standards is a necessary condition of high quality of
resources of training. The certificate of conformity ISO 9001 is acknowledgement of
satisfaction to standard requirements.
Standard ISO 9000/9001 is fundamental, the terms and definitions accepted in it are
used in all standards of a series 9000. This standard is a basis for understanding of base
elements of QMS according to ISO standards.
Requirements of standard ISO 9000/9001 can be used as criteria at the organization
and carrying out of monitoring of EER quality.
University council of experts. In the control system of EER quality the university
advisory council is the body which is responsible for adequacy assessment of EER
quality taking into account all criteria and indicators of quality. It adopts the Regulation
about ERR quality management system, defines the criteria of their quality, forms rules
of carrying out and confirms results of an assessment of quality, and also plans actions
for improvement of EER quality.
The university advisory council defines the procedure of carrying out of monitoring
of EER quality. It confirms the list of criteria of quality, their weight factors and values
of indicators of quality according to (1).
Support and upgrade of EER is the important part of work in QMS for improvement
and optimization of EER software at its use in educational process. Support EER is one
of the phases of the software lifecycle. The software logs the detection correction, and
add new functionality to increase efficiency. Support software is defined by standard
IEEE Standard for Software Maintenance (IEEE 1219), and the life cycle standard is
specified ISO 12207.
The important factor of increase of efficiency usage of EER is training of users and
maintenance them with regular support at work with the current software version.
3 Integrated and differentiated approaches in modeling and use
of EER quality management system
The control system of EER quality is a model which describes the business process
including actions and activity of services of university according to functionality of
structure described above the scheme of EER quality management (fig. 2). It is neces-
sary to notice, that some elements of this system possess the property of close interre-
lation and have various degrees of influence on it. Thus some elements of the system
(for example, «University Advisory Council» and «Standards and Certification ISO
9000/9001» at monitoring of EER quality) can be united in groups which we will name
services. Therefore for the purpose of allocation of major factors of a quality control
system, influencing quality of its work, on the basis of its structure (fig. 2) we form
three main places of maintenance of EER quality: service of quality monitoring, service
of quality assessment and EER support and upgrade service. We will define structure,
primary goals, requirements and expected results of work of these services.
The Service of quality monitoring is intended for the organization and carrying out
of EER quality monitoring which are used in educational process, by criterion of their
conformity to the international educational standards. The University advisory council
defines the order and rules of carrying out of monitoring of EER quality.
Service tasks: the coordination of parameters and development of criteria of EER
quality, taking into account the requirements of standards, carrying out of analysis of
EER by the developed and coordinated criteria.
Requirements: carrying out monitoring on fixed basis, completeness of coverage of
all kinds of EER, objectivity of application of criteria of quality.
Expected results: data of the analysis of EER characteristics for their assessment of
quality.
The Service of an assessment of quality makes EER assessment on the basis of the
confirmed criteria taking into account the opinion of users – both students, and teachers.
Feedback system can be used for automation of carrying out polls and processing of
results.
Service tasks: to assess of EER quality by the developed and coordinated criteria on
the basis of the analysis of their characteristics for maintenance of formation of rating.
Requirements: objectivity, publicity, competitive character.
Expected results: on the basis of quality assessment to generate the list of reclama-
tions to electronic resources of learning for performance of works on their elimination
and to make rating of EER for increasing of motivation of authors of resources for
improvement of their quality.
The Service of EER support and upgrade carries out the organization, planning and
performance of works on improvement of their quality by correction of the noticed
lacks, realization of new didactic properties and possibilities of electronic resources of
learning. Experts of this service give consulting services in acquiring new EER, and
also take part in training of teachers and employees to use them.
Service tasks: on a constant basis taking into account an assessment of EER quality
to perform works on their upgrade and as much as possible to satisfy inquiries of users.
Requirements: operatively, qualitatively and full performance of works.
Expected results: upgrade and introduction new and improved EER in educational
process of university.
3.1 Analysis EER QMS by criteria of its elements importance
Services of control system of EER quality provide the consecutive process of their
monitoring, assessment of quality and support. Thus Feedback system plays a feedback
role in this process. On fig. 2 the function chart of work of services of EER QMS is
presented.
According to methods of the theory of automatic control we will designate through
Wi (p) - transfer functions of EER quality of corresponding services (i = 1,2,3) and Feed-
back system (i = 4) [8]. According to rules of calculation of consecutive connection of
links of system and taking into account Feedback system transfer function of opened
system W(p) is expressed through the transfer functions of corresponding links Wi (p)
under the formula
W1 ( p) W2 ( p) W3 ( p)
W ( p) . (2)
1 W2 ( p) W3 ( p) W4 ( p)
It is necessary to notice, that the Feedback system can play a role both local negative
(–), and local positive (+) feedback. Thus the role of a negative feedback is more sig-
nificant and more often is used in work of EER QMS as the main mission of EER QMS
consists in revealing of resources of poor quality and their upgrade. At the same time
the system can be in a status of action of a local positive feedback in case of a mode of
popularization of the best practices on creation qualitative EER.
1. Quality monitoring
service
2. Service of EER qual- 4. Feedback sys-
ity assessment tem
3. Service of EER sup-
port and upgrade
Fig. 2. The scheme of service functionality in the EER quality management system.
With sufficient degree of generality it is possible to consider the model of ideal
strengthening of links of system. Then Wi(p) = ki (i = 1,2,3,4), where ki-factors of im-
provement of EER quality of corresponding i-links of system. Generally for factor k
improvement of EER quality of all the QMS from (2) we have expression
k1 k2 k3
k . (3)
1 k2 k3 k4
Considering, that the control system of EER quality is a global feedback in architec-
ture of control system of learning quality, the condition performance suffices for
maintenance of improvement of electronic resources quality k > 1 or
k1∙ k2∙ k3 > 1 ± k2∙ k3∙ k4. (4)
The correlation (3) together with a condition (4) allows to apply the differentiated
approach to the account of degree of importance of elements of EER QMS, and also to
optimize parameters of this system.
3.2 Methods of calculation and optimization of parameters of EER QMS
For the purpose of optimization of parameters of EER QMS we will apply the
method of consecutive allocation of the major elements of system by criterion of their
influence on system from the point of view of EER quality. In considered above the
model of ideal strengthening of links of system the factors of improvement of EER
quality can act as weight factors of the importance of elements of EER QMS of learn-
ing. The optimum combination of values of these factors will promote the optimization
of operating modes of all control system by quality of electronic resources. In practice
factors k1, k2, k3 and k4 are not the determined parameters, and have properties of ran-
dom variables with the known law of distribution therefore at modeling of optimum
statuses of EER QMS it is necessary to apply statistical methods of calculation and
optimization of parameters of system.
As example of use of statistical methods of calculation and optimization of parame-
ters of system the calculation of an average of distribution of factor k improvement of
EER quality depending on average of distributions of factors ki can serve. Optimization
of dispersion of values k is realized by imposing of restrictions on known values of
average of distributions and mean square deviations of factors ki.
4 Implementation and Empirical Evaluation of EER quality
management system
4.1 Method of expert evaluations of EER quality
In assessing the EER quality by the form of organization the method of collective
estimation is used with collective expert opinion. This method is used to obtain quanti-
tative estimates of the quality characteristics, parameters and properties. Analysis of
expert assessments involves filling each individual expert appropriate form, the results
of which are a comprehensive analysis of the problem situation and possible solutions.
The results of peer reviews are issued as a separate document.
The purpose of peer reviews of EER quality is an evaluation of EER quality indica-
tors with international, national and industry standards, the EER quality monitoring,
quality of the learning process through the use of qualitative EER and processing meth-
ods, criteria and forms for certification e-learning.
Objects and parameters of EER assessment:
Classification of EER types.
The weight factors of EER types (EER relative priority for their type).
Factors and criteria of EER quality for their types.
The following forms of expertise processing of EER quality are:
1. Definition of the competence of experts and the formation of the expert commit-
tee.
2. Evaluation of weight factors ranging of EER types.
3. Parameterization of EER quality indicators.
4. Expertise processing of EER quality.
5. Study the adequacy of the results of expertise.
Expert committee is created for the EER expertise with use of peer reviews method.
Delphi method is used in the formation of the expert committee and expertise pro-
cessing [9]. Top teachers, methodologists and researchers of higher education institu-
tions are involved in the commission of experts.
Since EER are classified as electronic publications for educational purposes and they
are software products, the examination of the quality of electronic educational resources
should be layered with regard to their classifications. Therefore, the EER quality should
be analyzed by the software and technological, psychological, pedagogical and ergo-
nomic features.
EER quality indicators are derivative of the requirements for them. Meeting the re-
quirements of program-technological, psychological, pedagogical and ergonomic ones
are a measure of EER quality assessment in determining their quality indicators
In this case the development of tools is based on modern fulfilled hygiene, ergo-
nomic and technical and technological standards to the use of computer technology and
is governed by existing regulations or standards. You can ask to have developed tech-
nology expertise of EER quality indicators that can be fully regulated in detail. How-
ever, there are problems of evaluating these indicators related to obsolescence of exist-
ing standards and the fact that definition of quality are not further developed.
4.2 The EER quality monitoring in educational institutions
Monitoring and evaluation (M&E) is a process that helps improving performance
and achieving results. Its goal is to improve current and future management of outputs,
outcomes and impact [10]. Consider the EER quality monitoring by the example of
DLS «Kherson Virtual University» [7].
Formation of the commission of experts. Determining the validity of each of the
three subjects of the educational process was made by expert evaluation method. 25
qualified experts (university teachers, graduate students, methodologists) was joined
the independent expert committee.
To define a point of evaluation for each subject Delphi method (for members of the
expert committee conditions for an independent individual work were created) was
used. The statistical processing of the results, which were presented to experts for final
approval, had been conducted.
Construction of weights ranging of EER types. The weight factor of EER type is
a numerical coefficient, a parameter that determines the value, the relative importance
of this EER type than other types that are classified EER on functional grounds.
Table 1 shows an example of a possible evaluation of EER weighting coefficients
values according to their types.
Table 1. The weighting factors of EER types.
# Name of EER Type Description Weighting
factor
1 Electronic textbooks and books Full course of lectures, encyclopedia 24,9
2 Lectures notes, laboratory and Lectures annotations, laboratory and 21,2
practical work notes practical work annotations
3 Lecture Presentation Author lecture in Power Point format 16,0
4 Video Lecture Author lecture in video format 19,5
5 Audio Resource Author EER in audio format 15,1
6 Learner's guide Electronic learner’s guide in discipline 26,9
7 Guidance for conducting semi- Full description of seminars, laboratory 18,8
nars and laboratory works and practical works
8 Laboratory work Virtual laboratory works in discipline 21,3
9 Test Full set of questions with indicating 17,6
correct answers
10 Library of electronic visual aids The library of visual learning objects in 26,3
a graphical format
11 Collection of tasks, exercises, Author's electronic resource 25,9
vocabulary
12 Training computer game Author's electronic resource 23,9
13 The work program of the course Approved author’s work program in 19,6
discipline
14 Questions to exam/credit, self- In accordance with the work program 17,2
control
15 Print and Internet resources Basic and advanced print and online re- 18,4
sources of discipline with active hyper-
links
16 Distance course in the discipline Correspond to international standards 98,1
Parameterization of EER quality indicators
The EER quality indicator is a numerical parameter that determines the evaluation
the EER under its qualitative characteristic (can be used a five point Likert's system).
Also the EER types are specified, which are measured by this indicator. Filling out the
list of EER quality indicators and their attachment to the EER types is held after ap-
proving the list of EER types.
Parameterization of EER quality indicators means evaluation of quality by scaling
method [11]. Table 2 shows an example of evaluation of EER quality indicators under
their quality point scale.
Table 2. The EER quality indicators.
Name of EER quality indicator. What EER types is Quality charac- Quality
Description applied to teristics indicator
Completeness of methodical support All types 1. Full 5
of discipline 2. Incomplete 4
3. Average 3
4. Below Average 2
5. Inadequate 1
Authorship of EER All types 1. Full 5
2. collaboration 3
3. Plagiarism 0
EER compliance with state educa- All types 1. Full 5
tion standards 2. Incomplete 3
3. No 1
EER compliance with international 1, 6, 9, 16 1. Full 5
standards:IMS, SCORM, ІEEE etc. 2. Incomplete 3
3. No 1
EER compliance to work program All types 1. Full 5
content 2. Incomplete 3
3. No 1
Completeness of presenting educa- 1, 2, 3, 6, 16 1. Full 5
tional material 2. Short 4
3. Note 3
4. Plan 1
The use of resources with respect to All types 1. High 5
the maximum possible 2. Mediate 3
3. Low 1
Structuring and formatting of educa- 1, 2, 3, 6, 7, 8, 11, 16 1. Yes 5
tional material 2. Partially 3
3. No 1
Text ergonomics 1, 2, 3, 4, 5, 6, 7, 8, 9, 1. Quality 5
10, 11, 12, 16 2. Mediate 3
3. Poor 0
Hypertext links use 1, 2, 3, 6, 7, 10, 15, 16 1. Yes 5
2. No 0
Use of visual methods in material 1, 2, 3, 6, 7, 8, 10, 12, 1. Quality 5
15, 16 2. Mediate 3
3. Poor 0
Using multimedia 1, 2, 3, 4, 5, 6, 7, 8, 9, 1. Quality 5
10, 12, 15, 16 2. Mediate 3
3. Poor 0
The use of interactive systems and 1, 2, 3, 6, 7, 8, 9, 10, 1. Yes 5
modules, simulation 11, 12, 15, 16 2. No 0
Using testing, the ability to control 1, 2, 3, 6, 7, 8, 9, 11, 1. Yes 5
knowledge, self-control 12, 15, 16 2. No 0
Use file formats standard 1, 2, 3, 4, 5, 6, 7, 8, 9, 1. Yes 5
10, 11, 12, 13, 14, 15, 2. Partially 3
16 3. No 0
Use tables, charts, figures 1, 2, 3, 6, 7, 8, 9, 10, 1. Yes 5
11, 15, 16 2. No 0
Compliance learning material to All types 1. Yes 5
knowledge level of students 2. No 0
Purpose of educational material to an All types 1. Yes 5
appropriate audience 2. No 0
Free access to educational material All types 1. Yes 5
2. No 0
The stylistic correctness of teaching 1, 2, 3, 6, 7, 8, 10, 11, 1. Quality 5
learning material 15, 16 2. Mediate 3
3. Poor 0
The sequence of teaching learning 1, 2, 3, 4, 5, 6, 7, 8, 1. Quality 5
material 10, 11, 13, 14, 15, 16 2. Mediate 3
3. Poor 0
Validity of test, tutorial 1, 3, 6, 7, 8, 9, 16 1. Yes 5
2. No 0
Automatic processing of test results 1, 2, 3, 6, 7, 8, 9, 11, 1. Yes 5
and knowledge control 12, 16 2. No 0
Accessibility of used informational 1, 2, 3, 6, 7, 8, 11, 15, 1. Yes 5
resources 16 2. No 0
Matching of EER components to All types 1. Quality 5
psychological requirements 2. Mediate 3
3. Poor 0
The study of the adequacy of experiment results
Expert evaluation of the EER quality can be considered sufficiently reliable only
when a good consistency of expert answers. Therefore, the statistical processing of the
results of experts evaluations should include an analysis of consensus of experts. Con-
cordance method is used to assess the degree of consensus of experts on the factors:
weights of EER types, parameterization of EER quality indicators, and average factor
of EER quality [12].
Experts were asked to complete the table 1 for peer review weighting factors of EER
types. The values of the weighting factors were selected from 100 point scale. The re-
sults of the survey of experts are presented in Table 3.
Table 3. Expert data on weights of EER types.
Ex- ERR Types
pert #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16
1 2 11 14 7 9 4 5 3 6 12 8 10 13 15 16 1
2 2 11 14 4 8 9 10 5 6 16 3 7 12 13 15 1
3 3 10 12 7 8 2 6 4 5 11 9 13 14 15 16 1
4 4 7 10 6 8 2 5 3 9 12 11 14 13 15 16 1
5 2 10 14 9 8 3 12 4 5 6 7 11 13 15 16 1
6 3 9 10 8 7 2 6 4 5 11 12 14 15 13 16 1
7 2 12 11 8 10 5 4 3 7 13 6 9 14 15 16 1
8 3 8 13 4 7 2 6 9 5 12 10 11 16 14 15 1
9 4 10 11 6 8 2 3 5 7 12 9 13 14 15 16 1
10 2 5 13 8 7 3 6 4 10 11 9 12 16 14 15 1
11 2 11 12 6 10 5 4 3 8 13 7 9 14 15 16 1
12 2 9 11 7 8 3 4 5 6 10 12 13 14 15 16 1
13 3 10 9 8 13 2 5 4 6 12 7 11 14 16 15 1
14 3 12 13 7 9 2 4 5 6 11 8 10 14 16 15 1
15 2 8 12 6 10 3 5 4 7 13 9 11 14 15 16 1
16 5 10 11 8 6 2 3 9 4 12 7 13 14 15 16 1
17 2 9 10 7 8 4 14 3 6 12 5 11 13 16 15 1
18 2 13 11 8 10 5 4 3 7 14 6 9 12 15 16 1
19 2 6 13 11 9 3 4 5 8 12 7 10 14 15 16 1
20 4 11 10 7 8 2 3 5 6 13 9 12 15 16 14 1
21 2 12 7 8 13 3 4 5 6 11 10 9 14 15 16 1
22 5 14 13 9 2 3 6 4 7 11 8 12 16 10 15 1
23 3 11 14 7 9 5 4 2 6 10 8 13 12 16 15 1
24 2 12 13 8 7 3 6 4 5 11 9 10 16 14 15 1
Δi -138 37 77 -30 -2 -125 -71 -99 -51 77 -8 63 132 149 169 -180
Concordance coefficient W is calculated according to the formula proposed by Ken-
dall [12]
12S
W . (5)
m ( n 3 n)
2
x 12 m(n 1) , m – number of experts, n –
n n m 2
Here S
2
i 1 i i 1 j 1 ij
the number of objects of examination (e.g., EER types), xij – assessment of the i-object
by j-expert. Coefficient of concordance may vary between 0 and 1. If W = 1, all experts
gave the same evaluations for all objects, if W = 0, the evaluations of experts are not
coordinated.
Using the formula (5) we calculated that coefficient W = 0,872 and it is significantly
different from zero, so we can assume that among experts there is objective concord-
ance. Given that the value of m(n – 1)W is distributed according to χ2 with (n – 1) is the
12S
degree of freedom, then W
2
= 314,1. Comparing this value with the
m n (n 1)
tabulated value T for n – 1 = 15 degree of freedom and significance level α = 0,01,
2
we find W = 314,1 > T = 30,578. Therefore, the hypothesis of consistency of expert
2 2
evaluations confirmed according to Pearson.
Thus, the results of pedagogical experiment confirmed the assumption that the
method of expert evaluations can be the basis for the EER quality monitoring.
5 Conclusions and Outlook
The system of EER quality monitoring is based on the multi-criterion analysis of
conformity of these resources to the educational standards. Criterion of EER quality
compatibility with standards IMS, SCORM can be chosen.
Criteria of EER quality are described on a basis the multi-criterion analysis taking
into account EER compatibility with the international standards.
The basic types of electronic resources of educational appointment for carrying out
of monitoring of EER quality are allocated. For each type of EER their weight factors
and quality indicators are offered. The criterion of quality of an electronic training re-
source which is the average characteristic of quality is developed.
Technologies for EER quality monitoring is based on the method of expert evalua-
tions. The criterion of EER quality is considered as the weighted average value of qual-
ity indicators. The weights of EER types and indicators of EER quality for their types
are evaluated in pedagogical experiment. Results of experiment confirmed the assump-
tion that the method of expert evaluations can be the basis for the EER quality moni-
toring. Concordance method is used to assess the degree of consensus of experts on the
factors: weights of EER types, parameterization of EER quality indicators, and
weighted average criterion of EER quality. The model of quality management system
is shown in the example of assessing the quality of the distance learning system re-
sources.
The offered system of an assessment of ERR quality is not unique and supposes
additions and updating. The assessment of monitoring of EER quality is given by a
corresponding commission of experts of university.
The method of testing is used for the experimental verification of the results of expert
evaluation of the ERR quality. Electronic educational resources are subject to testing
by means of their actual use in the educational process. As a result of comprehensive
testing, a system of adjustments is formed to improve the ERR. The process of testing
and further development of electronic educational resources is an iterative cyclical pro-
cess. It should continue until achieving compliance with the ERR quality requirements.
Therefore, the process of testing is an element of the quality management system of
electronic educational resources. That study of ERR quality management system with
their testing in educational process is the prospect of further work.
References
1. Kravtsov H.M. Design and Implementation of a Quality Management System for Electronic
Training Information Resources / In: Ermolayev, V. et al. (eds.) Proc. 7-th Int. Conf. ICTERI
2011, Kherson, Ukraine, May 4-7, 2011, CEUR-WS.org/Vol-716, ISSN 1613-0073, P.88-
98, online CEUR-WS.org/Vol-716/ICTERI-2011-CEUR-WS-paper-6-p-88-98.pdf. – P. 88
– 98
2. H. Kravtsov. Structure of the Management System of Quality of Electronic Learning Re-
sources / Information Technologies in Education. 10th Issue. – Kherson. – 2011.– P. 94-101
3. Peris-Ortiz, M., Álvarez-García, J., Rueda-Armengot, C.: Achieving Competitive Ad-
vantage through Quality Management. Springer International Publishing Switzerland
(2015). – URL: http://www.springer.com/gp/book/9783319172507
4. ISO 9000 - Quality Management. – URL: http://www.iso.org/iso/home/standards/manage-
ment-standards/iso_9000.htm, Learning services for non-formal education and training
5. Bykov V. Yu. Models of the Open Education Organizational Systems: Monograph. – Kyiv:
Atika, 2009. – 684 p.: ill.
6. H. Kravtsov, D. Kravtsov. Knowledge Control Model of Distance Learning System on IMS
Standard / Innovative Techniques in Instruction Technology, E-learning, E-assessment, and
Education. – Springer Science + Business Media V.B. – 2008. – P.195 – 198.
7. H. Kravtsov. Evaluation Metrics of Electronic Learning Resources Quality / Information
Technologies in Education. 3d Issue. – Kherson. – 2009. – P. 141 – 147.
8. Jay C. Hsu, Andrew U. Meyer. Modern Control Principles and Applications. McGraw-Hill
(1968)
9. Rowe, G. & Wright, G. Expert Opinions in Forecasting: The Role of the Delphi Technique.
In: J.S. Armstrong (Ed.), Principles of Forecasting - A Handbook for Researchers and Prac-
titioners, pp. 125-144. Boston, MA; Kluwer Academic Publishers (2001)
10. Wikipedia – The Free Encyclopedia. – URL: https://en.wikipedia.org/wiki/Monitor-
ing_and_Evaluation
11. Kolen, Michael J., Brennan, Robert L. Test Equating, Scaling, and Linking. Methods and
Practices. Springer-Verlag New York (2004)
12. Kendall M. Rank Correlation Methods, Charles Griffen & Company, London (1948)