Support of the quality of training is one of the main objectives of the university system. The results of modeling the quality management system of electronic educational resources (EER) on the basis of the analysis of its elements are presented. The subject of the study is the EER quality monitoring. Technologies for EER quality monitoring are based on the method of expert evaluations. The criterion of EER quality is considered as the weighted average value of quality indicators. The weights of EER types and indicators of EER quality for their types are evaluated in pedagogical experiment. Results of experiment confirmed the assumption that the method of expert evaluations can be the basis for the EER quality monitoring. Concordance method is used to assess the degree of consensus of experts on the factors: weights of EER types, parameterization of EER quality indicators, and weighted average criterion of EER quality. The model of quality management system is shown in the example of assessing the quality of the distance learning system resources.
Electronic educational resources (EER) is object of quality management system of
the educational process with the use of ICT [
Improving the EER quality is the main purpose of the quality management system
(QMS) [
Standards ISO 9000/9001 and ISO 29990 represent one of the models of
management of the institution to ensure the quality of the educational process [
Because EER are classified as electronic educational editions and at the same time they are software products then EER quality monitoring should be multilevel taking into account their classifications.
The basic types of electronic educational resources for EER quality monitoring
should be assigned. For each EER type the weight factors and quality indicators should
be offered. The general criterion of quality electronic resources should be used to assess
their quality. It is average weighted characteristic of quality and takes into account the
weights of resource types and their relative quality indicators. The assessment of EER
quality monitoring is given by a corresponding university commission of experts [
Task of the present work is the analysis, calculation and optimization of parameters
of EER quality management system with use of methods for the analysis of complex
systems [
Model of EER Quality Management System
The EER quality management system is a structural element of architecture of education quality management system in the higher educational institution. It plays a feedback role in EER quality management system of educational process.
The structure of EER quality management system is presented on figure 1 [
Let's list the basic elements of quality management system of electronic resources of learning.
Assessment of EER quality underlies a quality management system of electronic resources of learning. For an assessment of EER quality it is necessary: to carry out monitoring for control of EER quality on a fixed basis; to have a feedback with users of EER for the account of wishes in their improvement from positions methodical and program-technology requirements.
It is necessary to develop these criteria of EER quality for carrying out of monitoring of quality. The university council of experts confirms the criteria of EER quality developed by the methodical commissions. The university council of experts also confirms the recommendations about improvement qualities of EER received as a result of the analysis of users’ responses in Feedback system.
Results of an assessment of EER quality should be used on the one hand for improvement of their substantial part and satisfaction to technology requirements, and on the other hand for publication of a rating of electronic learning resources that also promotes the increase of their quality.
University council of experts
Monitoring of EER quality has a leading role at their assessment of quality. The
analysis of electronic resources of learning shows, that they have the following
classification: to a functional character they can be referred to learning editions, under the
form of representation they belong to a category of electronic editions, on the
technology of creation they represent software product [
At monitoring of EER quality it is necessary to consider, the certain typological
model of system of educational editions for high schools which includes four groups of
the educational information resources differentiated to a functional sign, defining their
value and a place in educational process has affirmed [
At monitoring of EER quality by criterion of compatibility with educational
standards at definition of quality indicators it is possible to use specifications IMS which
describe information model of educational objects. These specifications define the
standardized set of information blocks which contains data about an educational
resource. The IMS-package which contains educational object consists of two main
elements [
At monitoring of EER quality it is necessary to consider their typical classification: electronic textbooks and methodical manuals, practical and virtual laboratory works, tests and training simulators, etc.
Among all EER the special role is played by a distance learning course. It is the basic
educational object which is used in distance learning. It is compound training object
which unites various EER for the purpose of the organization of learning process with
use of special program environments – Distance Learning System (DLS). The example
of such program environment which allows to create, keep and use distance courses, is
DLS «Kherson Virtual University» [
The criterion of EER quality is considered as the average factor of quality K = (α1k1 + α2k2 + … + αnkn)/n, where αi – average value of quality indicators, ki – value of weight factor of i-type resource.
The general relative average criterion of EER quality can be calculated under the
formula [
N K i1 aiti .
(1) m
Here ai = ni∙γi – the quality metrics, i j i1 kij /kiM – average factor of quality, ni – weight factor, mi – quantity of metric indicators of quality, kij – j-indicator of quality, kiM – the maximum value of an indicator of quality, ti – the generalized factor of quality of i-type resource, N – quantity of EER.
The Feedback system serves as the tool for the organization of flexible and all-round polls of opinions of students and teachers of university. Usually the system takes questioning in an automatic mode. The generalized assessment of EER quality was received after statistical processing of results of questioning of users, it gives the opportunity to consider the degree of their demand at quality monitoring.
Standards and certification ISO 9000/9001. Certification is a documentary acknowledgement of conformity of production to certain requirements, concrete standards or specifications. It is necessary to notice, that conformity to standard ISO 9000/9001 does not guarantee high EER quality. However conformity to requirements and recommendations of these standards is a necessary condition of high quality of resources of training. The certificate of conformity ISO 9001 is acknowledgement of satisfaction to standard requirements.
Standard ISO 9000/9001 is fundamental, the terms and definitions accepted in it are used in all standards of a series 9000. This standard is a basis for understanding of base elements of QMS according to ISO standards.
Requirements of standard ISO 9000/9001 can be used as criteria at the organization and carrying out of monitoring of EER quality.
University council of experts. In the control system of EER quality the university advisory council is the body which is responsible for adequacy assessment of EER quality taking into account all criteria and indicators of quality. It adopts the Regulation about ERR quality management system, defines the criteria of their quality, forms rules of carrying out and confirms results of an assessment of quality, and also plans actions for improvement of EER quality.
The university advisory council defines the procedure of carrying out of monitoring of EER quality. It confirms the list of criteria of quality, their weight factors and values of indicators of quality according to (1).
Support and upgrade of EER is the important part of work in QMS for improvement and optimization of EER software at its use in educational process. Support EER is one of the phases of the software lifecycle. The software logs the detection correction, and add new functionality to increase efficiency. Support software is defined by standard IEEE Standard for Software Maintenance (IEEE 1219), and the life cycle standard is specified ISO 12207.
The important factor of increase of efficiency usage of EER is training of users and maintenance them with regular support at work with the current software version. 3
Integrated and differentiated approaches in modeling and use of EER quality management system
The control system of EER quality is a model which describes the business process including actions and activity of services of university according to functionality of structure described above the scheme of EER quality management (fig. 2). It is necessary to notice, that some elements of this system possess the property of close interrelation and have various degrees of influence on it. Thus some elements of the system (for example, «University Advisory Council» and «Standards and Certification ISO 9000/9001» at monitoring of EER quality) can be united in groups which we will name services. Therefore for the purpose of allocation of major factors of a quality control system, influencing quality of its work, on the basis of its structure (fig. 2) we form three main places of maintenance of EER quality: service of quality monitoring, service of quality assessment and EER support and upgrade service. We will define structure, primary goals, requirements and expected results of work of these services.
The Service of quality monitoring is intended for the organization and carrying out of EER quality monitoring which are used in educational process, by criterion of their conformity to the international educational standards. The University advisory council defines the order and rules of carrying out of monitoring of EER quality.
Service tasks: the coordination of parameters and development of criteria of EER quality, taking into account the requirements of standards, carrying out of analysis of EER by the developed and coordinated criteria.
Requirements: carrying out monitoring on fixed basis, completeness of coverage of all kinds of EER, objectivity of application of criteria of quality.
Expected results: data of the analysis of EER characteristics for their assessment of quality.
The Service of an assessment of quality makes EER assessment on the basis of the confirmed criteria taking into account the opinion of users – both students, and teachers. Feedback system can be used for automation of carrying out polls and processing of results.
Service tasks: to assess of EER quality by the developed and coordinated criteria on the basis of the analysis of their characteristics for maintenance of formation of rating.
Requirements: objectivity, publicity, competitive character.
Expected results: on the basis of quality assessment to generate the list of reclamations to electronic resources of learning for performance of works on their elimination and to make rating of EER for increasing of motivation of authors of resources for improvement of their quality.
The Service of EER support and upgrade carries out the organization, planning and performance of works on improvement of their quality by correction of the noticed lacks, realization of new didactic properties and possibilities of electronic resources of learning. Experts of this service give consulting services in acquiring new EER, and also take part in training of teachers and employees to use them.
Service tasks: on a constant basis taking into account an assessment of EER quality to perform works on their upgrade and as much as possible to satisfy inquiries of users.
Requirements: operatively, qualitatively and full performance of works.
Expected results: upgrade and introduction new and improved EER in educational process of university. 3.1
Analysis EER QMS by criteria of its elements importance
Services of control system of EER quality provide the consecutive process of their monitoring, assessment of quality and support. Thus Feedback system plays a feedback role in this process. On fig. 2 the function chart of work of services of EER QMS is presented.
According to methods of the theory of automatic control we will designate through
Wi(p) - transfer functions of EER quality of corresponding services (i = 1,2,3) and
Feedback system (i = 4) [
W ( p)
W1( p) W2 ( p) W3 ( p) 1 W2 ( p) W3 ( p) W4 ( p) .
(2)
It is necessary to notice, that the Feedback system can play a role both local negative (–), and local positive (+) feedback. Thus the role of a negative feedback is more significant and more often is used in work of EER QMS as the main mission of EER QMS consists in revealing of resources of poor quality and their upgrade. At the same time the system can be in a status of action of a local positive feedback in case of a mode of popularization of the best practices on creation qualitative EER.
1. Quality monitoring
service
port and upgrade
With sufficient degree of generality it is possible to consider the model of ideal strengthening of links of system. Then Wi(p) = ki (i = 1,2,3,4), where ki-factors of improvement of EER quality of corresponding i-links of system. Generally for factor k improvement of EER quality of all the QMS from (2) we have expression k
k1 k2 k3 1 k2 k3 k4
Considering, that the control system of EER quality is a global feedback in architecture of control system of learning quality, the condition performance suffices for maintenance of improvement of electronic resources quality k > 1 or
The correlation (3) together with a condition (4) allows to apply the differentiated approach to the account of degree of importance of elements of EER QMS, and also to optimize parameters of this system. 3.2
Methods of calculation and optimization of parameters of EER QMS For the purpose of optimization of parameters of EER QMS we will apply the method of consecutive allocation of the major elements of system by criterion of their influence on system from the point of view of EER quality. In considered above the model of ideal strengthening of links of system the factors of improvement of EER (3) (4) quality can act as weight factors of the importance of elements of EER QMS of learning. The optimum combination of values of these factors will promote the optimization of operating modes of all control system by quality of electronic resources. In practice factors k1, k2, k3 and k4 are not the determined parameters, and have properties of random variables with the known law of distribution therefore at modeling of optimum statuses of EER QMS it is necessary to apply statistical methods of calculation and optimization of parameters of system.
As example of use of statistical methods of calculation and optimization of parameters of system the calculation of an average of distribution of factor k improvement of EER quality depending on average of distributions of factors ki can serve. Optimization of dispersion of values k is realized by imposing of restrictions on known values of average of distributions and mean square deviations of factors ki. 4
Implementation and Empirical Evaluation of EER quality management system 4.1
Method of expert evaluations of EER quality
In assessing the EER quality by the form of organization the method of collective estimation is used with collective expert opinion. This method is used to obtain quantitative estimates of the quality characteristics, parameters and properties. Analysis of expert assessments involves filling each individual expert appropriate form, the results of which are a comprehensive analysis of the problem situation and possible solutions. The results of peer reviews are issued as a separate document.
The purpose of peer reviews of EER quality is an evaluation of EER quality indicators with international, national and industry standards, the EER quality monitoring, quality of the learning process through the use of qualitative EER and processing methods, criteria and forms for certification e-learning.
Objects and parameters of EER assessment: Classification of EER types. The weight factors of EER types (EER relative priority for their type). Factors and criteria of EER quality for their types.
The following forms of expertise processing of EER quality are: 1. Definition of the competence of experts and the formation of the expert committee.
2. Evaluation of weight factors ranging of EER types. 3. Parameterization of EER quality indicators. 4. Expertise processing of EER quality. 5. Study the adequacy of the results of expertise.
Expert committee is created for the EER expertise with use of peer reviews method.
Delphi method is used in the formation of the expert committee and expertise
processing [
Since EER are classified as electronic publications for educational purposes and they are software products, the examination of the quality of electronic educational resources should be layered with regard to their classifications. Therefore, the EER quality should be analyzed by the software and technological, psychological, pedagogical and ergonomic features.
EER quality indicators are derivative of the requirements for them. Meeting the requirements of program-technological, psychological, pedagogical and ergonomic ones are a measure of EER quality assessment in determining their quality indicators
In this case the development of tools is based on modern fulfilled hygiene, ergonomic and technical and technological standards to the use of computer technology and is governed by existing regulations or standards. You can ask to have developed technology expertise of EER quality indicators that can be fully regulated in detail. However, there are problems of evaluating these indicators related to obsolescence of existing standards and the fact that definition of quality are not further developed. 4.2
The EER quality monitoring in educational institutions
Monitoring and evaluation (M&E) is a process that helps improving performance
and achieving results. Its goal is to improve current and future management of outputs,
outcomes and impact [
Formation of the commission of experts. Determining the validity of each of the three subjects of the educational process was made by expert evaluation method. 25 qualified experts (university teachers, graduate students, methodologists) was joined the independent expert committee.
To define a point of evaluation for each subject Delphi method (for members of the expert committee conditions for an independent individual work were created) was used. The statistical processing of the results, which were presented to experts for final approval, had been conducted.
Construction of weights ranging of EER types. The weight factor of EER type is a numerical coefficient, a parameter that determines the value, the relative importance of this EER type than other types that are classified EER on functional grounds.
Table 1 shows an example of a possible evaluation of EER weighting coefficients values according to their types. Learner's guide
control
Print and Internet resources Guidance for conducting seminars and laboratory works
sources of discipline with active hyperlinks Distance course in the discipline
The EER quality indicator is a numerical parameter that determines the evaluation the EER under its qualitative characteristic (can be used a five point Likert's system). Also the EER types are specified, which are measured by this indicator. Filling out the list of EER quality indicators and their attachment to the EER types is held after approving the list of EER types.
Parameterization of EER quality indicators means evaluation of quality by scaling
method [
Name of EER quality indicator. What EER types is Quality charac- Quality Description applied to teristics indicator Completeness of methodical support All types 1. Full 5 of discipline 2. Incomplete 4 3. Average 3 Authorship of EER
4. Below Average 5. Inadequate 1. Full 2. collaboration 3. Plagiarism 1. Full 2. Incomplete 3. No 1. Full 2. Incomplete 3. No 1. Full 2. Incomplete 3. No 1. Full 2. Short 3. Note 4. Plan
2. Mediate 3. Low The use of interactive systems and 1, 2, 3, 6, 7, 8, 9, 10, 1. Yes modules, simulation 11, 12, 15, 16 2. No 2 1
Expert evaluation of the EER quality can be considered sufficiently reliable only
when a good consistency of expert answers. Therefore, the statistical processing of the
results of experts evaluations should include an analysis of consensus of experts.
Concordance method is used to assess the degree of consensus of experts on the factors:
weights of EER types, parameterization of EER quality indicators, and average factor
of EER quality [
Experts were asked to complete the table 1 for peer review weighting factors of EER types. The values of the weighting factors were selected from 100 point scale. The results of the survey of experts are presented in Table 3.
Concordance coefficient W is calculated according to the formula proposed by
Kendall [
W
12S m2 (n3 n) .
(5)
Here S in1 2i in1 mj1 xij 12 m(n 1)2 , m – number of experts, n – the number of objects of examination (e.g., EER types), xij – assessment of the i-object by j-expert. Coefficient of concordance may vary between 0 and 1. If W = 1, all experts gave the same evaluations for all objects, if W = 0, the evaluations of experts are not coordinated.
Using the formula (5) we calculated that coefficient W = 0,872 and it is significantly different from zero, so we can assume that among experts there is objective concordance. Given that the value of m(n – 1)W is distributed according to χ2 with (n – 1) is the 12S degree of freedom, then W2 m n (n 1) = 314,1. Comparing this value with the tabulated value 2
T for n – 1 = 15 degree of freedom and significance level α = 0,01, we find W2 = 314,1 > 2
T = 30,578. Therefore, the hypothesis of consistency of expert evaluations confirmed according to Pearson.
Thus, the results of pedagogical experiment confirmed the assumption that the method of expert evaluations can be the basis for the EER quality monitoring. 5
Conclusions and Outlook
The system of EER quality monitoring is based on the multi-criterion analysis of conformity of these resources to the educational standards. Criterion of EER quality compatibility with standards IMS, SCORM can be chosen.
Criteria of EER quality are described on a basis the multi-criterion analysis taking into account EER compatibility with the international standards.
The basic types of electronic resources of educational appointment for carrying out of monitoring of EER quality are allocated. For each type of EER their weight factors and quality indicators are offered. The criterion of quality of an electronic training resource which is the average characteristic of quality is developed.
Technologies for EER quality monitoring is based on the method of expert evaluations. The criterion of EER quality is considered as the weighted average value of quality indicators. The weights of EER types and indicators of EER quality for their types are evaluated in pedagogical experiment. Results of experiment confirmed the assumption that the method of expert evaluations can be the basis for the EER quality monitoring. Concordance method is used to assess the degree of consensus of experts on the factors: weights of EER types, parameterization of EER quality indicators, and weighted average criterion of EER quality. The model of quality management system is shown in the example of assessing the quality of the distance learning system resources.
The offered system of an assessment of ERR quality is not unique and supposes additions and updating. The assessment of monitoring of EER quality is given by a corresponding commission of experts of university.
The method of testing is used for the experimental verification of the results of expert evaluation of the ERR quality. Electronic educational resources are subject to testing by means of their actual use in the educational process. As a result of comprehensive testing, a system of adjustments is formed to improve the ERR. The process of testing and further development of electronic educational resources is an iterative cyclical process. It should continue until achieving compliance with the ERR quality requirements. Therefore, the process of testing is an element of the quality management system of electronic educational resources. That study of ERR quality management system with their testing in educational process is the prospect of further work.