=Paper=
{{Paper
|id=Vol-1844/10000440
|storemode=property
|title=Development of E-Learning Quality Assessment Model in Pedagogical
University
|pdfUrl=https://ceur-ws.org/Vol-1844/10000440.pdf
|volume=Vol-1844
|authors=Nadiia Balyk,Vasyl Oleksiuk,Galina Shmyger
|dblpUrl=https://dblp.org/rec/conf/icteri/BalykOS17
}}
==Development of E-Learning Quality Assessment Model in Pedagogical
University==
https://ceur-ws.org/Vol-1844/10000440.pdf
Development of e-Learning Quality Assessment Model in
Pedagogical University
Nadiia Balyk1, Vasyl Oleksiuk2, Galina Shmyger3
1
Volodymyr Hnatiuk Ternopil National Pedagogical University, Ternopil, Ukraine
nadbal@fizmat.tnpu.edu.ua
2
Volodymyr Hnatiuk Ternopil National Pedagogical University, Ternopil, Ukraine
oleksyuk@fizmat.tnpu.edu.ua
3
Volodymyr Hnatiuk Ternopil National Pedagogical University, Ternopil, Ukraine
shmyger@fizmat.tnpu.edu.ua
Abstract. The paper is concerned with the practice of implementation of life-
long learning in Volodymyr Hnatiuk Ternopil National Pedagogical University
(Ukraine), analyzed various models of e-learning quality assessment are ana-
lyzed. Based on many years of experience, a model of e-learning assessment in
Volodymyr Hnatiuk Ternopil National Pedagogical University (Ukraine) was
developed. It includes criteria and indicators for assessment of the e-learning
system quality. In order to implement and adapt the model into practical learn-
ing process an experimental research was conducted. The article presents the
contents of the study, the range of respondents and analyzed results.
Keywords: Lifelong learning, e-learning, pedagogical university, e-learning
quality, model.
Key Terms. Methodology, TeachingProcess, InformationCommunicationTech-
nology.
1 Introduction
1.1 The Problem Statement
Lifelong learning is becoming very important field of education in the world and es-
pecially in the developed countries. Today there are three main forms of lifelong
learning [9]:
─ formal education – primary, general secondary education, secondary vocational
education, higher education, education after graduation (postgraduate and doctoral
studies), training and retraining of specialists and managers with higher and sec-
ondary vocational education in institutes, on faculties and training courses and pro-
fessional retraining;
─ non-formal education – professionally aimed and general cultural training courses
in adult education centers, on TV, on various courses of intensive training;
─ informal learning – is a general term for education outside a standard educational
environment – individual cognitive activity that accompanies everyday life, im-
plemented by individuals’ own activity in their cultural and educational surround-
ings; communication, reading, visiting cultural institutions, travel, media and more.
Herewith, man turns the educational potential of society into an effective factor for
his own development.
The European Commission brought together various educational and training initia-
tives into a unified Lifelong Learning Programme. One of the priorities of lifelong
learning is the development and implementation of e-learning into both formal higher
education and non-formal education, and informal learning. With increasing growth
in popularity of electronic learning (e-learning) with the aim to obtain education and
training, the assessment of e-learning quality is a relevant issue [5].
1.2 The State of the Art
Various aspects of the problem highlighted in many scientific works of domestic (V.
Yu. Bykov, Yu. M. Bohachkov, R. S. Hurevych, M. I. Zhaldak, Yu. O. Zhuk, T. I.
Koval, A. Yu. Kravtsova, V. M. Kukharenko, N. V. Morze, Yu. S. Ramskyi, S. A.
Rakov, O.V. Spivakovskyi, O.M. Spirin, Yu. V. Tryus) and foreign scientists (M.
Allen, C. McLoughlin, D. Morrison, E. Allen, D. Lederman, AL Fritschler, K. Khan,
K. Thompson, E. Smyrnova-Trybulska). Many publications are devoted to a detailed
examination of the problem [11; 13; 14; 15]. Thus, M. Allen [1] singled specific tech-
nology and educational techniques that affect the improvement of the quality of e-
learning; B. Khan [12] offered hexagonal model of e-learning in higher education to
improve its quality; N. Morze [16] created a model for assessing the quality of e-
learning courses in the practice of Ukrainian universities and identified relevant crite-
ria.
Ukrainian researchers M. Shyshkina, O. Spirin, Yu. Nosenko indicate that the qual-
ity of educational resources needs to take into account the requirements of mainte-
nance management, interface design, ergonomics, hygiene and other [21]. They offer
to assess the quality of e-learning according to the following criteria: adaptability,
interactivity, integration and security. E-learning quality assessment requires appro-
priate training of qualified scientists on information and communication technologies
in education [22]. A distinctive feature of professional competences of these profes-
sionals is their commitment to lifelong learning and self-development [4].
Group of Iranian scientists conducted a study [10] during which the criteria for a
successful e-learning were analyzed. As a result, a model of assessment with five
groups of criteria was created: 1) infrastructure and technology; 2) human resources
3) plans, policies, strategies adopted by institutions in order to develop distance learn-
ing 4) development; 5) cooperation with other organizations and interested individu-
als.
American scientist B. Chaney [8] proposed a different model for assessing the
quality of e-learning in higher education, which includes the following criteria:
─ student-teacher interaction;
─ active learning techniques;
─ prompt feedback;
─ respect diverse ways of learning;
─ student support services;
─ faculty support services;
─ program evaluation and assessment;
─ strong rationale for distance that correlates to the mission of the institution;
─ clear analysis of audience;
─ appropriate tools and media;
─ documented technology plan to ensure quality;
─ reliability of technology;
─ institutional support and institutional resources;
─ implementation of guidelines for course development and review of instructional
materials;
─ course structure guidelines.
The following models of quality assessment and certification of e-learning are also
used in international educational practice:
─ ISO, IMS standards;
─ UNIQUE, DETC institutional systems;
─ software systems (ASIIN, CEL, eXcellence);
─ technological standards (IMS, ADL) [25].
Models and tools to ensure the quality of e-learning are currently defined [19; 20; 26],
so there is only a need to decide on their set and the use. Given the above, we note
that every international model for e-learning quality assessment requires adaptation to
the realities of Ukrainian pedagogical universities. The problem of e-learning quality
assessment in higher education needs to be solved, including the definition of criteria
and indicators of assessment related to the educational process in pedagogical univer-
sity.
1.3 The Purpose of the Article
Object of the article is to develop model of e-learning quality assessment and its ex-
perimental testing in Volodymyr Hnatiuk TNPU.
2 The Presentation of Main Material
Department of Computer science and Teaching Techniques of Volodymyr Hnatiuk
Ternopil National Pedagogical University has a considerable experience in the field of
lifelong learning, both formal and informal. It prepares students, undergraduates,
graduate students (education after graduation), holds retraining of specialists on train-
ing courses and professional training in the center of postgraduate education. An in-
ternational training center "Educational Innovation" and STEM-center "Digital eru-
dites" operate on the Department and they can be attributed to the formal institutions
of further education. As part of these centers staff conducts numerous workshops,
master classes, seminars, tours for students, teachers, school leaders, education coor-
dinators, auditors of the employment center.
In the context of lifelong learning, one of the urgent problems of the university is a
system of quality assurance. It is primarily about internal quality assurance. Now the
trend is to coordinate and unify standards of educational materials developed by dif-
ferent standard organizations such as IEEE, IMS, ISO / IEC JTC1 SC36 and others, as
well as to tie together the national standards with international ones [21].
In its day-to-day activities, to ensure quality of education in general, and quality of
e-learning, in particular, our university adheres to the following basic principles: mo-
bility (rapid response to customer requirements of educational services and the labor
market, strategic and tactical changes in the system of training and innovations in
education); complexity (optimal implementation of all activities (educational, organi-
zational, technical, scientific, educational, etc.), publicity and openness (discussing
the achievements and results of the university and its departments on different direc-
tions on university councils and other meetings, in cyberspace of the institution and
on media, collective and personal responsibility of teaching staff, support staff and
students for the organization, progress and results of the educational process).
E-learning system has been implemented in Volodymyr Hnatiuk TNPU since
2007. The implementation of e-learning in the university is guided by the following
documents: Law of Ukraine "On Education", Resolution "On Higher Education" by
the Cabinet of Ministers of Ukraine as of 23 September 2003 Number 1494 "On ap-
proval of the development of distance learning for 2004-2006"; Provision on distance
education, approved by the MES of Ukraine as of January 21, 2004; Resolution by the
Cabinet of Ministers of Ukraine as of December 7, 2005 № 1153 about State Program
"Information and communication technologies in education and science for 2006-
2010"; "Provision of electronic educational methodological complex discipline in
Volodymyr Hnatiuk TNPU" as of 26.06.2007, the "Regulations on distance education
in Volodymyr Hnatiuk Ternopil National Pedagogical University" as of 28.10.2014.
E-learning system is based on the following principles: adaptability, flexibility,
modularity, portability, accessibility, universality and individuality. The main soft-
ware components of the system are: the domain system preserving user accounts,
learning management system, service Fizmat-Wikipedia, institutional repository,
virtual private network, private and public (G Suite, Microsoft Office 365) university
clouds. Technically, the system provides eLearning servers, storage, LAN, Internet,
computer labs, Wi-Fi and more. All these services are integrated both with the use of
a single account of a student or a teacher, and at the level of content [17; 18].
In the process of creating a model of e-learning, cloud technologies have been
widely used. E-learning quality assessment system model was introduced in Vo-
lodymyr Hnatiuk TNPU taking into account the components of the overall model of
cloud based university environment formation, including its objectives and functions,
methods, approaches and principles, service models and eligibility criteria [6]. We
were guided by the following principles of cloud based university environment for-
mation: the principles of open education (mobility of students and teachers, equal
access to educational systems, structure formation and implementation of educational
services) and specific principles (adaptability, personalization, unification of man-
agement, full-scale interactive ICT tools) [7].
2.1 E-learning Quality Assessment Model
The model of quality education in Volodymyr Hnatiuk TNPU was formed in 2008
and is based on the standard ISO 9001: 2000 (ISO 9001: 2008). A task was set to
formulate a list of criteria to assess the quality of e-learning, adapted to the peculiari-
ties of Ukrainian education and e-learning in pedagogical university. Research on the
development of e-learning quality assessment model and its experimental testing is
the continuation of research on the topic "Implementation of technologies of e-
learning in higher and secondary educational institutions" (state registration number
0111U004875, 2011-2016) [2; 3].
Among the variety of options when creating and selecting the set of criteria for ed-
ucation quality assessment indicators and their structural features a working party that
works on the elaboration of criteria selected an approach, that is traditional for most
European systems of e-learning quality assessment. The choice of areas of assessment
and criteria was based on guarantees of a sufficient quality of education.
Analysis of different e-learning quality assessment models enabled to identify
those components that are the core of e-learning system for any educational institu-
tion. These include:
─ strategic management;
─ technical support;
─ development of curricula and courses;
─ work with teachers and students.
The model of e-learning quality assessment in TNPU is dynamic one. The proposed
model is a reference and can vary considerably depending on the specific institution
and how the process of e-learning will be developed. Based on the components, which
is the core of e-learning, we offer to highlight such groups of e-learning quality as-
sessment criteria: technological, educational, organizational and communication crite-
ria.
Technological criteria imply assessment of such indicators as:
─ compliance with generally accepted standards and technologies;
─ services internal integration;
─ safety;
─ reliability and system integrity.
Educational criteria imply assessment of such indicators as:
─ compliance of e-learning educational resources with training content;
─ pedagogical design;
─ possibility of individual learning course implementation;
─ variety of evaluation system.
Organizational criteria imply assessment of such indicators as:
─ practicability of using e-learning during different modes of study;
─ opportunity to monitor the actions of teachers and students;
─ availability of technical support.
Communication criteria imply assessment of such indicators as:
─ existence of systematic feedback teachers and students;
─ providing with synchronous and asynchronous communication using modern ser-
vices;
─ providing with collaborative learning.
2.2 Organization, Conduct and Results of Experimental Work
The survey was conducted among teachers, postgraduates and students of Vo-
lodymyr Hnatiuk Ternopil National Pedagogical University and among auditors of
postgraduate center who use e-courses available in the portal elr.tnpu.edu.ua during
the learning process. The survey asked 859 respondents participated, of which 515 –
students, 266 – teachers and postgraduate students, 78 – auditors of postgraduate cen-
ter (see Table 1).
Table 1. Distribution of respondents by age
Age category Amount
18-22 515
23-40 218
41-60 126
Among the 23-40 age category 158 teachers and postgraduate students, 60 auditors
of postgraduate center were surveyed; among the 41-60 age category 108 teachers and
18 auditors of postgraduate center were surveyed.
Expert teachers carried out the external assessment, based on groups of criteria that
determine the quality of the whole system, namely technological, pedagogical, organ-
izational, and communication criteria. Experts estimated indicators for the following
parameters: 0 points – the indicator is not observed, 1 point – the indicator is more not
observed than observed, 2 points – the indicator is more observed than not observed,
3 points – the indicator is fully observed. The indicator was considered positive if the
arithmetic mean value of its parameters was at least 1,5 [24; 23].
Results of the survey of experts represented in Table 2:
Table 2. Results of expert assessment of e-learning
Criteria Total points
Experts Technological Educational Organizational Communication
1 3 2 1 2
2 3 1 2 2
3 3 2 2 2
4 3 2 1 2
5 3 2 1 2
6 3 2 2 3
7 2 2 2 2
8 3 1 2 2
9 3 2 2 3
10 2 2 1 2
11 3 2 2 3
12 2 2 2 3
Arithmetic 2,75 1,83 1,67 2,33
mean
value
We used Kendall's coefficient of concordance to assess the consistency of ex-pert
opinion. Taking into account that the experts assessed different criteria with the same
points, and the ratio
m R ij
12 Rij ji
j 1 n
W z
m 2 ( n 3 n ) m ( Bk3 Bk )
k 1 (1)
where Rij – point of i assessment criterion, by j expert, B – number of related (simi-
lar) points of k expert, n – number of groups of criteria, m – number of experts, we
got a Kendall's coefficient of concordance W = 0,589.
It indicates the existence of average degree of consistency of expert opinion. How-
ever, this coefficient W is not objective, as it could be obtained due to random as-
sessing of particular criteria groups. To determine the degree of expert assessment
coordination, we calculated Pearson correlation coefficient χ2=21,138. Comparing it
with tabulated for n-1=3 degrees of freedom and α=0,05 for level of significance, we
get χ2=21,138> 7,814. So we can conclude that the value of W=0,589 is not acci-
dental and there is a consistency between the experts’ conclusions.
The above given criteria and quality indicators are more concerned with the pro-
cess and the result of training activities. Since foreign educational systems engage
students as experts, we also carried out e-learning quality assessment based on a sur-
vey of students of Volodymyr Hnatiuk Ternopil National Pedagogical University.
Students of 1–4 courses of the university with mixed form of education using e-
learning courses were selected as the group of respondents.
The proposed questionnaire included questions aimed at studying criteria such as
ease of use of electronic course; utility; interaction; individualization and contentment
(Table 3). For each criterion of the questionnaire, students answered on a 4-point
scale: "Disagree", "Almost agree", "Agree", "Absolutely agree". Table 3 shows the
basic criteria groups.
Table 3. Summary of criteria groups of students questioning
Criterion Indicator
Learning with the system was easy for me
Ease of use I find the system easy to use
The system quickly gives me what I need
Using the system I improved productivity of studying
learning material
Utility Taking online courses I improved studying of learn-
ing material
I spend less time learning with the help of the system
I can share information effectively
I can get support from cooperative learning and
Interaction
group work with other students
I can easily get advice and support from the teacher
I can choose time and pace for learning on my own
Existence of personalized learning support
Individualization
There are multilevel theoretical and practical tasks
within the system
Overall, I feel content with this learning model
I am pleased that e-learning meets my demands
Contentment
I would constantly use these tools during my learning
process
These criteria correlate with the e-learning assessment criteria assessed by expert
teachers (Table 4).
Table 4. Correspondence of e-learning assessment criteria (teacher / student)
Criteria for e-learning quality Criteria for e-learning quality as-
assessment (teachers) sessment (students)
Technological Ease of use
Educational Utility
Organizational Individualization
Communication Interaction, contentment
The survey results are presented in Table 5.
Table 5. Results of a students’ survey
Ease of Individualiza- Content-
Utility Interaction
use tion ment
Disagree 39 41 57 42 24
Almost
104 78 146 96 98
agree
Agree 234 269 222 267 288
Absolute-
138 127 90 110 105
ly agree
Analyzing the results of a students’ survey, we have to note that a large percentage
of objections falls on such system assessment criteria as interaction and individualiza-
tion (Fig. 1). For example, there are only 17.5% of fully pleased with system interac-
tion opportunities students and there are 11% of fully displeased students. Along with
this, we can say that a significant percentage of students acquire competency without
difficulty while working with e-learning system services of the university.
1400
1200
288
1000
Contentment
267
800 Individualization
Interaction
600 222
105 Utility
98
400 96 110 Ease of use
269
90
146
200 24 127
42 78
57 234
41 104 138
0 39
Disagree Almost agree Agree Absolutely agree
Fig. 1. Students’ assessment of e-learning system
E-learning quality in this case is guaranteed by the availability of effective education-
al process and outcomes that please the student, resulting in a positive assessment of
their knowledge.
3 Conclusions
As a part of the study we have created a model of e-learning quality assessment
adapted to teaching not only students, but also people of older age categories. There
are good reasons to include criteria related to the processes and results of learning
activities, including: organizational, technological, educational, and communication
criteria in order to assess e-learning.
Statistical processing of study data allows making scientifically substantiated con-
clusions on the correct choice of criteria and indicators of e-learning quality in peda-
gogical university, sufficient educational quality from the use of electronic courses, as
well as the likelihood of the results.
E-learning quality assessment model of Volodymyr Hnatiuk Ternopil National
Pedagogical University enables to: develop e-learning program for students, who
study lifelong; improve the quality of e-learning, identifying weaknesses and elements
for their improvement; use best existing e-learning practices.
Further research will determine the additional criteria and indicators of e-learning
quality assessment. A prospect for further research is to develop methods of specialist
training in the field of e-learning.
References
1. Allen, M.W.: Creating Successful E-Learning: A Rapid System For Getting It Right First
Time, Every Time. Alpina Publisher, Moscow, (2016) (In Russian).
2. Balyk, N., Gabrusev V.: Creating preparedness of teachers and students to the implementa-
tion e-learning or distance learning. E-learning and Lifelong Learning, Monograph, Sc.
Editor Eugenia Smyrnova-Trybulska University of Silesia, Studio-Noa Publishing, Kato-
wice-Cieszyn, 221 – 233, (2013)
3. Balyk, N., Shmyger, G.: Professional Training Teachers University in E-Learning. IT tools
– Good Practice of Effective Use in Education. Monograph Sc. Editor Eugenia Smyrnova-
Trybulska, University of Silesia, Studio-Noa, Katowice-Cieszyn, 323 – 335, (2015)
4. Burov, O.: Life-Long Learning: Individual Abilities versus Environment and Means. Proc.
12-th Int. Conf. ICTERI 2016, http://ceur-ws.org/Vol-1614/paper_86.pdf (2016).
5. Bykov, V.: Models of Organizational Systems of Open Education. Atika, Kyiv (2009) (in
Ukrainian)
6. Bykov, V., Shyshkina, M.: The cloud computing as imperative of the university education
and research environment modernization. Theory and practice of social systems manage-
ment: philosophy, psychology, pedagogy, sociology. 6, 55-70,
http://tipus.khpi.edu.ua/article/view/90005 (in Ukrainian)
7. Bykov, V., Shyshkina, M.: Theoretical and methodological principles of the cloud based
university environment formation. Theory and practice of social systems management:
philosophy, psychology, pedagogy, sociology. 2, 30-52, online:
http://tipus.khpi.edu.ua/article/download/73497/68881 (in Ukrainian)
8. Chaney, E.H., Eddy, J. M.: A primer on quality indicators of distance education. Health
Promotion Practice, 10(2) (2007).
9. Education for life: international experience and Ukrainian practice. (Analytical note of Na-
tional Institute for Strategic Studies), http://www.niss.gov.ua/articles/252/ (in Ukrainian)
10. Hanfizadeha, P., Khodabakshib, M. Нanafizadehc, R.: Recommendations for promoting
elearning in higher educational institutions: a case study of Iran, Higher education Policy,
24, 103-126 (2011)
11. Kear, K., Williams, K., Rosewell, J.: Excellence in e-learning: a quality enhancement ap-
proach. Changing the Trajectory. Quality for Opening up Education, 25-32 (2014)
12. Khan K., Badii A.: Impact of E-Learning on Higher Education: Development of an E-
Learning Framework. Life Science Journal, 9(4),
http://www.lifesciencesite.com/lsj/life0904/606_13425life0904_4073_4082.pdf (2012)
13. Misut, M., Pribilova, K.: Measuring quality in context of e-learning. Procedia – Social and
Behavioral Sciences, 177, 312-319,
http://www.sciencedirect.com/science/article/pii/S1877042815017012 (2015).
14. Morrison, D.: How Good is Your Online Course? Five Steps to Assess Course Quali-
ty. Online Learning Insights,
https://onlinelearninginsights.wordpress.com/2015/05/26/how-good-is-your-online-course-
five-steps-to-assess-course-quality/ (2015)
15. McLoughlin, C., Luca, J.: Quality in online delivery: what does it mean for assessment in
e-learning environments. Meeting at the Crossroads,
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.528.1492&rep=rep1&type=pdf
(2001)
16. Morze, N., Buinytska, O.: Open E-Environment – the key instrument of the education
quality. International Journal of Research in E-learning, 1, 27-47 (2015)
17. Nosenko, Yu., Shyshkina, M., Oleksiuk V.: Collaboration between Research Institutions
and University Sector Using Cloud-based Environment. Proc. 12-th Int. Conf. ICTERI
2016, http://ceur-ws.org/Vol-1614/paper_84.pdf (2016)
18. Oleksyuk, V.: Experience of the integration cloud services Google Apps into information
and educational space of higher educational institution. Information technologies and
learning tools, 35(3), http://journal.iitta.gov.ua/index.php/itlt/article/view/824/631 (2013)
(in Ukrainian)
19. Ossiannilsson, E. Landgren, L.: Quality in e-learning – a conceptual framework based on
experiences from three international benchmarking projects. Journal of Computer Assisted
Learning, 28, 42–51 (2012)
20. Plank, T., Villems, A., Pilt, L., Dremljuga-Telk, M., Varendi, M. & Sutt, E.: Quality assur-
ance processes in e-learning – an Estonian case. International Journal for Innovation and
quality in learning, 1(1), 20-28 (2013)
21. Shyshkina, M., Spirin, O., Zaporozhchenko, Yu.: Problems of Informatization of Educa-
tion in Ukraine in the Context of Development of Research of ICT-Based Tools Quality
Estimation. In: J. Information Technologies and Learning Tools, 27(1),
http://journal.iitta.gov.ua/index.php/itlt/article/view/ 632/483 (2012) (in Ukrainian)
22. Spirin, O., Nosenko, Yu., Іatsyshyn, A.: Current requirements and contents of training of
qualified scientists on information and communication technologies in education. In: J. In-
formation Technologies and Learning Tools, 56(6),
http://journal.iitta.gov.ua/index.php/itlt/article/view/632/483 (2016) (in Ukrainian)
23. Spirin, O.: Criteria and quality indicators of information and communication technologies
of learning. . Information technologies and learning tools, 33(1) (2013) (in Ukrainian)
24. Spirin, O.: The criteria for external evaluation of quality of ICT education. Scientific jour-
nal of NPU named after M. P. Drahomanov. Series 2. Computer-oriented educational sys-
tems, 9, 3-7 (2010) (in Ukrainian)
25. Tikhomirova, N., Kozlov, A., Yasnov, K.: Development of internal e-learning quality as-
sessment system on example the Moscow State University of economics, statistics and in-
formatics. Open Education, 1(108), 26-32 (2015) (In Russian)
26. Williams, K., Kear, K., Rosewell, J.: Quality Assessment for E-learning: a Benchmarking
Approach. The Netherlands: European Association of Distance Teaching Universities,
http://e-xcellencelabel.eadtu.eu/tools/manual (2012)