Methods for assessing the quality of education are discussed, including the possibility of predicting learning outcomes based on the calculation of negentropy. Negentropy is used as an integral informational index, demonstrating objective assessment of the learning model used. We propose to use the emergent learning model as a generalized projection of the learning process with a reasonable fusion of e-learning and traditional learning. To create a multi-criteria system for assessing the quality of education we propose to evaluate all components of the pedagogical system in interaction with each other, at the first and all levels of the hierarchical levels. We also propose to define as a special category of programs in the system that interacts with other elements of the system, and thus the system is organized as “student-centered”. The proposed methodology for predicting and assessing academic performance is based on building on building a hierarchical structure of multi-criteria learning quality system. At the first stage, we selected the main ones that affect the quality of education, which are organized in the first level of the Ishikawa diagram. Then, based on the knowledge of experts, we assigned each criterion an appropriate coefficient of importance for the quality of training. At the second stage, the same was done for the second stage criterion of the third level of importance. In cases where it was not possible to unambiguously estimate the coefficient of importance, we present the system of equations for calculating the membership function of a fuzzy set. At the third stage, the integral values of negentropy were calculated for three university blended courses and for a model situation.
The boom in online learning caused by the COVID-19 pandemic is unprecedented in the history of distance learning. A massive shift to distance learning technologies (DLT) at different levels of education was impossible to imagine recently, especially in schools.
On the one hand, the shift gives a wide prospect for conducting an experiment in literally “field” conditions and analyzing the learning outcomes at a scale, which was previously impossible to dream of. On the other hand, one cannot but take into account the circumstances when the distance learning technology was literally “thrown at the embrasure” in order to keep the educational process afloat. In the context of the pandemic, when there was no time to comprehensively think over the structure of courses’ content and learning scenarios, choice of tools, training of teachers and students, were there an opportunity to get a quality learning process? Let's honestly admit that no, it was not possible – as well as production and implementation of any complex product in conditions of a shortage of time, necessary funds and resources. But distance learning technologies have already been designated guilty of improper performance of the educational process. We will discuss some of the most common mistakes that have been committed in massive shift to implementation of distance learning technologies and measures for providing high-quality emergent learning in the future. 2
Features of “сoronalearning”
Probably, the first thing that was done almost everywhere was an attempt to completely transfer the face-to-face learning without any changes to online mode through video activities.
Traditional face-to-face classes timing was applied to all learning formats: lessons, seminars and lectures. Moreover, it was required to follow all traditional lessons timings in the webinar mode: if a traditional lesson lasts 45 minutes, then a webinar should also last 45 minutes. In some educational organizations, timing compliances was strictly monitored by administration. It was not taken into account that such requirements were violated the State sanitary and epidemiological rules and regulations, especially for primary schools.
We will formulate here only a few questions, answering which, perhaps, it will be more clear how to build the educational process using distance learning technologies and e-learning in modern conditions: ─ Is strict adherence to face-to face classes timing in transition to distance learning a prerequisite for the high-quality learning? ─ Do lectures in obligatory synchronous online mode guarantee the quality of education? ─ Who should form the scenario of the educational process: a teacher, an administrator of the e-learning platform used an administrator of the educational organization? ─ Is it legitimate to present the same requirements for the learning process and use the same type of distance learning models for all levels of education and fields of study – natural sciences, humanities, economics, engineering, medicine, etc.? The history of distance learning technologies and e-learning counts more than one decade, the fundamental principles of DLT and EL developed include the following: carefully planned design of the learning process, taking into account the target group, form and level of education; developed online course (preferably tested), which meets the requirements of DLT and EL; obligatory guide/guidebook for the course| both for students and teachers; registration of students’ independent work in the e-learning platform used; convenience of work with online courses for both teachers and students. For example, in case of large number of students located in different time zones, listening or watching video materials occurs at a time when it is convenient for each student. Insisting on the simultaneous participation of tens and hundreds of students, for example, in online lecture does not seem entirely reasonable. Moreover, within the MOOC framework it has already been convincingly shown how video materials can be used in the most effective ways; availability of technological, organizational and consulting support (administrators, curators, tutors, coaches, mentors) for teachers and students; participation of trained teachers (often authors and designers of their own e-course) with high level of ICT and distance learning competence, which include the following: methods of online education; educational software; e-learning technologies; relevant ICT and e-learning terminology; compliance of curriculum requirements, teaching materials, and learning conditions; integration e-learning resources into educational process; methods of developing and managing online-courses in electronic learning systems; communication in electronic learning environment; writing scripts for 3D, VRML-models, etc.; group work communication tools; methods of students motivation for efficient learning in electronic learning environment.
Serious problems with teachers’ ICT, DLT, and e-learning competencies were highlighted precisely in the shift to distance learning technologies in the context of the pandemic, aggravated by time pressure. These problems seems to be unexpected in the situation of implementation in recent years of numerous projects in the field of digitalization of education, development of state requirements to e-learning environments and resources and variety of programs for teachers’ ICT and e-learning training. But in a result teachers should have fulfilled the requirements of administration: to work online according to face-to-face classes timetable and answer students' questions by e-mail. Do these requirements apply to current trends in e-learning, blended learning or distance learning? Not at all. However, distance technologies are blamed that the educational process does not line up, videolessons do not give the expected learning outcome and turn the teacher's work into an “online hard labor”.
And, finally, there is one more problem – insufficient number of e-courses, developed by the teachers. These courses could be used in distance learning (not only in pandemic context), in blended learning as digital component of full-time traditional courses, as the extra students’ feedback channel and resource for practicing e-teaching skills.
Approaches to assessing quality of online learning
In modern realities, the issues of assessing the quality of the educational process
using online technologies remain relevant and are actively discussed in professional
communities, including from the point of view of assessing the effectiveness of using
information and communication technologies in the educational process. Based on
publications [
The term “quality of the educational process” is proposed to assess the compliance
of the educational process with a certain quality standard that is used in the
educational organization and takes into account all the components of the pedagogical
system. From this point of view, the assessment of the quality of combined or emergent
learning [
The requirements for the structure of resource support at a university described in detail are schematically presented in Figure 1.
If we analyze each of the five compulsory components of comprehensive support for a successful educational process, then the reasons for obtaining a “surrogate” rather than a distance or blended learning process become obvious.
Briefly, we can formulate some typical obstacles to implementation of DLT and EL: technological and psychological unpreparedness of some teachers and students to online learning; lack of proven methods and experience in organizing a large-scale educational process online; insufficient amount of quality e-learning courses; attempts to use the maximum of the Internet technological capabilities without reasonable justification.
Some modified forms of organization of the educational process in the conditions of EL and DLT are presented in Table 1. As shown in Table 1, there is no need to use exclusively video lessons in a synchronous mode for conducting classes, there is a wide range of alternative modified forms. Moreover, it should be emphasized that considerable experience has already been accumulated in their practical use, and therefore, it is possible to avoid typical mistakes made at the initial stages of development and implementation of various learning models. ─ way of carrying out: information- ─ lecture in VR, al (classic), problematic, binary ─ webinar mode (synchrodiscussions, provocative lectures, nous or asynchronous lectures-conferences, lectures- recording) consultations, ─ goals: learning, informational, educating, developing, ─ video lecture, ─ web lecture ─ content: academic, popular sci- ─ internal or external navience gation through Internet resources
Depending on the method of conducting: ─ seminar-conversation, ─ seminar-discussion, ─ seminar-conference, ─ problem seminar, ─ seminar press conference, ─ brainstorming workshop, ─ specialization seminar, ─ webinar (synchronous), ─ online seminar in the Moodle module with offline peer review; ─ webinar in on- or off-line modes, ─ wiki seminar; ─ work with animation models and simulators, ─ work with the glossary, ─ peer-learning workshop ─ work in VR or with AR
work
In a specially equipped room (with devices, construction kits, ─ a virtual laboratory (in machine tools, tools, reagents, uten- VR or with AR) for worksils, etc.) for: ing with simulators for real installations, research ─ mastering the technique of exper- objects, experimental iment, conditions; ─ experimental confirmation
theoretical statements; ─ formation of the ability to solve practical problems by setting up ─ electronic experiments, complex, ─ formation and development of ─ interactive manuals skills to work with devices, equipment, installations; of ─ remote laboratory (hardware and software complex), laboratory ─ formation of the ability to safely
work with chemical ─ reagents and labware ─ visual presentation of the conditions for performing the experiment, measuring instruments necessary for a real experiment; ─ selection of the optimal parame
ters for the experiment; ─ obtaining skills in drawing up plans, schemes for organizing a laboratory experiment
Preparation for seminars, labora- ─ working with an online tory and practical work, tests, course (text, photo, video Olympiads and conferences materials), ─ completing training tasks, ─ work with simulators and and tive ment
Formative summaassess
oral by tickets, written by tickets, Tests, execution and defense of a creative task,
defense of the final qualifying work
training modules, ─ work with test systems in
self-test mode ─ remote testing with proc
toring, ─ video survey, webinar, ─ completing tasks on ticket issues within the deadline and posting answers to the site
An integral assessment of quality of the educational process can be performed on
the basis of the identified major categories in the Ishikawa diagram [
Depending on the teaching model used, the form of education, target audience and indicators of achievement of competencies, the categories of all levels will differ, but the following regularities should be preserved: ─ the category of the first level should be no more than six or seven, taking into account the method of forming cause-and-effect relationships when constructing the Ishikawa diagram; ─ major categories should reflect the main elements of the system under consideration, in this case the pedagogical system; ─ the hierarchy of the system under consideration is a sign of its stability as a dynamic system, and then, the more detailed all the nested categories are revealed, the more clearly the strengths and weaknesses of the pedagogical system are revealed; ─ a set of categories of all levels provides the emergent properties of the pedagogical system.
On example of developing a methodology for assessing the quality of the
educational process in Chemistry courses the major categories that affect the quality of the
competencies formed in the course were identified. It was proposed to distinguish
seven such categories [
The weight coefficients are determined by the normalization method based on expert assessment.
The maximum values were received for such a category as “Laboratory” (see Fig. 2).
Then, in the hierarchy of importance, the category “Teacher” follows as a subject of the pedagogical system, which is beyond doubt since the teacher acts as a designer and organizer of the educational process and the second-level categories for all other basic criteria depend on the level of teacher’s competence and motivation.
Each of the major categories can be represented as a set of categories of the second and third levels. In this work the major category “Student” (including the categories of the second and third levels that affect student’s level of learning in the context of an emergent approach to learning) is considered in detail. Table 2 shows the categories associated with both the student's own psychological and intellectual abilities and the criteria determined by the external environment: the student group, the educational organization and the educational information and learning environment.
The degree of matching of expectation from the learning process and a real situation
The use of gamification elements (indication of learning progress, etc.)
Demand for the results of each student's activity in the success of the group / team
Convenience of schedule
Basic training formed at the previous stage of study and confirmed at additional entrance tests or tasks.
Ability to Willingness to change cognitive structures by receivlearn (formation ing and processing information of specified The level of development of cognitive processes: competencies) perception, thinking, memory, attention, speech
The level of development of the emotional-volitional sphere: perseverance, purposefulness, poise.
Electronic component: device and gadgets, software,
Internet access
Availability Electronic learning environment (system, platform) of resources Full-time component: equipped laboratories, materials, reagents, instruments, dishes
Teacher, tutor, coach, curator, site administrator
Communica- Among students within the group and withing the tion and support similar groups
Teamwork on joint projects, peer review and evaluation of works
Information and communication competences
User friendliness of the learning environment interface
Matching the psychological characteristics of the personality with the type of activity.
Psychological readiness to work in electronic learning environments
(ELE))
One of the ways to assess the quality of online teaching is an anonymous survey of students.
The results of survey of Geology students for the General Chemistry course are shown in Figure 3. The assessment was carried out on a five-point scale and demonstrated good accessibility of the course; high involvement of students in online learning based on the electronic educational and methodological complex (EEMK); satisfactory indicators for the electronic learning system (ELE) used, taking into account that the platform was unfamiliar to students.
A comparative analysis of qualitative indicators for assessment of academic performance for five academic years of work with the course and implementation of point-ranking assessment system is shown in Figure 4. As data in Figure 4 show, there is a decrease in the percentage of students who did not take the exam with practically insignificant fluctuations in the quality of academic performance and average score. The results obtained, in our opinion, can be explained by the fact that the combined use of the online course and point-ranking assessment system leads to an additive effect. This effect is expressed, on the one hand, for students – in the need to systematically work independently with the course materials and to pass tests to check the preparation for laboratory work; on the other hand, for the teacher – in requirements to update the learning outcomes data using the electronic journal module and the course materials on regular basis as well as actively communicate with students using feedback modules.
Calculation of criteria values based on fuzzy sets
The issues of using and calculating weighting factors for various groups of
categories remain open, although the processing of an array of expert data on taking into
account the significance of different-level criteria in a hierarchical pedagogical
system can be considered as a classic problem of using fuzzy set algorithms. It is shown
[
The calculation of the numerical value for the criteria of the second level of the Ishikawa diagram was carried out on the basis of systems of equations of fuzzy sets.
Based on the criteria of the second and third levels for all basic categories, assessed
dichotomously or using fuzzy set algorithms, when the membership function has an
asymmetric two-sided Gaussian distribution [
J=wi·ki,
where ki is the numerical value of the criterion, and wi is its weight coefficient. For
the calculation of negentropy, the methodology which made it possible to obtain
comparative data on the influence of “corona learning” on the educational process
carried out with DLT and EE [
Figure 5 shows the results of the calculated value of negentropy, taking into account all the major criteria that describe educational processes for two majors – Geology and Chemical Sciences and for a model situation when all the criteria are the most favorable and assessed with the maximum score.
Based on the set of criteria selected for the analysis, it is possible to make predictions about the achievement of a given level of quality of the educational process for various majors.
The desired assessment can be carried out by choosing those criteria that are
determined by the internal quality standard of the university / faculty / institute. To
observe the phenomenon of emergentism [