The authors consider the problem of improving the credibility of diagnosing the learning outcomes of university students in the process of e-learning. The main solutions include improving material, technical and regulatory framework of e-learning, the use of proctoring control and monitoring of distance certification tests, the enhancement of professional and pedagogical competence of lectures including those engaged in the implementation of digital teaching technologies, and their readiness for innovative activities in electronic informationand-educational environment, as well as the development and use of diagnostic materials complying with the requirements of credibility and validity.
The diagnostic component is one of the key integral parts of the study process. It is entirely responsible for receiving feedback, determining the efficiency of training, and making the necessary adjustments. The rapid development of information and communication technologies in modern society has led to the digital transformation of education at all its levels, and the increasingly widespread use of e-learning. This process accelerated significantly in 2020 due to the forced transition of the education system to a distance learning format due to the dangerous epidemiological situation caused by the new coronavirus infection. Educational institutions that purposefully and methodically developed their electronic information-and-educational environment, upgraded their material and technical base and actively enhanced the digital competence of lecturers and students proved to be better prepared for such force majeure circumstances. But even such institutions faced a number of difficulties in the transition to e-learning, including those associated with the need to ensure credible and valid results of assessing students' learning outcomes. In this respect, the use of proctoring understood as controlling and observing distance certification testing is considered one of the main prerequisites needed to solve this problem.
1) to study in-depth the concept of pedagogical diagnostics and its basic principles; 2) to identify specific features of diagnostic procedures in e-learning, including the use of proctoring;
3) to formulate pedagogical conditions for ensuring the credibility of diagnosing students' learning outcomes in e-learning.
The problems of ensuring objective assessment of students' knowledge have interested educators at
all times. The word “diagnosis” is translated from Greek as “recognition”, and diagnostics is a process
of determining the state of an object [1; 2]. The information obtained as a result of diagnostics
contains data on the state of the object, which allows assessing the degree of its compliance or
noncompliance with the preset normative requirements. Based on this information, it is possible to predict
the further development of the object and to exert corrective effects on it [
Pedagogical diagnostics should comply with the principles of reliability, validity, objectivity, consistency, predictability, operational feedback, and academic ethics.
Any assessment procedure is generally perceived by people as a certain stress, and therefore, in the process of pedagogical diagnostics, it is so important to create a comfortable psychological environment for students [4; 5]. Diagnostic procedures should perform developmental functions instead of punitive ones, but at the same time, if you do not immediately outline the boundaries and set the diagnostic rules, recurrent cases of academic dishonesty on the part of students are highly probable, including those occurring during online exams [6; 7; 8].
The study process, implemented in a distance format based on digital technologies, is characterized by specific features that have been examined and described in many works [9; 10]. Of course, e-learning is not able to completely replace live communication between a lecturer and students, but in some cases it can be very valuable and even irreplaceable, for example, for teaching people with disabilities or for organizing distance learning in a self-isolation mode during an epidemic period. Therefore, studies aimed at improving the theory and practice of the digitalization of education, at optimizing e-learning, including diagnostic procedures, will long remain in the area of increased attention of the scientific and pedagogical community.
The study was carried out on the basis of the Faculty of Humanities and Pedagogy of Russian State Agrarian University – Moscow Timiryazev Agricultural Academy.
The research methodology is based on a polyparadigm approach, which organically interconnects the systemic, activity-based, competence and personality-oriented approaches.
Research methods include the analysis of scientific and pedagogical literature on pedagogical diagnostics and the organization of distance learning in an electronic information-and-educational environment; included surveillance; expert review; interview; pedagogical experiment of organizing proctoring during distance certification tests.
The experience of implementing e-learning classes during the self-isolation period gained at Russian State Agrarian University – Moscow Timiryazev Agricultural Academy identified a number of problems associated with both ongoing tests, course exams and state final certification, as well as with the organization of school olympiads. Most lecturers, as a rule, know their students personally, so the identification of the students’ personality was rather formal (procedural compliance with the university's local acts on the use of e-learning and distance learning technologies was recorded). However, when organizing olympiads for schoolchildren, the organizers found it necessary to reliably identity the participants, since the assessment commission saw them for the first time. In addition, in both cases, it is important to objectively assess the knowledge of students, to prevent acts of cheating and getting external assistance from third parties. To solve this problem, use can be made of proctoring. It is a control procedure performed during an online exam or testing, where the whole process is watched by an administrator – a proctor who identifies a person taking the exam, watches his/her actions using a webcam and sees an image of the computer desktop.
The term ‘proctoring’ was first used in 2008 in the United States – initially in a simplified version, when the administrator monitored the course of the exam through a webcam and manually recorded violations. Later on, auto-proctoring appeared, where the program independently verifies the student's personality, monitors his/her behavior and gaze direction, analyzes sounds in the room, fixes violations on video, and prepares reports. A combination of the proctor activity and the autoproctoring program seems to be the best option.
An important condition for proctoring to be organized properly, objectivity, and impartiality is that it should be carried out either by a specially trained professional, or an independent expert from another educational institution, training department, or private company.
In addition to online training, proctoring is used for recruiting purposes. For instance, using proctoring-assisted tests, HR managers can check the professional competencies of applicants from other regions and countries; select the best candidates for a position, thus saving their own time and their company’s money. Furthermore, it can be beneficially applied for testing and certification of personnel (when diagnosing hard skills of employees before personnel rotations or when making decisions on salary increments). Proctoring helps to objectively assess the knowledge of employees and prevent any attempts to falsify test results.
For example, ProctorEdu, cooperating with many companies and educational organizations, conducts proctoring when hiring employees, attesting personnel, examining students, etc., preventing the facts of changing the identity of test takers, leaking internal materials and cheating. The credibility of the results is achieved through face recognition throughout the session and intelligent behavior analysis during testing. With such technologies, companies can be assured of the competence of their employees, and educational institutions will be able to reliably check the level of students’ knowledge. The trust assessment algorithm and the face recognition system for identity verification and confirmation of online exam results works well in a browser and does not require the installation of extensions, plugins and third-party software. Relevant proctoring scores include trust score with details of violations, biometric verification, video protocol, PDF report. The results are uploaded by API and in the table form.
The proctoring system works on the cloud storage basis with the Internet access, it is installed on the server and requires the facilities of the organization conducting the test, provides complete data control and can work within the local network without the Internet access. Updates, maintenance and all technical difficulties are hidden features, and users are provided with the required service only. The proctoring system integrates with the testing system of a company or educational institution, for example, the StartExam, Stepik and Moodle platforms. The system verifies the identity of test takers and monitors human behavior throughout the test. Upon completion of the test, the proctoring system analyzes the data, estimates the credibility level of the results, and generates a report. There are three main proctoring options: full automation (the proctoring system observes a test taker using 13 behavioral trackers, and issues a report on the credibility of the results after the end of the exam: a camera with sound and screen recording; identity verification; behavior tracking; credibility report); human verification (the proctoring system observes a test taker, after which the proctor can view the video to confirm the credibility of results); and verification of protocols by the proctor (proctors monitor the testing process in real time, observing the process through the camera and the test taker's desktop in order to prevent acts of academic ethics violations).
An analysis of practical experience in conducting various online tests has shown that exam participants, as a rule, tend to commit fewer violations if the proctoring system is used, even as compared to test procedures conducted in the classroom.
Prior to full transition to distance learning at the University level, the University approved key provisions for the implementation of e-learning and the use of distance learning technologies for all study programs, including recommendations developed for organizing ongoing exams, presenting course papers and negotiating academic debts, and carrying out State final certification on the basis of the following ICT tools:
1. Personal account of students and staff members designed for operational interaction of the study process participants enabling to upload and exchange files for study purposes.
2. WhatsApp – an instant text messaging system for mobile and other platforms with voice and video support.
3. Skype – a software solution that provides text, voice and video communication over the Internet.
4. Mirapolis – a system for web conferencing, online training, taking exams, etc. 5. Zoom – video conferencing with real-time messaging and content sharing.
6. LMS Moodle – learning management system (an electronic educational resource with recording the study progress).
Since the existing systems for video conferencing lacked auto-proctoring capabilities, when taking exams and tests, regardless of the software that the lecturers chose, they had to act as proctors, i.e. check the quality of video and sound, carry out identification of the student's identity (the students gave their last names, first names and patronymics, showed their identity documents: a passport, a grade book, or a student card), observe compliance with all requirements while preparing for communication, avoiding possible violations with handing over and summing up the results, taking into account the appropriate organizational and pedagogical conditions.
About 140 full-time lecturers at the University's Faculty of Humanities and Pedagogy were assigned to give exams in such an unconventional format. More than 70% of lecturers noted that during the certification procedures, some students tried to violate the requirements and recommendations of the university's local regulations related to the implementation of e-learning and distance learning technologies (Fig.1). It should be noted that if there were an auto-proctoring system, this percentage could be much lower, since the automated system is much more objective. However, the use of such a system requires significant financial costs on the part of the educational institution, and, in our opinion, it should be applied at the most critical stages of study at the university, which include the State final certification (provided it is carried out in an online format). The current certification procedure can be implemented without the use of auto-proctoring, limiting itself to the activities of a teacher-proctor only, which will reduce the cost of purchasing the appropriate software and additional personnel training. These conclusions are consistent with the opinion of lecturers, who note that in case of ongoing assessment in an online format, they are ready to carry out the proctoring procedure independently, without attracting additional funds (this opinion was expressed by more than 80% of respondents).
During the State final certification (including the State exam and the defense of a final qualification project), the secretary of the State Examination Commission was already acting as a proctor, while implementing all control measures under the supervision of the Commission Chairman. More than 150 bachelors and masters graduated from the Faculty of Humanities and Pedagogy of the University in the 2019/2020 academic year, and the examination commissions carried a heavy workload associated with procedural compliance with all the requirements prescribed in the regulatory documents. During the State certification activities, the Commission members also noticed certain procedure violations by students (attempts to open documents on a local computer or other device, the use of text cheat sheets, or assistance from third parties). All these violations were recorded and instantly stopped, which required close attention from the Commission members. The experience gained calls for wider application of auto-proctoring systems to ensure a more objective and unbiased control over the certification procedure.
Summarizing the experience gained while organizing the State exam for students of the Faculty of Humanities and Pedagogy of Russian State Agrarian University – Moscow Timiryazev Agricultural Academy, we will formulate the pedagogical conditions for ensuring the credibility of diagnosing students' learning outcomes in the process of e-learning:
1) the availability of high-quality equipment and Internet connections for all participants of the study process , as it is vital for the implementation of stable video and audio communication, as well as the ability to record the certification procedure for possible enquiry on results (if necessary); 2) the availability of software that would make it possible to implement auto-proctoring to reduce the workload on the examiner and increase the objectivity and credibility of the procedure itself and the results of pedagogical diagnostics. As an option, we suggest using the Examus online proctoring system. This tool both provides for personal identification using a face recognition system, and automatically monitors human behavior in terms of compliance with the rules for remote diagnostics of learning outcomes as well as minimizes possible violations;
3) approval of normative acts on the use of e-learning and distance learning technologies with clearly prescribed procedures and requirements, and full informing of students on all the peculiarities of organizing certification procedures;
4) increasing the professional and pedagogical competence of lecturers (in particular, its digital component), their readiness for innovative activities in the electronic information-and-educational environment [11; 12], to the development and use of diagnostic materials that meet the requirements of credibility and validity.
To sum it all up, we can emphasize the fact that the digitalization of society in general and the professional education system, in particular, encourages educational institutions to focus on innovative development models, quickly master modern technologies, and overcome the longstanding obstacles characteristic of the education system. To ensure the high quality of personnel training, it is necessary to modernize all components of the pedagogical process: target, meaningful, technological, and diagnostic ones, especially in terms the wider use of e-learning. Improving the quality of diagnostic materials, their diversification according to the levels of mastering the studied material, bringing them into compliance with the requirements of credibility and validity, the use of proctoring in the process of certification activities – all this factors are certain to increase the efficiency of assessing the learning outcomes and raise the level of public confidence in e-learning and distance learning technologies.
[12] P.F. Kubrushko, L.I. Nazarova, Professional Development of Technical University Lecturers in Field of Innovation Teaching, in: 2013 International Conference on Interactive Collaborative Learning, ICL 2013 (2013) 467-469. doi: 10.1109/ICL.2013.6644627.