The article contains the results of research on the procedures for creating and adjusting tests in learning management systems based on an active dialogue with students and automated analysis of test results. The authors performed research on the procedures for creating, evaluating quality, and adjusting tests based on the electronic learning management system JetIQ. To evaluate the optimized method of creating and adjusting tests in learning management systems, the authors use modules to assess the quality of tests, feedback; modules for the analysis of answers to questions. Students assessed by tests do this several times: by topic; on the intermediate control of knowledge (colloquium) and the final control of knowledge (exam). This approach allows to make changes to the procedure of the final assessment of the knowledge, make adjustments to questions, and select the most correct to combine in the exam. The resulting student activity profile allows the teacher to be more objective when using test scores. The results of the evaluation of the method of optimization of test adjustment procedures are also presented, which indicate a significant efect in saving time.
The learning management system should be an information ecosystem and cover all educational processes. The principle of the ecosystem involves the reuse of information that is entered once into the system. Among the various modules of the learning management system on the example of the author’s system, JetIQ VNTU can be distinguished primarily by the ofice of teacher and student.
They form the basis of the information ecosystem. Educational processes are also automated using the electronic dean’s ofice module. Information is provided through news systems and jet sites of departments. The ofice of the teacher of the learning management system contains various modules. The “My repository” module is used to download electronic resources. Providing access to students under the educational programs of the specialty is implemented using the module Navigator of educational resources of the discipline. The IQ test module is used to control knowledge by testing students.
Improving testing modules in the learning management system is a topical issue for educational institutions that actively use testing tools. The format of distance and blended learning involves the use of tests both during training and during control activities.
Among the known practical approaches to the formation of test tasks and improvement of
testing tools in learning management systems include the use of tools for quantitative and
qualitative assessment of students’ knowledge in learning management systems, analysis of
learning outcomes for the formation of test groups [
The testing module is one of the most complex and intelligent modules for the following reasons: • The testing module should provide the opportunity to create questions of diferent types and conduct testing in diferent modes - training and examination. • The testing module should make it possible to assess the quality of the test according to certain parameters and, ideally, such an assessment should be carried out automatically. • Test validation and verification procedures should also be automated.
In various platforms of learning management systems (Moodle, Collaborator etc.) [
Best practice in using tests involves using a test to assess students’ input knowledge; during
training to strengthen the acquired knowledge and skills. In addition, it is advisable to use
questions in diferent tests to identify errors and features [
The practice of active use of feedback tests, especially instant feedback, allows to open a new
level of dialogue "student-teacher", in addition, the teacher will better form an understanding
of the student’s knowledge and activity [
Active use of learning management systems involves the use of various tools for assessing the knowledge and activity of the student. We can note that student activity can be efectively used both to motivate learning and to collect the use of this data to improve the quality of testing.
Distance and blended learning involves the active use of formative and final assessment of
students’ knowledge through tests with built-in modules for the accumulation of statistics and
feedback. Such tools for analytical evaluation of the obtained results are partially implemented
in diferent LMS, but are not used fully enough. One example is the use of analytical assessment
in a civil engineering course at an Australian university [
Student activity, their assessment is the basis for forming a student profile. It is based not only on test statistics but also on other student activities. In particular, they can receive statuses, awards, medals for activity, evaluation of dialogues, for the number of tests performed, qualitative assessment of the knowledge, etc.
This comprehensive approach to assessing and motivating students allows them to form their profile and obtain quantitative and qualitative assessments for the study of the discipline.
Providing students with information about their achievements activates the mechanisms of motivation and increases the level of their activity in the e-learning system and includes additional feedback loops.
This allows to run procedures to actively verify the quality of test questions based on teacherstudent relationships. Current training during the school year is optimal in terms of time to gather information on the quality of test questions and identify those that do not meet the requirements.
The experience of developing and implementing a training management system and
supporting scientific and methodological activities of teachers JetIQ VNTU [
The general data of the student’s activity include completed tests, tasks, files sent to teachers, participation in lectures, use of lecture material, dialogues in chat and forum, etc. (Fig. 1).
The well-known method of trial and error is actively used by teachers around the world when creating tests. The main phases of this method are presented in Fig. 1. The test will be called a pool of questions on a particular topic and/or the entire discipline. This approach has its historical roots in the use of paper-based testing. But even today the phase of improving the test, which is created in an electronic system that has no special modules for quality assessment and verification of issues.
Development. The duration of this phase depends on many parameters: 1. The total number of questions in the tests; 2. Areas of knowledge; 3. Availability of formulated closed and open answers; 4. Availability of calculation parts in the test questions; 5. Availability of graphics in questions and answers; 6. Use of tests in which several correct answers are chosen; 7. Using tests to compare questions and answers.
It is extremely dificult to identify the main factor influencing the time of test development and therefore we can only roughly focus on data on test development based on the experience of VNTU teachers.
Consider the features of the procedures for creating and adjusting the test using special electronic modules of the JetIQ VNTU system.
The teacher creates a test, publishes it in an electronic test system. Students take this test and receive test results. They form an information base for the automatic analysis of test data.
All the data obtained on the evaluation of the quality of tests, errors in them, the distribution of scores in the questions form an information base for the adjustment of the test.
If the tests are presented on paper and are static in any electronic testing system, or such systems do not have analytical units for assessing the quality of tests, then all the procedures for adjusting the tests are carried out by the teacher himself. After adjusting the test, it can be re-applied.
Improving the quality of tests is a repetitive procedure that is performed cyclically. Subject to the formation of tests on topics and the assessment of students several times, such an adjustment can be made according to the results of each test. The level of test quality increases gradually by implementing the following steps: • Error questions are corrected after errors are identified by the teacher and / or students; • The issue of the correct derived mainly or mostly wrong answers holes in them are not qualitatively formulated and well adjusted; • To increase the level of quality assessment, the teacher should try to design the test in such a way that on the one hand, the student’s answers to the questions allow to assess the level of knowledge as accurately as possible. On the other hand, test questions should be designed in such a way as to minimize the percentage of guessing the correct answers. In our opinion, this criterion is best met by questions with randomized input conditions and a computational test program. It is also important to have a large enough number of questions. With our estimates for one discipline, they should be at least 80-100 for one credit of the discipline.
The experience of using the testing unit in the JetIQ VNTU system shows that such an adjustment is widely used during the training process. Also, tests are used not only on individual topics but also on intermediate control measures. Their corrected questions are the basis for combining the final exam tests.
Also, many student activities are recorded in the e-learning system. These include completed tests, tasks sent to teachers, participation in lectures, use of lecture material, dialogues in the chat and forum, etc. (Fig. 3).
Such work with tests is a motivation for students to increase the level of their activity in the e-learning system and gives the student a sense of partnership in correcting questions. In general, feedback on the level of correctness of questions is one of the types of dialogue with the student during testing. This comprehensive approach to assessing and motivating students allows to form a student profile, to understand how active he was during the study of the discipline.
The student’s profile is based not only on statistical data of passing tests but also on other activities of the student, in particular the badges received by him, medals for activity, estimation of dialogues, a quantity of passing of tests, qualitative estimation of knowledge, etc.
Let’s estimate the duration of the process of optimization of the quality of the test at the use of an electronic system of training. The duration of the stage of its creation is , similar to the creation of tests on any medium and in any electronic system. The main time is spent on the formation of questions, answer options, calculation tasks, comparison tests, preparation and implementation of graphics, checking the formed tests, and reviewing - as students see them.
Let form the results of the analysis of the quality of the test and its components ( 1 − 1, ).
According to the analysis, the teacher forms the necessary changes and makes the system − 1, be the testing time of students. After testing, the modules of the electronic system 1 − 2, the number of errors and inaccuracies in the questions in the previous stage was reduced, then for the next stage their number usually decreases, ie the duration of subsequent phases decreases from 2 − 2, , which is why the organization of time for training tests should not be limited. Repeated passing of tests is recommended for students in the mode of studying materials. This will allow the teacher to make adjustments if necessary almost continuously before the exams. The results of the adjustment afect the growth of the test quality 0 − , determined by: and decrease the variance in the answers to questions 1 − , .
Calculate the test optimization time. Let , be the time to create the test; - time to correct questions in the nth iteration. The total time to create and optimize the test is − −1 =
− , adjusted if 0 < 1,
Let’s perform a situational calculation of the required number of adjustments to the test questions. Let the test have 0, questions that need adjustment. The test will be considered
Let’s estimate the adjustment time on the −ℎ, phase of the iteration, taking into account the types of questions. For a test that has a total number of questions N on the −ℎ, iteration, the results of the correction form a general adjustment Q, which consists of adjusting the following , - question of incorrect wording; , - questions with incorrect answers. types of questions , - a question with an ambiguous answer; , - questions of easy guessing; = + + + .
We introduce correction coeficients to the number of fixed for each type of question. Consider the case where the questions belong to the same type = 1 − After the first adjustment we have the following number of incorrect = ∗ (1 − ) + ∗ (1 − ) + ∗ (1 − ) + ∗ (1 − ) , which will characterize the ratio of questions with errors (1) (2) (3) (4) (5) type will be indicated in lower case)
= + + + . adjusted questions can be calculated as We will assume that in the process of their correction the teacher may make mistakes with probabilities ; ; ; according to each type of question. Then the number of correctly
For each stage, the calculation equation for Q f will be similar (the number of questions by questions:
1 = 0 ∗ = 0 ∗
ln 0 >
ln For the presence of questions of diferent types of expression can be represented as follows (9) From the last formula, we can conclude that the required number of iterations should be equal to (6) (7) (8) (10) = 0 ∗ = 0 ∗ < 1 > ln 10 ln > ln + ln + ln ln ln ln + ln ln We introduce the value t, which will characterize the average time of the teacher to correct one question of a certain type. Then the total time to adjust the test questions can be written as ln ln ln ln = Δ + Δ +
Δ + ln ln ln Δ ln Note that the calculations are valid for cases where students pass incorrect questions in a suficient number of times
. Therefore, the minimum duration of repeated students and each of them is randomly ofered questions, we can write should depend on this value and the time interval between the phases of the test the factors that afect the total number of passes . Consider the ideal case when reaches the minimum value of 1. Then, for the total number of questions in the test , when it passes − −1 (Fig. 4)
. Determine =
∗ = ∗ ∗ (11) (12) (13) (14) (15) To reliably diagnose incorrect questions, the number of tests done should be as large as possible. But in practice, this number is limited to a certain amount of .
The duration of the interval
− −1 − −1 = ∗ ( + ) =
∗ ( + ) ∗ ∗ Where the interval - characterizes the period between the possibility of re-passing the test and - the average time to answer the test questions.
For example, exams and control tests can take several months. This significantly complicates obtaining the necessary amount of data for quality diagnosis of incorrect issues. For training tests, the interval can be significantly lower. In the case of automated training systems, it can be reduced to zero. Then the minimum value of the phase of the period between the possibilities of passing the retest is reduced to zero (no special passage is required, this is done by students in the phase of training and / or intermediate control and the duration of the interval −1 − −1 = ∗ ∗ =
∗ ∗ ∗ ∗ Thus, the optimization of adjustment time is associated with the organization of periods of opportunities for students to take tests.
The motivation of the teacher and the student in organizing the organization of training tests with a suficient number of passes is that students gain knowledge for the number of attempts (learning is excluded by the content, diferent types of tests, and their random mixing). Also, the teacher receives statistics on the verification of test questions. For this purpose, the teacher uses test questions on topics in colloquia or practical or laboratory classes.
If the teacher does not use the training tests, the adjustment phase increases significantly. Verification statistics will be obtained by the teacher only after the exam. This will mean that the correction of test questions will be a posteriori and its results can be used only for the following groups.
To quickly optimize the test parameters, it is necessary to provide the following conditions: 1. Form a test by topic and use it in the current learning process for a large number of students. This will allow to implement the test procedure many times and identify poor quality issues. 2. Such training tests have no restrictions on repeating the test. 3. The period of data preparation for their automated analysis is proportional to the total number of questions in the tests and inversely proportional to the number of students taking the tests, as well as the number of questions ofered in the test.
Static tests that do not change and are not adjusted, especially on paper, have low suitability for optimizing their quality due to long periods of their application and the dificulty of accumulating suficient data.
Exam tests should be formed based on a combination of training, verified at a suficient level of tests.
The required quality of tests can be provided by special software modules that allow accumulating and process test data, to carry out a continuous process - "testing-processing-adjustment" for a short period of test quality assessment. Such software modules should have mathematical tools for analyzing the answers to the test questions, assessing the quality of the questions, the reliability of the results of the whole test, etc.
The testing module in the learning management system is one of the most complex. The testing process involves the active work of students and providing them with opportunities to feedback from the teacher on the results of testing. In addition, the accumulated statistical information allows you to analyze the results of assessment of students’ knowledge and the level of quality of tests and their individual questions.
That is the general statistics on the use of test control during 01.09.2019 - 02.11.2020: tests total number - 3238, answers to the questions of electronic tests TestIQ: 302365. Given that only the first modules were conducted and there were no final tests, it can be concluded that the active use of training tests and the accumulation of analytics base for their adjustment. Training tests are the basis for the formation of examination tests.
The prospects for the development of the testing module are: 1. Modification of the test evaluation system. 2. Development and implementation of new procedural modules such as reminders on the timing of training tests, development of error test reports, etc. 3. Development of the concept of using elements of artificial intelligence in the testing modules.
We express our sincere gratitude for the activity and patience to the teachers and students of VNTU, who participated in the processes of developing and improving the quality of tests in the JetIQ system.