In this paper the algorithm for adaptive testing of students' knowledge in distance learning and an assessment of its efectiveness in the educational process has been proposed. The paper provides an overview of the results of the application of modern test theory, a description and block diagram of the proposed algorithm and the results of its application in the real educational process. The efectiveness of using this algorithm for the objective assessment of students' knowledge has been experimentally shown.
Modern approaches to assessing students’ academic achievements are based on the use of
classical testing theory and Item Response Theory (IRT). The mathematical background of
pedagogical measurement theory was created by Andersen [
The purpose of this paper is to develop an algorithm of adaptive testing for objective assessment of students’ knowledge in distance learning, which becomes especially relevant in the quarantine of COVID-19.
The educational standards of the new generation are based on a competency-based approach to assessing the quality of a student’s training, when it is not his knowledge that is tested, first of all, but his readiness to apply it in practice and to act productively in a non-standard situation, the ability to create the required mode of action. Therefore, the quality of training is understood as the degree of the student’s readiness to demonstrate the relevant competencies.
The generalization of the world experience in the implementation of the competence-based approach to assessing learning outcomes allows us to make the following conclusions that determine the main approaches to assessing the level of competence mastery, the main of which are the following: • competencies are dynamic, since they are not an invariable quality in the structure of a pupil’s personality, but are able to develop, improve or completely disappear in the absence of an incentive to manifest them. Therefore, we can talk about the level of competence, assess it quantitatively, and monitor it. • when assessing learning outcomes, it is necessary to consider them in dynamics, which requires diagnostics of the educational process using monitoring procedures. • the level of possession of a competence is a hidden (latent) parameter of the pupil and direct measurement is not amenable. It can be estimated with a certain probability.
Therefore, when evaluating it, a probabilistic approach should be used.
It follows from this that in order to create tools for the automated assessment of the learning outcomes, it is necessary, first of all, to solve two problems: 1) develop theoretical and methodological foundations for modeling and parameterization of the learning process and the diagnostic tools used to evaluate its results; 2) theoretically substantiate and implement software-algorithmic means for processing the results of participants’ diagnostics (testing, questionnaires), as well as tools for assessing learning outcomes and the quality of diagnostic tools.
The theoretical and methodological basis for solving these problems was the study results,
ifrst of all, by Brown [
The theoretical basis for the creation of tools for automatic assessment of the results of the
educational process has received its further development due to the creation of the IRT (Item
Response Theory), the foundations of which are set out by Andrich [
Adaptive testing is a type of testing in which the order of presentation of test items and the dificulty of the next task depends on the participant’s answers to previous items. The basis of adaptive testing systems are statistical models. Very easy and very dificult tasks are automatically uninformative. Therefore, for most tests, the optimal level of dificulty is the item, to which the correct answer is given by about half of the test participants.
The dificulties of the test items is determined experimentally, and the measurement process consists of determining the percentage of participants who are able to give the correct answer to the task in previous experiments.
The problem of developing adaptive algorithms has been considered by Al-A’ali [
The Rush model was used to construct the adaptive testing algorithm. This model is defined by formulas: =
exp( − ) 1 + exp( − ) where is the probability that the participant , = 1, . . . , with the ability correctly performs the task , = 1, . . . , , with the dificulty . To start the algorithm it is necessary to determine the initial levels of dificulties. To this end, at the beginning of the testing session the accumulation of primary information about the level of preparation of the participant is carried out. To do this, participant receive tasks with an average level of dificulty. Tasks to determine the initial level of the participant are chosen by the teacher. Then, using the received answers, the initial estimation of the ability level of the student is calculated, and also recalculation of the dificulty level current values of test items is carried out.
The initial assessment of the ability level of the -th student (in logs) is based on the formula: 0 = ln ︂( )︂ , = 1, 2 · · · , where is the number of test participants, is the proportion of correct answers of the -th
The dificulty level of test items in logs is determined by the formula: participant to all tasks, is the proportion of incorrect answers ( = 1 − ). 0 = ln ︂( )︂
, = 1, 2 · · · , where is the number of test items, is the proportion of correct answers of all participants to the -th test item, is the proportion of incorrect answers.
At the next stage, the initial values in the logs of the ability level of participants 0 and the
initial values in the logs of the dificulty level of the test item
scale [
Pre-calculating the average value of the initial logits of the students’ ability level we get the formula for calculating the ability level logit of the -th student: where = ︁( 1 + 2.829 )︁ 21
.
The obtained values allow to compare the level of students’ ability with the level of test dificulty item dificulty. If − is a negative quantity and is large in modulus, then the problem of is too dificult for a student with the ability level , and it will not be useful for measuring the level of knowledge of the -th student. If this diference is positive and large in and the standard deviation of the initial values distribution of the parameter we obtain a formula for calculating the doficulty level logit of the -th item where Similarly, calculating = =1 ∑︀ 0
2 = =1 ∑︀ (︀ 0 − )︀ 2
, = + · 0, = 1, ,
√︂ = 1 + 2
⎯⎸⎸ ∑︀ (︁ = ⎷⎸ 0 − − 1
︁) 2 = + · 0, = 1, , (3) (4) (5) modulus, then the task is too easy, it has long been mastered by the student. If = , then the probability that the student correctly completes the task is equal to 0.5.
The information function of the -th problem for the Rush model (1) ( ) is defined as the
product of the probability of the correct answer ( ) to this problem on the probability of the
incorrect answer ( ) [
After calculating the information function, the measurement error is calculated, the value
of which is used to check the condition of the end of the test procedure. In the Rusch model,
the measurement error depends on the level of training and is calculated by the formula [
If the error takes a value less than the threshold set by the teacher, the adaptive testing algorithm ends. Otherwise, the following test task is selected. To select the next task, use the value of , calculated by formula (5). The next task is the one whose dificulty level is ( ) = 2 · ∑︁ ( ) =1 closest to the current assessment of the ability level of the participant. This task has the largest information contribution and its choice reduces the total number of required test tasks.
Thus, the developed adaptive testing algorithm consists of the following stages: 1. Selection of 5 tasks of average dificulty from the bank of questions, which is determined by the teacher. 2. Finding the initial level of student’s ability 0 and the initial dificulty level of items 0 by formulas (2) and (3). 3. Reduction of the obtained initial values 0 and 0 to a single interval scale using formulas (5) and (4). 4. Calculation of the information function of test tasks to which the student answered by formulas (6) and (7). 5. Finding the measurement error by the formula (8). 6. If the measurement error is less than the threshold, the adaptive testing is completed. 7. If not, then the next task is selected from the condition | − | = min. 8. Then the algorithm is repeated starting from point 3.
The block diagram of the algorithm is shown in figure 2.
Let us consider the procedure for calculating the parameters of student ability level and item dificulty parameter from empirical data. As initial data we will take results of testing of students in Moodle system on discipline “Higher Mathematics” of the Mathematics and Modeling Departement of the Donbass State Engineering Academy (Table 1). Table 1 shows the records of the first 10 test participants. A total of 50 participants took part in the testing.
The test in this discipline consisted of 20 questions. First, it is necessary to calculate the proportions of correct and incorrect answers of participants. These values are calculated
= , = 1 − , (9) where is the number of correct answers for the -th test item, = 1, 2, ..., , and is the number of items in the test.
For example, for the first participant of testing we have
Using the statistical module Moodle, the following characteristics were obtained for test tasks: facility index(F), standard deviation (SD), random guess score (RGS), intended weight, efective weight, distinction, distinction eficiency. These data are shown in table 2. -11.54% 6.93% 44.07% 22.91% 11.72% -1.53% 43.38% 26.08% 17.26% 68.31% 14.64% 27.00% 23.81% 17.26% 23.81% -2.60% 45.46% 13.80% 21.10% -28.62% 11.28% 65.85% 39.11% 23.66% -2.22% 70.79% 35.44% 23.76% 84.84% 35.69%
− 1
Next, we calculate the angular coeficients [
calculate the scaled values and .
In tables 4 and 5 scaled parameter values are provided.
= 1 +
= 1.104 = 1 +
= 1.63 = − 2.103 + 1.104 0 = 1.86 + 1.63 0 √︂ √︂ 2 The sum of the scaled dificulty levels of test items is -27.93.
This means that the test items are very easy. This test is not balanced, it contains a lot of easy items. It is necessary to strive to ensure that this amount is close to zero. Thus, the assessment of latent parameters allows to determine noninformative items that should be excluded from the quiz. The use of the developed adaptive algorithm will allow to objectively assess the level of students’ knowledge.
The graph of the information function of test items and the test as a whole, defined by formulas (6) and (7), is shown in figure 3.
Figure 3 shows that the information function has one clearly expressed maximum. This is a
0 ability level 2 4 6 8 sign of a “good” test. However, it can be seen that the test contains a lot of easy test items with dificulties in the interval (-3; -2), which can be excluded from the test. Also in the test there are many easy tasks with the same dificulties, which can also be excluded from the test without violation of its information content. However, the more dificult tasks (with dificulties of 1-2 logits) are clearly not enough in the test, so it is necessary to add more complex tasks.
The graph of the measurement error, depending on the level of training, is shown in figure 4.
It can be seen from the graph that the measurement error is large for the values of the ability in the interval (2,4), which is associated with the lack of test items of increased dificulty.
The purpose of this paper was to automate the process of testing students’ knowledge, which is especially relevant for distance learning. To achieve this goal, an adaptive testing algorithm based on the Rush model was proposed and the modeling of the students’ knowledge assessment process using this algorithm was carried out. The results of testing their knowledge in the course “Higher Mathematics” obtained in the Moodle system were taken as the initial values of the tasks complexity and the levels of the students’ ability.
0 ability level 2 4
As a result of modeling, the levels of students’ abilities were recalculated, the information functions of the test tasks and the entire test as a whole were built, the standard measurement error was calculated, depending on the student’s ability level. The analysis of the obtained results allows us to conclude that the test is not balanced, contains too many easy tasks. In this case, these are tasks with numbers 1, 3, 11. Removing them from the test will reduce the number of test items and speed up the process of determining the student’s level of training. A change in the assessment of the student’s ability level as a result of testing indicates the need to introduce an adaptive testing system into the educational process, which will improve the quality of assessment of student knowledge.
These conclusions are confirmed by the works of other authors. So, Al-A’ali [
As a result of this work, the following results were obtained: 1. An algorithm of adaptive knowledge assessment based on the IRT approaches was proposed. This algorithm consists of an initial assessment of the dificulty level of test items and students’ abilities, scaling of these parameters, selection of the next question based on minimizing the module of their diference and estimation of the measuring error of the knowledge level by the information function of the proposed question. 2. The test parameters were evaluated on the basis of IRT theory, which identified noninformative test questions that should be excluded from the set of test items.
The results of the study showed the efectiveness of using IRT to assess knowledge.