An approach to assessing the level of assimilation of educational material by education seekers is proposed. This approach involves using adaptive Item Response Theory (IRT) models to evaluate knowledge based on the one-parameter Rasch model, design a combined testing procedure, and verify the balance of test tasks. This enables the determination of the objective level of preparedness of the education seeker according to the testing procedure and the assessment of the quality of the preparation for test tasks.
Modern higher education institutions in developed European countries are actively working to improve the education system [1], which can integrate into the European educational and scientific space [2]. One of the driving mechanisms for such improvement is ensuring fair competition in the educational services market with an appropriate quality of higher education.
Training high-quality specialists should involve not only modern methods and means of acquiring new knowledge, but also an objective assessment of their level of assimilation [3]. The effectiveness of the evaluation depends on compliance with the didactic principles of systematicity and objectivity, as emphasized in the work [4, 5].
Currently, a relatively wide range of research assesses the knowledge of education seekers. Their analysis has shown that the results of knowledge control of existing systems are not sufficiently informative and objective. An important parallel direction of improvement is the control of data errors to enhance the reliability of processing results, a methodology explored in detail in related works [6]. Testing procedures must use test tasks that are uniformly selected in terms of complexity to ensure the maximum informativeness and objectivity of the control results. This can be done using adaptive testing.
Adaptive testing (AT) is a type of test assessment that evaluates knowledge, where the sequence of test task presentation (including their complexity) and the number of functions depend on the student's answers to previous test tasks [7, 8]. The use of an adaptive approach enables you to consider the individual capabilities of students, assess their cognitive skills, increase the accuracy of determining their knowledge level, utilize both rating and interval scales for evaluating test results, and enhance student motivation to study [9].
However, the adaptive approach has the following limitations: the presence of linear processes in testing procedures, the absence of feedback; the lack of ranking of test participants who gave the correct answer to the same number of test tasks, which affects the objectivity of the assessment [10]; the imbalance of test tasks in terms of content and complexity, which affects their quality [11].
Therefore, to ensure proper informativeness and objectivity of the control results, it is proposed to assess the level of assimilation of educational material by education seekers based on adaptive Item Response Theory (IRT) knowledge assessment models, with the possibility of ranking test participants who gave the correct answer to the same number of test tasks, which are balanced in content and complexity.
In the approach to assessing the level of assimilation of educational material by education seekers and ensuring proper objectivity, it is proposed to use adaptive IRT models of knowledge assessment with existing procedures: organizing test assessment of knowledge; determining the conditions for the need to change the level of complexity of the test task; determining the balance of test tasks in the test (test quality).
The adaptive testing model involves: a testing organization procedure (determining the initial level of knowledge of the education seeker, determining the set(s) of test tasks and their level of complexity, determining the estimated weight of the correct and incorrect answer to the test task, establishing the form of obtaining the final results for testing); a procedure for conducting testing (establishing the level of complexity of the task from which testing will begin, determining the method for assessing the correctness of the answer to the current test task, determining the rules (conditions) regarding the order of changing test tasks according to the procedure and its completion to obtain the final result); a method for determining the balance of test tasks in the test (test quality).
The organization and implementation of knowledge testing should be examined through the lens of the following classification groups of adaptive testing methods: 1. Pyramidal Testing is an adaptive form of testing where the complexity of the test task changes according to the correctness of the answer to the previous one. At the beginning of the testing procedure, the starting level of complexity of the test task is set. The rule of dividing the complexity scale of test tasks in half at each stage is applied.
of teachers. Still, the formula 4n is typically used, where n represents the number of test tasks for each student.
The drawbacks of this testing procedure include the significant resources required for its development and the necessity of independently assessing item complexity, which can potentially compromise the objectivity of the results obtained. 2. FlexiLevel is a type of adaptive testing organization and implementation aimed at comparing each test participant's qualification level with the degree of complexity of the test tasks given to them. The testing procedure begins with a level of complexity that is arbitrary. After that, depending on the test participant's answer, he is given a task of a higher or lower level of complexity. There is a gradual approach to the real level of preparedness of the test participant.
the difficulty level of test items.
To determine the complexity of test tasks, initial statistical data is collected based on fixed-form testing procedures. The next step involves determining the test item's complexity parameter by calculation according to the formula: measurement of the student's abilities. more difficult tests by difficulty level. where: – the count of tested participants who successfully completed this test item; – the count of tested participants who answered this test item.
all difficulty levels. FlexiLevel testing often focuses on assessing individual aspects of knowledge rather than their integrated application. This can result in students who are good at the material but have gaps in individual topics receiving a lower score. 3.
= 1 − ,
assess the starting abilities of the education seeker. Their number is variable and depends on the purpose of the test. After the initial knowledge of the education seeker is determined, the assessment procedure switches to adaptive mode, and test tasks are provided according to the selected adaptive algorithm.
The advantages of this procedure in organizing and conducting testing include factors such as the accuracy of assessment, time efficiency, task balance, flexibility in test design, and participant psychological comfort. The disadvantages include the complexity of development and implementation. Stratified testing operates within specific strata, which may limit flexibility in adaptation. If a participant makes mistakes in the initial stages of testing, they may be moved to a lower tier, where the tasks will become too easy for them to complete. This may lead to an underestimation of their real knowledge.
Therefore, each of the above procedures has certain peculiarities in its application, which determine its advantages and limitations. This necessitates the use of a combined approach in the organization and conduct of testing in the proposed model.
According to the procedure for organizing testing, the initial level of preparedness of the applicant for education is average. It is proposed that three sets of homogeneously selected test tasks be formed, differing in complexity. The estimated weight of the correct answer to the test task is 1, and the incorrect one is 0. The method of obtaining final results involves calculating the ratio of correctly answered questions to the total number of test tasks, taking into account their complexity.
According to the testing procedure, the level of complexity of the task from which the testing will begin is set to medium, which is typical of the pyramidal approach. When answering the test task, the student launches an adaptive branching algorithm by processing test tasks of different levels of complexity. If the student answers the test task correctly, the complexity level of the next test increases; if incorrectly, it decreases (flexible procedure).
The processing of the condition for the need to complicate (ease) the complexity of the test task, depending on the current answer of the education seeker, will be entrusted to the appropriate procedure that assesses the probability of the education seeker's correct answer to the current test task.
Currently, Rasch and Birnbaum's logistic models [12] are widely used to determine such a probability [13].
The one-parameter Rasch model (1PL) utilizes two values: θ – the knowledge level of the test taker, and δ – the task's complexity level, represented as ; , ; .
The one-parameter Rasch model (1PL) involves the introduction of two main parameters: θ, representing the test participant’s knowledge level, and δ, representing the task complexity level [14]. This is the probability that a test participant with a level of preparedness will correctly complete a test task of complexity . The likelihood of success depends, in fact, only on one parameter – the difference , and is calculated by the following formula: where: = ++11 , p i – the proportion of correct answers to test tasks, q i – the proportion of incorrect answers to test tasks; = 1 + ++11 ; i = 1, 2, … , m – number of testing participants; j = 1, 2, … , n – number of test tasks.
lim = 1, lim = 0 (
P = 0.5 if .
The one-parameter Rasch model is most effective for analyzing dichotomous tasks, where the answer can only be binary: "correct" or "incorrect". This model considers only one parameter – task difficulty (δ), which makes it ideal for tasks where additional characteristics such as guessing level or discrimination need not be considered. The model assumes that all items have the same level of discrimination (i.e., all items are equally good at distinguishing between strong and weak test takers). The one-parameter Rasch model is well-suited for tests with a limited number of items, as it requires fewer parameters to be estimated. This model is best suited for open-ended test items where there is no way to guess the correct answer.
In the two-parameter Rasch model (2PL), the adaptive test consists of n test items. In the model, the response variables Uj, j = 1, 2, …, n, can take the value one if the answer is correct, and zero if it is incorrect, that is: = 0, if the answer is incorrect 1, if the answer is correct
= 1| = 1+ ( (−− ) ). (
Discrimination helps to select tasks that best distinguish between different levels of knowledge or skills. It allows you to identify tasks that effectively distinguish between test takers with high and low levels of expertise. The model is suitable for organizing and presenting test tasks where the test taker must choose one correct answer from several proposed ones.
If the test contains open-ended questions or questions that do not have clearly defined right/wrong answers, the 2PL model may be less suitable. For tasks where it is essential to consider additional factors (for example, individual characteristics of test participants), more complex models may be required (for example, a three-parameter model).
By Birnbaum's logistics model (3PL), the probability of the accurate response = 1| on j questions with knowledge level θ is calculated by the formula: 1.7 − = 1| = + (1 − ) 1 + 1.7 −
(
This model takes into account three key parameters for each test item: difficulty (helps to select items that match the participant's current level), discrimination ability (allows you to choose items that best distinguish between different levels of knowledge), and guess probability (taken into account when scoring, especially when the items are multiple choice).
The 3PL model involves calibrating tasks before testing to ensure assessment accuracy, even with a fixed test format. The model is most suitable for tests where participants select one or more correct answers from a set of provided options.
A limitation of the 3PL model is that it is difficult to calibrate (for accurate application of the model, a large amount of data is required to estimate task parameters), is sensitive to task quality (if tasks have low discriminatory power or inadequate complexity, the model may produce inaccurate results), and has limitations for test tasks with partial scores (scoring on score scales).
In the proposed adaptive testing model, the single-parameter Rasch model will be used to evaluate the correct answer to a test task and determine the condition for changing the test task's complexity level.
This is because it is most suitable for tests with dichotomous tasks, equal task discrimination, a small number of functions, and a low level of guessing.
Probability of a correct answer Pj(θ) obtained during testing. If Pj(θ)≥0.5, then the task's difficulty level must be increased; otherwise, it must be decreased.
To assess the quality of selected test tasks, which should reflect the structural hierarchy of the training model in the academic discipline, it is necessary to address the problem of test composition, specifically to evaluate the internal consistency (balance) of tasks within the test.
In this case, the expert needs to assess the effectiveness of the scheme proposed by the developer and the method used to arrange tasks in the test. The concept of balance involves the proportional filling of the test with functions of varying levels of complexity.
Thus, the analysis of the test composition reveals the degree of harmonious presentation of key elements of the academic discipline's content, the adequacy of their reflection in the test, and the appropriate level of differential difficulty of the test tasks.
Currently, the following approaches and methods are most common in the field of assessing the quality of test tasks: 1. Methods based on the criteria of mathematical and statistical methods of analyzing test task statistical characteristics (properties). They enable the detection of hidden defects in test tasks. Models of dispersion, factor, cluster, discriminant analysis, and time series regression analysis are used. 2. Methods that use expert assessment of the consistency of tasks in the test. The procedure for specialist assessment of test quality involves the following stages: semantic capacity evaluation, prediction of test task success, and comprehensive examination of test quality and task performance, which includes approbation testing. The goal is to establish, verify, and evaluate its measuring capabilities by approving representative samples. 3. Methods based on analyzing the degree of balance of test items. The calculated correlation coefficient (balance) is used as a numerical indicator of the validity of the test.
Among the methods of the third group, the Chelyshkova method of checking the balance of test weight coefficients attracts special attention [15]. This method has proven very effective in identifying extreme test tasks, that is, those that are too difficult or too easy. For this, the balance of their weight coefficients is checked. This check is carried out to analyze the complexity of test tasks and inform the expert about the feasibility of replacing extreme questions in the test, which negatively affect the objectivity of assessing the student's knowledge [16].
Thus, preference is given to a combined method of organizing adaptive testing [17]. The singleparameter Rasch model will be used to determine the condition for changing the complexity of the 166 test task. Experimentally selected coefficients provide the optimal value of the indicator of differential ability for test tasks, the balance of which is achieved using the method of M.
Thus, preference is given to a combined method of organizing adaptive testing. The singleparameter Rasch model will be used to determine the condition for changing the complexity of the test task. Experimentally selected coefficients provide the optimal value of the indicator of differential ability for test tasks, the balance of which is achieved using the method of M.
1. Initialization of the testing procedure. The task is randomly selected from a set of functions corresponding to the specified difficulty level (“easy”, “medium”, or “difficult”). The first task is always of average difficulty. Depending on the answer to it, the next task will be proposed by the adaptive algorithm as easier or more difficult than the previous one:
∈ ⃗∪ ⃗∪ ⃗, i=1, . where: – test task of the appropriate level of complexity; ⃗, ⃗, ⃗– set of tasks of difficulty level “easy”, “medium”, “difficult”;
( ) ∈ {0; 1}.
= ∑ =1 ( ) .
= = 100%.
. = ∑ =1 ( )
.
= 1 − .
=
= 1 +
+1 .
+1
+1 .
+1
(
where: ( ) – the answer to the test task of the appropriate level of complexity, = 1, = 2, = 3;
A threshold value of the probability of a student's correct answer, Pj (θ) = 0.5, serves as a condition for changing the task's difficulty level during the testing process. The task's difficulty level increases when Pj (θ) ≥ 0.5 and decreases otherwise.
vice versa. The essence of this operation is to ensure the correspondence between the task's difficulty level and the student's knowledge level, thereby enhancing assessment objectivity. This correspondence is particularly crucial, as the discriminant ability of the item refers to its capacity to distinguish learners based on their level of preparedness. The higher the discriminant ability, the better the division of learners into those who are prepared and those who are not.
( ) = ( − )
1+ ( − ).
in the weight of the test task, and for the case where the final level of success is determined by the condition of completing the total number of test items (Fig. 1).
A balance check of their weight coefficients is performed to find extreme test tasks, i.e., those that are too difficult or too easy. This check is performed to analyze the complexity of the functions and inform the expert about the feasibility of replacing extreme questions in the test that negatively affect the objectivity of the student's knowledge assessment.
The balance of the weight coefficients of test tasks is proposed to be checked using the
In any testing process, the results of calculating ̃ the statistical estimates and ̃ the statistical estimates will differ from the existing exact values. In this sense, the estimates are certain functions of the initial random values of the elements of the test results matrix.
This matrix is a table, with the rows corresponding to the number of education seekers (N) and the columns corresponding to the number of indicator variables (M). In the case of testing educational achievements, the indicator variables are test tasks. A number that characterizes the education seeker's response to this task is at the intersection of the rows and columns. In this work, a binary matrix is used, a matrix of test results for the dichotomous case, where the education seeker's responses are characterized by two values: 0 and 1. Zero corresponds to an incorrect answer, and units to a correct one.
number of correct responses to the total number of items answered.
Similarly, the proportion of correct answers for the j-th test item pj is calculated as the ratio of the total number of correct responses by all students to the aggregate number of answers to that item. In both scenarios, the proportion of incorrect answers qj is obtained by subtracting the proportion of correct answers from unity.
= ∑ =1 ( )
, = 1 − . = ∑ =1 ( )
, = 1 − .
Calculate the initial level of knowledge of the education seeker i0 (
(
= ++11 , = 1 + +1 +1
.
= ∑ =1 , = ∑ =1
. Calculation of variances S and S to reduce the parameters into a single scale of standard estimates, and calculation of the slope coefficients a a .
obtained from several samples will, of course, differ. If the sample size is large enough, one can calculate stable values of the parameters θ and δ, which will be the most effective estimates and can be accepted as objective estimates of θ and δ.
Thus, the question arises of finding the mathematical expectations and variances of these random variables. It is necessary that the mathematical expectation of the corresponding estimates coincide with the corresponding exact values, and the variance of the forecast is minimal.
= ∑ =1( )2−
. = 1+ /2,89 1+ /8,35, = 11++ //28,8,395 .
= + , = + .
Calculation of the balance of the test task complexity indicator (
∑ = ∑ =1 .
The determination of the final grade according to the traditional learning scheme (unsatisfactory, satisfactory, good, very good, excellent) is carried out using conversion scales based on the student's level of success.
To evaluate a user who has passed the test, it is necessary to determine their success and compare this value with those set by the developer when creating the training task. Based on this comparison, a score can be assigned according to the traditional training scheme.
To evaluate a user who has passed the test, it is necessary to determine their success and compare this value with those set by the developer when creating the training task. Based on this comparison, a score can be assigned according to the traditional training scheme.
={x1, x2, ... , xN}={1, 2, ... , 10}; Y ={y1, y2, ... , yN}={1, 2, ... , 10}; Z ={z1, z2, ... , zN}={1, 2, ... , 10}; The set of answer options q={0;1}.
N – number of test questions, N=20. _ =1, _ =2, _ =3.
The set of task complexity weights K ={k1, k2, k3}={1;2;3};
1 =
1 = ∑ =1 ( ) =2x1+2x1+3x0+2x1+2x0+1x1+1x1+2x1+2x1+3x0+2x1+2x1+ +1+1+2+2+3+0=25. 1 =
11 100% = 60
= 0.7, = 1, … , 3 1 = 2 = … 20 = 1 0.7 0.3 1 = 1 − = 1 − 0.7 = 0.3.
∑ =1 ( )
∑ =1 ( ) = = 2 3 3 3 The number of test tasks in terms of difficulty
Based on the calculations, rating indicators of the success of education seekers were determined. 1, 2, 3 Allow us to construct an objective series of precedence:
Student 2 (61.67) < Student 1 (41.67) < Student 3 (33.33) 3.
3 0.7 0.3 4 0.4 0.6 5 0 1 6 1 0 7 1 0 8 0.7 0.3 9 0.7 0.3 10 0.4 0.6 11 1 0 12 1 0 13 0.4 0.6 14 0.7 0.3 15 0.7 0.3 16 1 0 17 0.7 0.3 18 1 0 19 1 0 20 0.4 0.6
(
Calculate the initial level of knowledge of the education seeker i0 and the initial level of difficulty of tasks i0 .
For example, 1 = 1 = 1 + + 1 + 1
= + 1 + 1 1.7 1.3 = 1 +
For questions 4, 5, 10, 13, and 20, a reformulation is required to ensure that the proportion of accurate responses exceeds 0.5.
As a result of practical implementation, performance rating indicators were obtained, allowing for a sufficiently objective ranking of students on a preference scale. Test tasks requiring reformulation were identified based on task complexity balance indicators.
Using the integrated development environment Microsoft Visual Studio [18], the high-level objectoriented programming language Microsoft Visual C# [19]. The developed class library, NET, and the .NET Framework platform [20] were used to create a Windows Application that incorporates the logic for testing according to the proposed approach. Figure 2 illustrates the screen forms that demonstrate the operation of the test information system in test passing mode, as well as the process of obtaining the test result.
• • • • •
An approach to assessing the level of educational material assimilation by education seekers is proposed.
According to the procedure for organizing testing, the applicant's starting level of preparedness for education is average. It is proposed that three sets of uniformly selected test tasks be formed, differing in complexity. The estimated weight of the correct answer to the test task is 1; the incorrect one is 0. The method of obtaining final results involves calculating the ratio of correctly answered questions to the total number of test tasks, taking into account their complexity.
According to the testing procedure, the test task's complexity level, with which the testing will begin, is set to medium. When answering the test task, the student launches an adaptive branching algorithm by processing test tasks of different levels of complexity. If the student answers the test task correctly, the complexity level of the following test increases; if incorrectly, it decreases (flexible procedure).
In the proposed adaptive testing model, the single-parameter Rasch model will be used in the procedure for evaluating the correct answer to a test task and determining the condition for changing the test task's complexity level.
This is because it is most suitable for tests with dichotomous tasks, equal task discrimination, a small number of tasks, and a low level of guessing.
Probability of a correct answer Pj(θ) obtained during testing. If Pj(θ)≥0.5, then the task's difficulty level must be increased or decreased, otherwise.
Experimentally selected coefficients provide the differentiating ability (discriminative power) of test tasks, the balance of which is achieved by the method of M. Chelyshkova. This enables the determination of the student's objective level of preparedness and the assessment of the quality of preparation for test tasks.
Compared to existing counterparts, the described approach enables the objective ranking of learners based on test evaluation results, offering high clarity and moderate computational complexity.