Nowadays among the problems of distance education the problem of the automatization of knowledge testing occupies a special place. The issues of computer testing are of great interest for university teachers and software developers of such testing systems. Meanwhile, the issues of computer knowledge testing is not completely covered in theory, and the interest in them is often realized by creating an ordinary computer-testing program with a pre-determined set of control tasks. Adaptive testing tools are not sufficiently covered and processed.
instructor;
2021 Copyright for this paper by its authors. distance learning form quickly, implementation of which is envisaged by the National Informatization Program in Ukraine.
The controlling function [
The educational function [
The essence of the diagnostic function of the control [
The prognostic function of the revision serves to obtain early information about the educational process. Result of the knowledge testing are the grounds for the prediction about the course of a certain part of the educational process: whether the specific knowledge, skills and abilities are formed enough to master the next portion of the educational material (section, topics). The results of the prediction is used to create a model of further behavior of the student, detect the mistakes of this type or that he has certain problems in the system of cognitive activity. The prediction helps to make good conclusions for further planning.
Adaptive testing means a computerized system of scientifically based review and evaluation of learning outcomes, which has high efficiency due to the optimization of procedures of generating, presenting, and evaluating the results of adaptive tests.
The efficiency of control procedures and evaluation is improved by using a short-term strategy of task selection and presentation based on algorithms with full context dependence, in which the first step is performed only after the results of the previous step (or steps) have been evaluated. After a test taker completes a task, there is once a need to decide on the level of difficulty of the next assignment depending on whether the preceding answer was good or bad.
The sets of test tasks that are developed must be adequate in terms of the subject area, and the procedure of adaptive testing (including developed tests, test algorithms, knowledge evaluation algorithms). They must be reliably controlled in the process of development and tested as a product.
Thus, the selection of such a testing algorithm, test structures that would meet the requirements and characteristics is not a trivial task and is further defined as a research task.
Linear programming models for constructing systems of partial testing are based on the criterion of maximum information, which allows taking into account various limitations for different test structures. The possibility of including such limitations in the systems of partial adaptive testing has led to attention to this method. One of the types of the step-by-step approach is parametric testing.
When determining the route of the student (the trajectory of answers to the given questions) it can be noted that if the change in the order of presentation of the test tasks is carried out at each cross section of the test (constant adaptation), then the complexity of the questions that are presented will not increase. If the decision to change the order of test tasks is made after analyzing the results of the test on a special block of tasks (block adaptation), the scheme of passing a set of test tasks sometimes reflects the constant complication of test questions.
Linear programming methods introduce constraints through the task selection algorithm. The latter two approaches put all the constraints directly into a group with a number of tests from which the test process itself is then controlled. The difference between the methods leads to the following aspects: • to the level of adaptation that is possible during the test; • to expand the description of the task; • to the possibility of an expert rating about the real meaning of the tests; • to the nature of the realization of the limits, which are controlled by the testing process; • to the possibility of violation of these restrictions.
Methods provide the renewal of knowledge assessment after each test task, and in this way, they assume the maximum level of adaptation. However, in order for the test procedure to be successful, both methods take into account all related to the task conditional attributes. If a potentially important attribute is missed, the content of the test may become imbalanced. In addition, only weighted sums of deviations from constraint are minimized, therefore, some of these constraints can be lost even when the encoding of test task attributes is completely fulfilled.
In the sequential approaches, the selection of tasks and the implementation of constraints are related. Although the sequential selection of test tasks allows optimal adaptation, the sequential implementation of constraints is not ideal. Algorithms with such peculiarities have a tendency to select tasks with the greatest number of links to other topics at the beginning of the test. However, the choice of these questions may not be optimal for further testing. In this case it will lead to the fact that the result of the knowledge evaluation will be less adequate than the optimal adaptive test or/and to the impossibility of completing the test without violating restrictions.
The algorithm for selecting and presenting assignments is based on the principle of feedback, in which a correct response by the subject of instruction causes the longer assignment to be chosen as the more important one, and a wrong answer causes the next easier assignment to be given rather than the one to which the student gave the wrong answer. It is also possible to assign additional questions on topics that the student does not know well in order to assess the level of knowledge in these areas more accurately. Thus, it can be said that the adaptive model is like a teacher on the exam - if the student answers the given questions confidently and correctly, the teacher quickly gives them a positive grade. If the student starts giving wrong answers, then the teacher asks additional question (or questions) of the same level of difficulty or on the same topic. And finally, if the student answers poorly from the very beginning, the teacher also gives a very quick but negative grade.
Adaptive testing is defined as "a set of processes of generation, presentation and evaluation of the results of adaptive tests, which provides increase of efficiency of measurements in comparison to traditional testing due to optimization of selection of characteristics of tasks, their quantity, sequence and speed of presentation in relation to peculiarities of student training".
With adaptive testing in the process of passing a test (or a set of tests) a model of the student is created, which is used to generate or select the next test tasks depending on the level of training of the student. In complex systems, the obtained model can also be used in the learning process. Nowadays, adaptive testing is implemented mainly in the form of algorithms of computer testing.
Adaptive testing must meet the following requirements: • the ability to regulate the proportions of easy, medium, and difficult assignments depending on the number of correct answers of the student; • the ability to regulate the proportions of the suggested different topics of the educational program in the test; • the possibility of regulating the levels of complexity of the tests taken, considering the semantic competence of the test taker; • activation of the adaptive mechanism of transferring to a higher level of tasks at the same level of the proponed tasks; • each higher-level assignment is evaluated by higher scores.
The choice of test algorithms nowadays is actually limited to the forms of presentation of test tasks and algorithms for evaluation of test results. Achieving better results and increasing learning motivation in the outcome is the main goal of any test. The test process is conditioned by the test algorithm and should be as formalized as possible and, at the same time, flexible in order to obtain adequate assessments of the knowledge of the test takers. In addition, the ability to regulate the proportions of the easy, intermediate, and major tests depending on the number of correct answers is a non-trivial consideration. This is due to the fact that, as a residual result, statistical methods for valued approximation of Rasch's success function are used when evaluating learning subject's abilities.
The sets of test tasks that are developed must be adequate to the subject area, and the procedure of adaptive testing (including developed tests, test algorithms, knowledge evaluation algorithms) must be reliably controlled in the process of development and tested as a product.
Thus, the selection of such a testing algorithm and test structures that would meet the requirements and characteristics is not a trivial task and is further defined as a research task.
Evaluation of the existing methods of computer adaptive testing leads to a very important dilemma. An algorithm with optimal properties would have to choose the task in such a sequence that allowed achieving optimal adaptation and simultaneously taking into account all limitations in order to prevent violations of some of them or not to achieve suboptimal adaptation during further testing.
The possible solutions to this dilemma are the following: • implementation of the algorithm with the ability to go back in order to improve the next decisions; • implementation of a march algorithm that would take into consideration the consequences of decisions made in the future. During adaptive testing, it is not possible to make a random order, the algorithm is applied on a real-time scale, and the initial selection cannot be different. Thus, there is only one possibility - to use an algorithm that selects a new task one time. This is an exceptionally new class of algorithms.
It is necessary to provide a scenario of the system in order to determine the basic algorithm. The scenario is based on the paradigm of checking the exam by a teacher as a model of adaptive testing. The choice of the scenario for the system is based on the fact that, firstly, this procedure is historically formed long time ago and is well formalized; secondly, while designing tests their developers need to rely on generally accepted, known and used methods with minimal modification. The adaptive testing algorithm must be accompanied at every step (while moving from one task to another) and be foremost informative (provide maximum information about the student's answers to each question that was asked). At the same time, the subjective qualities of the student, which can be expressed as a lack of comprehension of the obvious question or task, should not be neglected.
Under certain conditions, additional questions are an integral part of the tests, but we should also take into account the next points: • not all additional questions are identical both in terms of complexity and in terms of full compliance with the main question; • chains of additional questions represent logically connected sequences; • questions (additional along with main ones) are asked sequentially, meaning it is impossible to ask two or more questions at the same time; • the frequency of additional questions may vary.
The representation in the form of logical networks and mathematical models of first-order finite predicate calculus was chosen as a basic model of the system of adaptive testing. It has the following qualities: • the main unit is a question of a certain complexity, which may have a sequence of additional questions; • the choice of an additional question is determined by the probability of an additional question appearance as a stream of elementary events with a logical conclusion of choosing a wrong answer; • the system of tests is enclosed in the sense of logical networks, which means that if the question (primary or secondary) is a state si, which has a predicate of the test task performance ( ), then the logical sum of the test answer predicates and the predicates of the system in a state is identically defined as:
=∨ ( ) = 1
The specified state, which represents the test task in the form of the process of student knowledge and skill evaluation, is reached in case of getting the right answer to the one of those questions and/or additional question (or their sequence). The removal of one of the questions should not lead to the state being assigned to a zero value.
The last requirement lets endlessly traverse the questions, so the termination of the test is possible in the following cases: • all questions in the bank of tasks are used; • the end of the test is reached; • the level of knowledge is assessed with sufficient accuracy; • the level of student knowledge considered insufficient to achieve the criterion of passing the test; • student demonstrates his/her inability to pass the test.
Representation of the testing algorithm in the form of a predicate description of logical networks is not exhaustive. As it was noted before, no matter whether the answer was right or not one of the following decisions will be taken: • transition to the next main question with possibility to choose it’s complexity; • transition to the next additional questions (to their trees), in addition, it is necessary to discard already asked additional questions (individual ones along with their trees); • return to the main question if the answer to an additional question (or questions) is received; • termination of the test.
The test task as an object of the aforementioned testing process supposes that student follows the rules set by the examiner. This corresponds to the traditional process of taking an exam, so depending on the answer an examiner takes above-mentioned decisions. Doing so, he takes into account fragmented answers (score for each question and the average score), as well as the whole chain of student’s answers according to certain logical rules. In addition to the general rules, the main approach to the accepted methods is a composition of tasks from different parts, (pictures, tables, multimedia content), given as stimuli. It allows us to save resources of stimulus allocation and select the answerprocessing program (general rules in different parts of the created logical network).
This approach does not allow us to individualize the test questions sufficiently – first of all, it is connected with the fact, that each question combines both the direct task and the decision, which is connected to the student’s solution of this task with the answers to additional questions.
Taking into account all the aforementioned, in order to provide flexibility in decision making, simplicity in question creation and logical rules, which defines decisions to a specific question, it is appropriate to combine the question and the decision-making procedure related to it. Such an approach simplifies the testing procedure as well as the testing system itself in terms of meeting the requirements of minimizing the complexity of the applied algorithms.
Making common decisions for the whole testing process, requires common approaches during the one testing session. These approaches are defined by: • the application of the general method (or methods), which determines the step in the testing process, when the additional information regarding the knowledge of a student will be redundant; • the procedure of starting the testing system (the choice of the first question) and the strategy of moving from one question to another; • provision of the detailed test results, both in a natural form and in the processed one with the help of one method or another.
It requires the usage of the testing and application protocols: • algorithms of logical operational and statistical analysis of the test results in terms of redundancy or lack of information; • algorithms that determine the student’s level of preparation; • algorithms that provide the stochastic transitions within a network of test tasks. • In fact, the student model in a particular session is defined by: • the protocol of the examination; • knowledge evaluation results.
Thus, this approach forms the testing paradigm that is natural for the teacher, has an analogue in the classical sense of the exam and can be defined as a model of a student.
We propose to perform the procedure of providing the additional questions in two ways. The first method is that no action is taken before the set of additional questions connected with the main one (or a single question) is finished, except for recording the answers to the questions and subsequent transfer of the protocol to the main question, where one of the following decisions is taken: • consider the answer to the main question as the correct one with the possible adjustment of the complexity; • consider the answer unsatisfactory and move to a lower-level group of questions; • ask the additional question once again with the inclusion of the remaining additional questions; • ask the main question again without involving additional questions.
The second method: the decision-making functions on the correctness of the answer and/or the transition to the other main, additional question or to the end of the test is transferred to the additional question.
The latter is a general form, i.e. the pseudo-interactive procedure can be reduced to the marching one by removing the logical analysis and decision-making (logical transition in the theory of logical networks).
The presence of logical analysis and decision making in a test task models the behavioral rule of the examiner. Adopted situational approach is based on the rules for a specific behavioral question (transfer of control to another question) within a single task. This allows to take into account not only the elementary set of the answer results (true or false), but also which answer from the set of incorrect answers is chosen considering the state of the examination protocol before interaction with a specific question.
Additional questions can be selected from the main ones with any level of complexity and/or created separately (for a specific test). The level of difficulty must be assigned for them as well as for the main ones. Thus, it is possible to summarize the questions in a set of test tasks as a structure (class) of data and determine its behavior. The questions should have: • the property that determines whether the question is main or additional; • the property that defines the level of complexity; • the property that determines the affiliation and relation with another topics and issues; • the property that defines the questions, with which the connection was terminated in the process of testing; • the property that determines whether the wrong answer is able to interrupt the process of testing; • the property that defines connections (those questions, which can be switched to), due to inference; • the property that determines connections (those questions, which can be switched to in a usual for logical networks way); • the property of «stopping» the question in a logical networks, i.e. the case when the question has already been asked; • the property that defines whether the question is unique for a given network; • the property that determines which other question (or questions) can be triggered by the current one, i.e. which question can receive the control.
The following provisions are required: • ensure the acceptance of control from the previous question with obtaining the current protocol of the examination; • ensure that the test assignment and the answer options are provided; • ensure that the countdown is started at the moment of issuance of the assignment; • maintain the logical function that determines the reduction of the level of complexity in case of exceeding the limits of time allowed for answering the given question and/or making a decision (under certain conditions) for the current question on whether the received answer is correct/incorrect; • approve a decision on the continuation of the test or its termination; • make a decision and choose the next question according to the given stochastic algorithm or deduce if the “stop” property for this item is active; • transfer the examination protocol to the next question and to the network controller; • transfer the properties which determine the state of the question to the network controller; • check the operation of the question (the control is expected to be transferred to the network controller or to the next question).
Figure 1 shows the fragment of the logical network of the questions (main question along with additional ones).
The usage of logical networks based on the first-order finite predicate calculus and predicate operations assumes the existence of a system model as an abstraction which has a set of states = { 1, . . . }. These states are mutually exclusive and the transition from one state of the network to another are carried out in different ways, due to the occurrence of an event, which in general case represents a predicate of the event. Each state of the system si must have at least on input and one output, so that so called “no-goes” or end states are absent. In fact, any state of the system can be chosen as an initial one.
However, in practice the choice of the initial state is determined by a priory information about the model of a student. After the initialization of a certain initial state, the operation of the model, created within a specified framework of an algebraic system of the logical network, is based on the infinite traversal of the state network of the system, in which it remains for a certain period.
We should note that the direct use of an application of the logical network representation does not provide the solution to the given problem, because there are a number of logical limitations, which define the transition from one state to another and state removal as well as the limitation of transitions (the test cannot be carried out continuously and the questions must not be repeated).
The latter method is better, as it is checked whether the set of the given characteristic is formed (such as complexity of questions for the whole group of students and whether the set of predicates for answer distribution is formed depending on their complexity for a particular student). Furthermore, the growth of the testing algorithm complexity will not lead to a significant growth of resource consumption.
The difference between the levels of complexity of the main and additional questions and the proposed relation between the main questions and the branches of additional ones lets minimize an amount of student answers needed to determine the level of their knowledge as well as significantly improve the adaptive qualities of the test.
The best results are achieved when the difference between the predicted and the actual level of knowledge of the group of students and the difference between the predicted and the actual complexity of tasks is significant.
However, with a small difference between aforementioned predicted and actual characteristics of the tested group the application of the proposed methods gives a positive effect, from which we can conclude that the application of these methods of logical network creation for the implementation of the knowledge testing stage in the adaptive testing systems is effective for any capacity and other characteristics, which define testing as a process.
To form a purposeful program to solve a complex intellectual problem, the IP must adequately respond (perceive) the behavior or state of the phenomenon, the object of the real world in which it operates and respond quickly to any changes within it. To this end, it is necessary to create a model of IP, which should implement algorithms to reproduce the "mental" transformations of the information environment and build on this basis a plan for solving the next problem, as well as algorithms for solving this problem and control comparison of expected and actual results of planned actions.
We can talk about the expert model and the theory of this model. The carrier of this model is a set of manifestations of the received signal in the field of perception of the expert, for example, in the form of the image of a picture of signals or a signal image. Model predicates are predicates of these manifestations, for example, when recognizing images of a spectral picture – predicates of image description. The properties of these predicates, the general form, the axioms of the theory, etc. are introduced.
As the results of research have shown, the mathematical means of predicate algebra allow to solve the above-mentioned research problems: problems of mathematical description of expert intelligence and problems of formal description of artificial (machine) intelligence or IS of automatic image processing of signal images and signal images.
Suppose that the studied picture of the signal implements a predicate ( 1, 2, . . . , ), and to obtain an image of the picture, the researcher offers sets of signals from the set , which are selected by him depending on the objectives of the study. The information converter itself may have restrictions on the class of input signals.
An operation ′ = ( ) = ∧ that maps a set into itself is called resetting the predicate by . The zero operation on assigns values to the predicate Р within the domain the same as those of the predicate, and outside it replaces them with zeros.
In another words:
( 1, 2, . . . , ), if( 1, 2, . . . , ) ∈ M, ′( 1, 2, . . . , ) = {
0, if( 1, 2, . . . , ) ∉ M.
The operation of zeroing in brings certainty to the predicate – those of its values that are unknown outside the region , it replaces with zeros, and within the region the values are experimentally in the process of experiments on the subject. Thus, the predicate is distorted, but becomes quite definite in the whole subject space . In fact, the values of the predicate realized by the subject outside the area are unknown, but objectively they exist.
The main task of the theory of intelligence is the construction of models, the formation of their properties, the construction of model theory. With the help of mathematical means of the theory of intelligence, models of various mechanisms of intelligence are built, such as the model of color vision and the theory of this model. The carrier of this model is a set of light radiation, and the predicate of the model is a color predicate. The properties of the color predicate, its general form, axioms of the theory, etc. are introduced.
Consider another example: a theory for a model of a natural series of numbers ⟨ × N, Q⟩. is the set of natural numbers, ( , ) – predicate of the account + 1 = . and are variable predicates, which are defined by axioms as logical equations.
Another example is the theory for the model of belonging of an element to a set ⟨ × B, P⟩, where is the set of elements, is the set of names of subsets of the set , ( , ) is the predicate of belonging of the element to the set :
(x,y) = 1 ⇔ ∈ Several other models of intelligence theory are also known.
Operations on models are used in cases when it is necessary to combine particular results of intelligence modeling. Models are associated with conditions (model properties). The activity of one or many subjects is modeled. It is often necessary to move from general to particular models, and in some cases the simulation results are combined into a broader result.
For example, if models ⟨ 1, 1⟩ and ⟨ 2, 2⟩ are constructed that characterize the behavior of the subject in the same task, but for different sets of input signals 1 and 2, then there is a need to combine them into one broader model. In this case, the properties of the model are expressed using seconddegree predicates. A more general case is possible, when it is necessary to combine into one model two different models ⟨ 1, 1⟩ and ⟨ 2, 2⟩ characterizing the behavior of the subject in the areas 1 and 2 and for different tasks 1 and 2. The standard model is equivalent to a predicate on a domain. In case you want to get a model that characterizes all that is common in the models and, or to decompose the model into several simpler models, you may need other operations on the models.
Models ⟨ 1, 1⟩ and ⟨ 2, 2⟩ are called compatible if any set of items ( 1, 2, . . . , ) belonging to the set 1| 2 is true for 1( 1, 2, . . . , ) = 2( 1, 2, . . . , ).
If the tests are performed with the same task 1 = 2, but in different areas 1 and 2, then the models are automatically compatible. Why do we need predicate values outside its scope? For example, to minimize the database.
Let 1 and 2 be compatible models. A model = ⟨M, P⟩ is called a union of models 1 = ⟨ 1, P1⟩ and 2 = ⟨ 2, P2⟩ and = 1 ∪ 2 is written if = 1 ∪ 2 and = 1 ∨ 2.
The model = ⟨M, P⟩ is called the intersection of the models 1 = ⟨ 1, P1⟩ and 2 = ⟨ 2, P2⟩ and = 1 ∩ 2 is written if = 1 ∩ 2 and = 1 ∧ 2.
The relations connecting the models are expressed by second-degree predicates, which are written in the form of closed quantifier expressions [8].
If you narrow the set of input signals or expand the same task, you will turn on the models. If 1 ⊆ 2, then the model 1 is called a submodel of the model 2, and the model 2 is called a supermodel of the model 1.
The description of the actions of the expert in the perception and analysis of signals and images is reduced to a mathematical description of the processes of converting input information into output formalized information in the form of distinguishing features (or properties) to determine the types of objects. Such a mathematical description of the processes of the expert's activity is called identification.
Thus, it is necessary to obtain a mathematical description of the function ( ) = that maps the set to the set and characterizes the course of the operator. The function is called the characteristic function of the operator (or the process of assigning marks to a certain type). 5. Conclusion 1. Such systems allow to release the teacher from routine work during examinations and intermediate assessment of knowledge in the traditional educational process, and when learning with the use of distance technologies become the main means of control. 2. Provide the ability to automate the processing of results, the objectivity of control and the speed of checking the quality of preparation of many tested on a wide range of issues. This allows you to identify the sections that are most difficult to learn, and possibly adjust the learning process depending on the test results. 3. Provide an opportunity to implement the learning function. 4. Allow individualization of the process of learning by students. 5. The functions of knowledge control are not only controlling, but also educational and developmental. 6. Forms of test tasks can be presented in the following forms: • Closed form. • Open form. • For compliance.
• To establish the correct sequence. 7. The analysis of the existing KST showed that in most cases they are focused on conducting tests, rather than on their development. In the implementation of testing, none of the considered KST does not support adaptive methods of testing, poorly developed polytonic assessment of test tasks. 8. Testing is an important element not only in the control of knowledge, but also in learning. During the training test, the user after the test is provided with links to those sections of the training material, the questions on which he answered incorrectly. 9. Considering computer control systems of knowledge from the point of view of their practical implementation, it is necessary to note that all of them contain the following components (sometimes not allocated explicitly in structure of system).
• Test preparation subsystem. • Testing subsystem.
• Subsystem for analysis of test results. 10. The inclusion of modern information technologies in the educational process creates real opportunities to improve the quality of education. However, it should be recognized that the level of informatization of educational and scientific activities is still quite low, the legal framework for distance learning is insufficiently developed.