New approaches to evaluation in testing problems are considered. The closed type questions are investigated and the substantiated formulas of the definition of testing results are offered. To formalize testing, the apparatus of decision theory and expert technologies are successfully used. The classification of questions during testing is given. A unified approach to the formalization of different types of test tasks is introduced. It is proposed to use an algebraic approach to determine the evaluation results in the testing problem. The problem statement, the algorithm for evaluating test results with multiple choice, and an example that illustrates this type of testing are given. The problem of evaluation of test results in conformity assessment tasks is considered. The problem statement, formalities for establishing conformity, variants of formulas, and an illustrative example for calculating the degree of similarity between answer variants are described. The problem statement and algorithm for solving the problem of estimating the correctness of the sequence established by the respondents are described.
Testing is ambiguous, debatable, but reliable, powerful, effective, and, for some areas of human life, an irreplaceable tool. In particular, the testing procedure is fruitfully used in programming, engineering, medicine, psychiatry, education, etc. Moreover, for example, pedagogical testing performs several functions simultaneously: educational, diagnostic, evaluative, stimulating, developing, educational, and so on.
Today, there are many opinions about the appropriateness of using tests. On the one hand, tests are seen as a means of positive improvement of the educational process in the direction of its technologicalization, reducing complexity and objectivity. On the other hand, tests are seen as a means of reducing the role of the teacher, and test results are sometimes considered unreliable.
The truth, as always, is somewhere in the middle. It is necessary to use and develop the best features of this approach and reject those that negatively affect the use of the tool. It should be noted that testing itself is gradually becoming the main form of exams. After all, the tests eliminate the shortcomings of empirical control: the test consists of a number of tasks in a certain direction and a standard known to the teacher that is a sample of complete and correct performance of the task.
At the same time, incorrect approach to the organization of testing or unreasonable assessment can lead to appeals, demotivation of respondents, claims against teachers, allegations of unfair and non-transparent assessment, disputes, and other misunderstandings. Therefore, the study of some non-trivial types of questions used in testing is relevant and necessary.
Despite comprehensive and partly well-founded criticism, testing the educational process has a number of advantages: it is a qualitative and objective method of assessment, as procedures for conducting and verifying the quality of test tasks for all respondents in the group are standardized; testing is a fairly accurate tool, as the scale of the test depends on the number of questions included in it, and can vary greatly; it is a sizable tool that allows you to determine the level of knowledge of the respondent throughout the discipline as a whole and in its individual sections; it is a soft tool that puts all respondents on equal conditions, using a single procedure and common evaluation criteria; it is a fair method of assessing knowledge, which puts all respondents on an equal footing in the process of control and assessment of the knowledge, significantly eliminating the subjectivity of the teacher: there is information that testing can reduce the number of appeals more than three times; is cost-effective because the main costs of this method of evaluation are one-time and are much lower than in written or oral control. 2
The problem of studying and improving testing for many years is the focus of
researchers of different countries. Various aspects and various problems of knowledge
control are studied by many domestic and foreign scientists [
Many researchers study various aspects of testing. Therefore, there are different classifications of tests that have different degrees of justification. For the purposes of this work, the authors propose a modified classification of test tasks. Test tasks are traditionally divided into two large groups: closed-ended and open-ended.
Today there are about 300 types of test questions. There are different approaches to the classification of test questions. In this article, we will follow this approach to the classification of test questions.
closedended questions Type 1.1 Type 1.2 Type 1.3 Type 1.4 Type 1.5 Problems using open-ended questions will not be considered in this paper. We formalize the first five types of questions in different classes of problems and unify the assessments on these questions. 4
To optimize knowledge control, it is necessary to use the following procedures: unification of assessment; ensuring the information unity of the evaluation; ensuring systematic evaluation; unambiguous interpretation of questions; unambiguous formation of the assessment on the basis of answers.
The implementation of these procedures is associated with the formalization of
question types, procedures for their evaluation, and criteria for determining a comprehensive
assessment [
Let the total number of questions that cover the subject area, according to the ontology, and form a logical scheme of testing, be equal n .
They are all divided into k types, depending on what type of answer should be matched the question.
TQhe seQt1o,fQquestionsisdescribed as follows:
2 ,..., Qn Q1, Q2 ,..., Qn1 , Qn1 1 ,..., Qnk1 , Qnk1 1,..., Qnk QQu2 estiQoni,sAoifa1t,yap2e,..1.,.a2par,1e formally presented as follows:
1 , QQu1estiQoni,sAoi ft0y,1pe,11.1 are formally presented as follows:
, where Q1 the subset of questions of type 1.1, Qi1 the questions of the first type, Qi1 Q1 , i 1,..., n1, Ai1 the set of possible answers to i th question, 1 - the number A1 , i 1,..., n1. of answers to be selected from the set of values i where i n1 1,..., n2, a j possible answers to questions of type 1.2, j 1,..., pi, pi the number of answer options.
QQu3estions of type 1.3 arep d,edfiin,ed by the following formalism:
Qi , Ai a1 , a2 ,..., a i where i n2 1,..., n3, a j possible answers to questions of type 1.3, di the possid const 2, pi and determined a priori or i ble number of answers, moreover i d a variable, di 1, pi.
Questions of type 1.4, where the elements of one set should be placed in accordance with the elements of another set, are called matching questions and formally described as foll4ows:Qi , Ai Ai1, Ai2 , f Di1, Di2
Q
, follow5ing Qfoir,mAialai1s,ma2:,..., ap , Ri ,
Q
i where i n3 1,..., n4 , Ai , Ai2 1
the sets of elements between which we should to establish a correspondence, f :D1i Di2 the answer in the form of an established correspondence between the sets Ai1, Ai2 .
Questions of type 1.5, where the correct sequence of actions or words (answer options, etc.) should be established, are used in ordering test tasks and described by the where i n4 1,..., n5 , a j answers options to questions of type 1.5 of the set of elementsRfior wahich it is necessary to establish the correct sequence, that is to build a ranking i1 ai2 ... ain5 .
Questions of type 1.1 and type 1.2 are trivial, so let us take a closer look at approaches to formalizing the last three types of questions. Each of the questions of type 1.3-1.5 contains ambiguity and uncertainty, so they should be formalized for the unambiguous perception of such questions. (1) (2) (3) (4) (5) (6)
Consider a formal description of multiple choice in closed-ended testing questions.
Note that models and methods of multiple choice based on the axiom of
non-displacement were studied in the monograph [
Suppose that we have a set of answer options ai A, i I 1,..., n, the number of which is equal to n, n A . Part of the answers n1 , n1 n, are correct and they form a subset A1 , A1 A, and the other part of the answers n0 , n0 n, are false and they form a subset A , A0 A, , moreover A1 A0 A. In addition, we assume that 0 all answers to the test ai A, i I , are equivalent.
For many test tasks, this statement is natural and logical. For example, to choose from given variants of numbers those that are divisors of a given number. There are many options for this type of task. That is, such an approach takes place in everyday life and the task of its formalization in testing is relevant. The peculiarity of such tasks is that they reflect the well-known truth: "Many men, many minds." Therefore, the decision must be justified to the measure prompted by the logic of its construction, the evaluation policy determined by the test organizers, common sense, and so on. For the specified statement of the testing problem, it is expedient to apply the algebraic approach to the definition of results of estimation which is successfully used in the theory of decision-making and at the application of technologies of expert estimation. In the algebraic approach, formalization involves the calculation and justification of all possible answers. The maximum number of points for a reliably selected subset of options is equal to B . The number of points for a correctly selected element of a subset of correct answers b B/ n1. 5.2
Therefore, problems that a priori depend on the subjective component cannot be solved without the use of heuristics. The heuristic formula of determination of a point estimation for a choice of answer variants in the form of the set V A which is generated by V set V can be different: from 0 to n, answers of the respondent is offered. Moreover, the number of elements 0 n.
selected by the respondent be 1 , and the number of incorrect answers, which he identified as correct be 0 , 1 0 n . Accordingly, the number of answer options that are not involved in the respondent's response to the question is equal to n .
Heuristics H1. The value of the fine k for each mismatch of the answer is entered. It is equal to: H1.1 - some reasonable coefficient k that reflects the subjective perception of the testing organizers about the "error price", for example, k 2 ; H1.2 - the value of the expression k 1 p0 , where p0 the probability of incorrect answer; H1.3 - the value of some function k f Е1 n0 , p0 , set by the experts, which depends on the number of incorrect answers and their probabilities.
Heuristics H2. For the incomplete answer, that is when n1 , a partial proportional assignment of points is assumed: H2.1 - in accordance with the ratio of the received correct answers 1 , to the total number of correct answers n1 ; H2.2 - the value of some function f Е 2 n1 , p1 , set by experts, where the p1 probability of obtaining the correct answer.
Of course, a partially correct answer can be guessed by the respondent with a higher probability, but the scores for it are also attributed proportionally lower.
Heuristics H3. For situations when the respondent did not select any answer 0 or all answers are marked as correct, i.e. n, the penalty is a zero score for lack of selectivity: B 0 .
Important and ambiguous is the situation when the respondent did not identify any correct answer. In this case, a respondent may indicate a different number of incorrect answers. Depending on the policy of planning test tasks and the position of the decisionmaker, this situation can be described and regulated by additional heuristics.
Heuristics H4.1. In the absence of correct answers, the score is always zero, regardless of the number of incorrect answers.
Heuristics E4.2. When the number of correct answers is zero, fewer incorrect answers are preferable to more incorrect answers.
To formalize this heuristic, we use the lower limit of the described situation. To do this, consider the decision-making situation, which is formalized by a tabular function c 1 1, 0 n0 , when the respondent identified one correct answer v1 1 and all n0 incorrect answers v0 n0 . According to the described technology, the value of the estimate is determined by the following heuristics.
Heuristics H5. We will assume that the situation c 1 0, 0 1 is next to the situac 1 1, 0 n0 tion and worsens the resulting assessment by one step, i.e. c 1 0, 0 1= c 1 1, 0 n0 - c 1 1, 0 n0 1 . The following situations of determining the resulting assessment are calculated in one of the ways: H5.1 - descending function: c 1 0, 0 i = c 1 0, 0 i 1/ k for i 2,..., n0 . E5.2 - the situation c 1 0, 0 n0 is equivalent to the situation 1 0 n : c 1 n1 , 0 n0 i.e. its consequence is a zero assessment of the respondent. In this case, estimates for different numbers of incorrect answers number of correct answers 1 0 are determined as follows: c 1 0, 0 i = c 1 1, 0 n0 - i c 1 1, 0 n0 / n0 , i 1,..., n0 . 0 1,..., n0 with zero 5.3
Let us consider the situation of constructing a test task with the following parameters: n 5, n1 3, n0 2. Without reducing the generality, we will assume that in the options for answers to the test question, the first three answers are correct, and the last two are incorrect. That is, the question is posed in such a way that true and false answers can be presented in the form of a vector: (1,1,1,0,0).
In the vectors that correspond to the answers of the respondent, the elements will be marked as follows: "1", if the respondent chose the correct answer, "0", if the respondent chose the wrong answer, and "*", if there is no answer.
Heuristics H1.1, H2.1 and H3 are used in the construction of this illustrative test assessment. Thus, heuristics H1, H2, H3 are transformed into such answer options and such estimates are calculated for them.
Answer 1: a1 1,*,*,*,* *,1,*,*,* *,*,1,*,* ca1 1/ 3 ; Answer 2: a2 1,1,*,*,* *,1,1,*,* 1,*,1,*,* ca 2 2 / 3 ; Answer 3: a3 1,1,1,*,* ca3 1; Answer 4: a4 1,*,*,0,* *,1,*,0,* *,*,1,0,* 1,*,*,*,0 *,1,*,*,0 *,*,1,*,0 ca 4 1/ 3 / 2 1/ 6 Answer 5: a5 1,*,*,0,0 *,1,*,0,0 *,*,1,0,0 ca5 1/ 3 / 4 1/12 ; Answer 6: a6 1,1,*,0,* *,1,1,0,* 1,*,1,0,* 1,1,*,*,0 1,*,1,*,0 ca 6 2 / 3 / 2 1/ 3 Answer 7: a7 1,1,*,0,0 *,1,1,0,0 1,*,1,0,0 ca7 2 / 3 / 4 1/ 6 ; Answer 8: a8 1,1,1,0,* 1,1,1,*,0 ca8 1/ 2 ; Answer 9: a9 1,1,1,0,* 1,1,1,*,0 ca9 1/ 2 . ; ;
When constructing test content, you can use heuristics that will have different patterns and correspond to other configurations of answers. Depending on the choice of heuristics, the sensitivity of the function that determines the value of the resulting estimate changes. But the problem of selecting such heuristics is not the subject of this work.
Statements of testing problems in the analysis of the answers of type 1.4 can be varied and their comprehensive review is not the subject of research in this paper. Let us consider only some aspects of research problems that may arise when using this type of testing. 6.1
Suppose we have some set A a1,..., an of n elements, which we will call the set of definitions, and a set B b1,..., bm of k elements, which we will call the set of values. The test (true, reliable, known, correct, ideal, etc.) correspondence of the elements of the0 set of definitions A to the elements of the set of values B given by1 the mapping f : A B . The task of the respondent is to establish the mapping f between the 0 sets A and B , and which will be as close as possible to the test mapping f . Based 0 1 on the differences between the test reflection f and the reflection f given by the respondents, a reasonable and fair assessment should be determined.
It is clear that this type of test task can have different relations: injection, surjection, and ideally bijection. The type of relation in the test task can be reported to the respondent in advance, before testing, or not be reported, then the student's task becomes more complicated. 6.2
A, A A At such statement of a problem various configurations of initial data can take place: all elements of the set A must be matched to all elements of the set B ; all elements of the set A must be matched to some elements of the set B , B B ; A, A A some elements of the set , must be matched to all elements of the set; some elements of the set
, must be matched to some elements of the set B , B B .
The following problem can be formalized using known formalisms: injective mapping, when a relationship is established between the elements of two sets, in which two different elements of the set A are never compared the same element of the set B ; surjective mapping, when for each element of the set B there is at least one element of the set A , such that f a b ; bijective reflection, which is both injective and surjective.
It is clear that the bijection between sets A and B can be established only if they are of equal power.
For further presentation, we introduce additional notations. Let A0s be a subset of objects of the set A , A0s A , the e0lements (element) of which correspond to the subset B s B in the test mapping f , s 1,2,... . The value of the index s 1,2,... depends on the specific statement of the problem and, in particular, on what mapping it is formalized: injection, surjection or bijection. The corresponding subsets given by the respondents will be denoted by Als A , l 1,..., k , where k the number of respondents.
The formulas for determining the similarity measures between the test mapping
and the mapping performed by the l th student, l 1,..., k , which are proposed in
[
l s1 3 2 Sl Als A0s /Als A0s / Sl l , , , (7) (8) (9) where the number of elements of the set Al is denoted by Al ; Sl , l 1,..., k , the s1 ,
(10) number of subsets of the set B used by the l th student in answering the test task.
Thef4ourthSl forlms ula proposed in [
1/ Sl , iflls, ths res1p,.o..n,dSent set whelrse the values l , l 1,..., k , are defined as follows: correct mathing of s th element, 0, if l th respondent was mistaken.
6.4
There are different interpretations of the described problem. Consider a situation where the set of definitions A consists of 9 elements: A a1,..., a9 , and the set of values B consists of 3 elements: B b1,b2 ,b3. The test mapping f 0 looks like a1, a3 , a7 b 3 .
1 , a2 , a5 , a6 , a8 b2 , a4 , a9 b The answers of the four respondents will be presented in the form of a table: b3 a 4 , a9 a4 a7 , a9 a6 , a9 4
0 0,778 0,556 0,444 0,333 Indices of respond
ents
The cells of the table contain subsets of the elements of the set A , which each of the respondents put in accordance with the subsets of the set B , each of which in this example consists of one element. The order of the elements of sets is insignificant and indicates only the fact of assignment to a subset, not its importance. The sign of an empty set means that the 2nd respondent did not refer to the third subset of the set B any element of the set A .
As a result of calculating the similarity measures given by formulas (7) - (10), we obtain table 3.
Based on the similarity measures calculated in this way, it is possible to determine the estimates of the respondents based on the test results. To do this, let us enter additional heuristics.
Heuristics H6. Choosing the similarity measure among those described by formulas (7) - (10). ForHeexuarmistpilces, Hcl7.Chlot osiMng, awfhoerrme ulal to tthreansscloartee,thelstimilsaerlietyctemdesaismurielarinittyoma esacsourree.
c for the l th respondent using the heuristics H6, M the maximum value of the rating scale.
This paper considers a closed-ended problem, namely the problem of arranging list elements in a certain sequence, i.e. determining the order of elements (objects, alternatives, entities), sequence of actions, operations, processes, calculations, chain of events, judgments, etc. In this case, the respondent is offered a list of concepts, phenomena, dates, words, etc., which he must arrange in the correct sequence. Such test tasks occur in various fields, for example: - establish a chronological sequence of events; - determine some logical sequence; - to formulate some definition from a set of randomly given words; - arrange some numbers in ascending or descending order; - to restore the order of proof of some theorem; - write a sequence of calculations when writing program code, which provides the definition of the value of a given formula, etc.
Such tasks help to formulate algorithmic thinking in students, consolidate the relevant knowledge and skills.
Consider a formal description of the ordering of elements in closed-ended questions
during testing. Note that the models and methods for determining the competence of
respondents on the basis of the axiom of immutability in the ranking of alternatives
were studied, in particular, in the monograph [
number of these elements is equal to a A, i I 1,..., n, is given, the Suppose a set of elements of a complete answer i n, n A .
fRor0 wahi1 ich ai2 the... reaspino,ndi ejnt I , rejceiIv.es linear (complete) order on this set, i.e. a strict ranking of the given elements of the answer. We will indicate the correct order of elements, which is known to the teacher, and R0 , the maximum score, by Thus, the testing procedure can be formalized in the class of ranking problems.
Note that the ability to guess the answer is the main reason for the negative attitude of teachers to the closed-ended form of tasks. To eliminate this shortcoming, even the correction of test scores on guess is used, the essence of which is that from the total score obtained by each respondent the number that can be guessed is subtracted in accordance with the provisions of probability theory.
Since the number of possible answers for the problems of ranking elements is equal to n! , then even if n 5, for example, n! 5! 120 , and it increases significantly with an increasing number of elements to be arranged. That is, the probability of guessing the correct answer is extremely low. Therefore, educators' warnings about the possibility of guessing are unfounded. When evaluating the tasks of ordering the set of given elements, the dichotomous evaluation of the task is most often used: "Yes" - "No", 0 or 1. Some heuristic evaluation rules are used less often. For example, a correctly completed task is evaluated with three points, the error at the end of the task is evaluated at 2 points, the error in the middle of the task is evaluated at 1 point, and the error at the beginning of the task entails zero evaluation value. It should be noted that sometimes in these tasks it is advisable to establish only a dichotomous, binary assessment. But a large number of test tasks allow for variation of estimates in a wide range.
We will apply the algebraic approach where it is appropriate and justified. That is, the value of the respondent’s assessment C R * will proportionally depend on the distance of 0 his answer R * to the correct (or - ideal, reference) answer R , and symbolically indicate this in this way: C R * B (1 d ( R 0 , R*) / d M ) , where B the maximum possible score for the answer, d M the maximum possible distance, i.e. the distance from the correct answer to the complet0ely aincorrect (the opposite to the ideal) answer. For example, when the correct answer is R 1 a2 a3 a4 , then the farthest from it, the "worst" answer is R* a4 a3 a2 a1 .
According to works [
The maximum possible distances between the standard and the worst ranking: d K M n 2 / 2
n ; and for even for Cook metrics of the form (11) d K M (n 2 1) / 2 for odd n ; for the Hamming metric of the form (12) the maximum distance is d H M n(n 1) / 2 .
It should be noted that the partial answers of the respondent should also be perceived and fairly assessed. It is clear that this procedure must be justified and formalized. That is, the approach described in this paper can be generalized in case of incomplete answers (when the respondent could not complete the test for technical reasons, did not have time to complete the test, does not know the complete correct answer, but is sure of its fragments, does not want to give a full ranking of a given set of elements). This situation can be considered as a case of incomplete rankings. 8
The paper proposes new approaches to calculating the assessment in testing using different types of questions: multiple choice questions; matching questions; ordering questions.
The approaches proposed by the authors are reasonable and formalized, so they can
be applied in different subject areas. A positive feature of the proposed approaches is
the transparency of a priori of testing rules set by the organizers, the absence of
situations of uncertainty during the evaluation procedure, monotony of the behavior of the
function, which reflects the integrated evaluation of the respondent. In addition, the
described approaches allow the possibility for further development and improvement.
Approaches previously used in practice have been proposed primarily because of their
simplicity. However, due to the development of soft computing [