Development of information technologies has contributed significantly to the teaching methods and students' assessment. Tutorial programs, interactive tests, different monitoring studies and state programs for automated evaluation of knowledge and skills have been introduced recently.
Testing has been applied widely in distance education and during implementation of the Bologna Process for student's self-education. Automated testing application has been expanded to the manufacturing, where personnel management is transformed into a continuous process of training (of course, with the subsequent testing and assessment of trainees). The distinct feature of such systems is that the role of a teacher in the process of learning and assessment is much narrower, and the results have been evaluated automatically which has been caused by the requirement for simultaneous estimation of a large number of trainees, and by the ideology of automated learning itself -self-consistent learning and independent evaluation. One of the major tasks is the comparability of the results of different tests, ranging of students level of knowledge, formation of the final scoring for the test sets. Use of so-called "raw" scores, i.e., totals for the successful implementation of items resulting from the test might be applied to the very limited extent (if testing is limited to the identifying of the level of knowledge on particular topic and cannot be integrated with other results). Effectiveness of the test score depends not only on the quality of the test, but also on the methods of comparison and interpretation of primary (raw) score of test group.
Therefore one can assume as important to analyze the existing methods of comparison and integration of scores of various tests, study the quality of the students' group assessment, understanding the diversity of evaluation points as a quality criterion for estimating methods.
In 1999 the Polish Government decided to take part in the Bologna Process [1] and because of that all Polish educational institution at the beginning of the process of designing curricula define learning outcomes which students have to acquire during their studies and their teachers have to verify their acquirement by the students.
The mathematics curriculum in Polish secondary schools was approved by the Polish Ministry of Education [2]. This curriculum considers 5 learning outcomes which students are taught during three-year studies. At the end of the studies students take the written final secondary school examination.
The examination is dived into two parts, the first one is consisted of 25 items of multiple choice for which students can earn 1 point for answering correctly and the second part is consisted of 9 items for solving which students can earn more than 1 point. In this paper the data, we use to discuss the methods of preparing rankings, refers only to results of the first part of this examination.
The data comprised with results of the final secondary school examination in mathematics in one of Polish secondary schools in 2015. This examination was written by 149 students. For solving the first part of this examination students can earn up to 25 points. This paper presents discussion on the group of 18 students who earned the number of points equal to 18, so they all achieved the same position in the ranking of students based on the number of earned points or the average mean.
Sometimes we might encounter situations when there is a need to distinguish between these students, for example we would like to choose 5 out of these 18 students. So this ranking does not help us choose 5 better students unless we decide to apply more criteria.
In this paper we will discuss methods of preparing rankings [6,7] of students taking into consideration the results of the test in mathematics and experts' assessment of levels of verification of learning outcomes by test items and we discuss the results of 18 students who answering the first 25 items of the final secondary school examination in mathematics earned total score 18 points in one of Polish high schools.
In order to calculate levels of acquiring learning outcomes by students we will use the theory of fuzzy sets which was introduced by L.A. Zadeh [12] in 1965. In 1975 he generalized the concept of type 1 fuzzy sets and introduced type 2 fuzzy sets [13].
To build the relations between learning outcomes and items we will use the description of four leaning outcomes LO1 -LO4 written in [2]: ˗ LO1 -Student interprets mathematical texts. After solving the tasks, student interprets the achieved result. ˗ LO2 -Student uses simple, well known mathematical objects. ˗ LO3 -Student chooses a mathematical object to the simple situation and estimates the pertinence of model critically. ˗ LO4 -Student applies strategy which results clearly from the content of the task.
On the basis of principles of assessment published in [3], where for each learning outcome the experts indicate items prepared for verifying its acquirement by students, the crisp relation 1 R between learning outcomes and items was prepared, so the value of this relation
with the membership function defined as follows:
))) , (
In [9] there is shown that for educational purposes the algebraic T-norm and S-norm are better than the most popular T-norm minimum and S-norm maximum, so we will use S-T composition with algebraic T-norm and S-norm.
Learning outcomes
In this case all these students acquired learning outcomes LO1 -LO4 on the level 1 except student S10 whose level of acquirement of leaning outcome LO4 was 0. Thus we cannot distinguish between these students with the exception of S10 and we cannot prepare the ranking.
According to the Item Response Theory (IRT) mathematical ability (understanding mathematics and skills in solving tasks) is a latent trait which cannot be measured. This theory describes the method of measuring this ability on the basis of results of the test which was solved by the group of students [4,8,10].
According to the algorithm [10], the mathematical abilities described in learning outcomes LO1-LO4 (all together) and then in these learning outcomes taken into account separately were calculated and put to Table 2. On the basis of these values, the rankings of students were prepared and they are presented in Table 3. The main problem with levels of mathematical abilities is too few values (too many students achieved the same level of this ability) in order to prepare rankings and differentiate students. For example, when we take into account all learning outcomes (LO1-LO4) there are only 3 positions in the ranking, on the first position there are 7 students (S1, S3, S5, S7, S11, S12, S17), on the second one there are 9 students (S4, S6, S8, S10, S13, S14, S15, S16, S18) and on the third one there are 2 students (S2, S9).
The similar situation we encounter if we take as the basis of the ranking learning outcomes LO1, LO3 and LO4 (there are only 3 positions) and only for learning outcome LO2 there are 6 positions.
If we have more requirements according to the ranking, e.g. assume that learning outcome LO1 is the most important, then LO2, LO3 and the least important learning outcome is LO4, then we can distinguish students and using the lexicographical order we can prepare the ranking of the students presented in Table 4.
Now, we will fuzzify the relation between learning outcomes and items by letting the experts who define levels of verifying learning outcomes by items to use values from the interval [0,1]. The membership functions of the relation between the given learning outcome and items are presented in Fig. 2. 5. Now we can prepare rankings on the basis of levels of acquirement of each learning outcome LO1 -LO4 (separately) by the students. However, since they should acquire all learning outcomes, we prepare the ranking based on all of them using the lexicographical order. Assume that learning outcome LO1 is the most important, then LO2, LO3 and finally LO4. After using this information we can prepare ranking of students in Table 6. This ranking shows that student S2 is the best one when learning outcome LO1 is the most important one. Of course, if we choose the different order of importance of learning outcomes, the ranking will be different. The possibility of preparing different rankings according to specific criteria is really important for recruitment officers because they need candidates with specific and skills.
Comparing the fuzzification and the IRT method we can see that the IRT takes into consideration only the difficulties of items and the examinee's abilities. Our method enables to calculate the levels of learning outcomes' acquirement taking into consideration one, a few or all learning outcomes.
Now, we fuzzify further the relation between learning outcomes and items by letting the experts who defined levels of verifying learning outcomes define their own value belonging to the interval [0,1]. Hence the sample membership functions of the relation between the given learning outcome and items are presented in Fig. 3. In order to prepare another ranking of students, we will use type 2 fuzzy relations [11,13].
Since students can earn 0 or 1 points, so the secondary membership functions of the type 2 fuzzy relation between items and students can be the Gauss functions defined as
for each , j i and x . After the S-T composition, the sample secondary membership functions of the type 2 fuzzy relation between learning outcomes and students are presented in Fig. 4.
Student S2 Fig. 4. The membership functions of the relation type 2 fuzzy relation between learning outcomes LO1-LO4 and students. The next step is to find the method of comparing the results of students on the basis of the calculated secondary membership functions of the relation between learning outcomes and students. Let the level of acquirement of learning outcome j by the student i be equal to first coordinate i j
x , of the maximal point of the secondary membership function. Moreover, as we can see (Figure 3) some of the functions are "slim" and some are "wide". Hence we can assume that if the function is "slim", so the likelihood of the result is higher and when the function is "wide", so the likelihood is smaller.
Thus the likelihood of the level of acquiring learning outcome j by student i , called the range, is defined as follows:
we got the set of values of acquiring learning outcomes with their ranges. The pairs of these values for learning outcomes LO1 and LO2 all students are put to Table 8. To prepare the ranking we have to defuzzify the achieved results [11,4]. We assume that the first value should be greater and for students who achieved the same first values, the second one should be smaller. Thus on this basis we can prepare the ranking of students according to each learning outcome but as in the previous sections, we will present the ranking based on the lexicographic order assuming that learning outcome LO1 is most important, then LO2, LO3 and LO4. Hence we achieve the following ranking of students presented in Table 9. Comparing the rankings presented in Tables 6 and 9 we can notice that the first and last positions are the same and position of other students are similar. However, using the type 2 fuzzy relations we have more information because we can also describe the likelihood of acquirement of the learning outcomes by given students.
Even if this ranking did not differentiate students S6 and S18, it is not worse than the previous rankings. Moreover, we have got more information about likelihood of acquirement of learning outcomes.
Nowadays rankings of students are prepared very often. For example, in Poland all universities to admit students for the first year course prepare the ranking of candidates based on the results of the final secondary school examination and the grades of specific subjects. Universities could choose students based on the levels of acquirement of learning outcomes, not only on the average mean. All students discussed in the paper are on the same position after the first part of the examinations.
The paper presents different methods of preparing rankings based on the IRT algorithm and three different level of fuzzification of the relation between learning outcomes and items: crisp (two different position in the ranking), type 1 (we can prepare the ranking using additionally the lexicographical order of levels of acquirement of learning outcomes by students) and type 2 (the ranking based on the lexicographical order of levels of acquirement of learning outcomes by students gives additionally the information about likelihood of this acquirement).
The IRT method takes into account only difficulties of items and examinee's abilities described as learning outcomes. Moreover, when we prepared the ranking based on examinee's abilities of all discussed learning outcomes we have got only 3 positions.
The ranking prepared on the basis of the crisp relation between learning outcomes and items had only 2 positions. After the first and second fuzzifications, we have got the rankings with 17 positions and in the case of type 2 fuzzy relations we have got more information (the likelihood of levels of learning outcomes acquirement). The next step is to find more precise measure than range to calculate the likelihood of acquirement learning outcomes by students, for example the area between x-axis and Gauss functions.