The article describes the stages of modeling an intelligent system for evaluating multilevel test tasks based on fuzzy logic in the MATLAB application package, namely the Fuzzy Logic Toolbox. The analysis of existing approaches to fuzzy assessment of test methods, their advantages and disadvantages is given. The considered methods for assessing students are presented in the general case by two methods: using fuzzy sets and corresponding membership functions; fuzzy estimation method and generalized fuzzy estimation method. In the present work, the Sugeno production model is used as the closest to the natural language. This closeness allows for closer interaction with a subject area expert and build well-understood, easily interpreted inference systems. The structure of a fuzzy system, functions and mechanisms of model building are described. The system is presented in the form of a block diagram of fuzzy logical nodes and consists of four input variables, corresponding to the levels of knowledge assimilation and one initial one. The surface of the response of a fuzzy system reflects the dependence of the final grade on the level of dificulty of the task and the degree of correctness of the task. The structure and functions of the fuzzy system are indicated. The modeled in this way intelligent system for assessing multilevel test tasks based on fuzzy logic makes it possible to take into account the fuzzy characteristics of the test: the level of dificulty of the task, which can be assessed as “easy”, “average", “above average”, “dificult”; the degree of correctness of the task, which can be assessed as “correct”, “partially correct”, “rather correct”, “incorrect”; time allotted for the execution of a test task or test, which can be assessed as “short”, “medium”, “long”, “very long”; the percentage of correctly completed tasks, which can be assessed as “small”, “medium”, “large”, “very large”; the final mark for the test, which can be assessed as “poor”, “satisfactory”, “good”, “excellent”, which are included in the assessment. This approach ensures the maximum consideration of answers to questions of all levels of complexity by formulating a base of inference rules and selection of weighting coeficients when deriving the final estimate. The robustness of the system is achieved by using Gaussian membership functions. The testing of the controller on the test sample brings the functional suitability of the developed model.
Test control is increasingly becoming an integral part of the educational process for all types and levels of educational institutions. Having become widespread in Western European countries and the United States, it is gradually gaining new positions in the domestic higher education. There are many practical implementations of automated testing systems, both in individual disciplines, and universal knowledge assessment systems, fully or partially invariant to specific disciplines and allowing teachers to edit their information content. Analysis of the efectiveness of automated testing in educational institutions shows that the most significant disadvantages of modern approaches to automated testing include [1, p. 4]: • the need to formulate options for answers to test items on the principle of “one is absolutely correct” – “other N are absolutely wrong”; • the primitiveness and inflexibility of the procedures for calculating the final grade, which can be reduced either to determining the ratio of the number of correct answers to the number of questions asked, or to the summation of points assigned for each correct answer; • impossibility of automating various methods of knowledge control, widely used in pedagogical practice; • significant laboriousness of manual formation of such a set of test tasks and options for answers to each of them, which makes it possible to exclude or minimize the likelihood of presenting the same task to diferent people while simultaneously checking their knowledge.
From this it follows that it is necessary to develop an automated knowledge control system, which requires the use of fundamentally diferent approaches to the presentation and processing of information based on methods and models developed within the framework of the theory of intelligent computing and knowledge engineering.
A lot of studies in pedagogy are devoted to the issue of assessing knowledge. In particular:
monitoring the quality of education (Cherednichenko and Yangolenko [
Models based on fuzzy logic are more flexible, as they mostly allow taking into account the experience and intuition of a specialist in a particular field. They are more adequate to the simulated reality and make it possible to obtain a solution correlated in accuracy with the initial data [51].
As a rule, the following characteristics are referred to fuzzy test characteristics: 1) the level of dificulty of the task, which can be assessed as “easy”, “average”, “above average”, “dificult”; 2) the degree of correctness of the task, which can be assessed as “correct”, “partially correct”, “rather correct”, “incorrect”; 3) time allotted for the execution of a test task or test, which can be assessed as “short”, “average”, “long”, “very long”; 4) percentage of correctly completed tasks, which can be assessed as “small”, “medium”, “large”, “very large”; 5) final mark for the test, which can be assessed as “bad”, “satisfactory”, “good”, “excellent”.
Among the fuzzy models for evaluating test results, adaptive models are interesting. In the work of Rudinskiy [1, p. 49], an adaptive model for evaluating the results of a “fuzzy” test is described. The idea is that the set of reference answers for each test item has a fuzzy grading scale. This fuzzy scale corresponds to the normalized numerical scale (1, 1, 2, 3, 0), where ∈ (0, 1), = 1..3. All answers, except for the correct one, are assigned a subsequent question with a subset of answers. If an inaccurate answer is given to question at the -th step of testing, a clarifying question is asked next, and the subset of answers contains both “more correct” (“correct”, “incomplete”) and “less correct” (“uncertain”, “wrong”) answers. If this question is answered diferently from the correct one, no further additional questions are asked (otherwise the laboriousness of compiling such a structure of questions with subsets of answers to them would be very great), testing goes to the next step (question). Thus, the testing process can be represented as a movement along a directed graph, where vertices are questions, and arcs are transitions from the previous question to the next.
An adaptive testing model using the apparatus of fuzzy logic is considered by Duplik [52,
p. 60]. As a scale for evaluating test results, a 12-point scale proposed by Bespalko [
Danilova [53, p. 17] developed an adaptive fuzzy model for evaluating the results of automated
testing with division of tasks according to the levels of assimilation, proposed by Bespalko [
Belov [54] considers the problem of building an automated testing system (ATS) with the analysis of the respondent’s answers in natural language (NL). To recognize the responses of the person and the reference in the automated testing system, a linguistic analyzer module has been developed, which processes text in NL. The result of the surface-syntactic analysis of the phrases of the reference and user answers are syntactic dependency trees, including the word forms of the phrase, with the definition for each of them morphological descriptors and syntactic properties that combine words into syntactic fragments and groups.
A limitation of the presented comparison model is the use of well-formed sentences. A sentence that is not well-formed is discarded by the linguistic analyzer with the requirement to the respondent to reformulate the answer. Each type of response is associated with a so-called syntactic template (SynT), which determines a set of typical syntactic constructions of a sentence and their significance. The obtained result – the degree of correspondence (relevance of phrases) – is taken as the degree of “correctness” of the respondent’s answer on the scale [0; 1].
Thus, all the methods for evaluating test methods that we have considered have both advantages
and disadvantages, which we have summarized in table 1 for clarity.
Rudinskiy [
Duplik [52] The use of a fuzzy logic apparatus to ob- The 12-point assessment scale, proposed tain an integral assessment of test results. by V. P. Bespalko, is used only to evenly The integral assessment is influenced by distribute the traditional 5-point scale on such indistinct characteristics of the test it and is not tied to the levels of assimilaas the current level of training, the per- tion of knowledge. centage of correct answers, the complexity of the task, and the time it takes to complete the task.
Danilova [53] The sophistication of models for assess- The set of fuzzy production rules for evaluing test tasks, adaptive testing, integral ating test tasks with an open-ended quesassessment of test results. tion is applicable only to test tasks of the “Substitution” type.
Belov [54] Revealed classification of question types The graph comparison method is very laand corresponding types of answers in bor intensive and complex. Firstly, the natural language. syntactic templates of all reference answers must be built in advance, and secondly, the proximity of two phrases is determined on the scale [0; 1] by means of a complex algorithm, which would be easier to do using the apparatus of fuzzy logic.
The considered methods for assessing students are presented in the general case by two methods: using fuzzy sets and corresponding membership functions; fuzzy estimation method and generalized fuzzy estimation method. The assessment system should be regularly reviewed and improved to ensure its suitability to assess students impartially and fairly.
It makes sense to use a fuzzy model to describe an object when we do not have its analytical description, or it is too cumbersome to use, but at the same time there is a suficiently large amount of experimental data on the behavior of an object and/or heuristic rules for its functioning.
In this work, the Sugeno production model is used as the closest to the natural language. This closeness allows for closer interaction with a subject area expert and build well-understood, easily interpreted inference systems.
It is important for us to develop an assessment strategy based on fuzzy sets, which requires careful consideration of the factors included in the assessment. These include: the level of dificulty of the task, the degree of correctness of the task, the final mark for the test, which can be assessed as “bad”, “satisfactory”, “good”, and “excellent”. The system is presented in the form of a block diagram of fuzzy logical nodes in figure 1 and consists of four input variables, corresponding to the levels of knowledge assimilation and one initial one. With this method, the system contains two nodes. The first node takes into account the level of complexity of the task and the degree of correctness of the task, depending on the supported task type of the automated system that is used for testing, for example Moodle [55].
The next three nodes behave like a fuzzy logic controller with two inputs with corresponding weights and one output, as in figure 2.
From the subject expert, we get the value of the matrix and the dimensions that describe the degree of importance of each question in the fuzzy domain, that is, the set of all allowed atomic values of the matrix column. The clear values are given as a vector. In the first node, the resulting data will be the experimental data, while the next nodes work as a fuzzy controller, the input of which is the output of the previous node (corresponding to the levels). The output of each node can be in the form of fuzzy values or in the form of linguistic variables. Each node has weighted coeficients that can be set equal to one with the equal influence of each input parameter. The output occurs according to the inference mechanism of the Sugeno fuzzy system. Here is a description of the system.
System name: Correctness.
Input variables: Level 1, Level 2, Level 3, Level 4.
Initial variable: Final grade.
The names of the terms of input variables: correct, wrong.
The names of the terms of the original variable: correct, almost correct, partly correct, rather correct, probably wrong, wrong, zero.
Fuzzy membership functions of the system are defined in the interval [0; 100] (see figure 3), the parameters of the input and initial ones, respectively, are given in tables 2 and 3.
Set of rules “ If . . . then”: 1. If (level1 is wrong) and (level2 is wrong) and (level3 is wrong) and (level4 is wrong) then (final grade is zero) (1)
2. If (level1 is wrong) and (level2 is wrong) and (level3 is wrong) and (level4 is correct) then (final grade is probably wrong) (1)
3. If (level1 is wrong) and (level2 is wrong) and (level3 is correct) and (level4 is wrong) then (final grade is probably wrong) (1)
4. If (level1 is wrong) and (level2 is wrong) and (level3 is correct) and (level4 is correct) then (final grade is partly correct) (1) . . .
14. If (level1 is correct) and (level2 is correct) and (level3 is wrong) and (level4 is correct) then (final grade is almost correct) (1)
15. If (level1 is correct) and (level2 is correct) and (level3 is correct) and (level4 is wrong) then (final grade is partly correct) (1)
16. If (level1 is correct) and (level2 is correct) and (level3 is correct) and (level4 is correct) then (final-grade is correct) (1)
As a result of modeling this system in the MATLAB application package, in particular the Fuzzy Logic Toolbox package, we obtained the response surfaces of the system at constant values of the input variables level3 and level4 equal to 50: in figure 4a – manually configured by an expert; in figure 4b – configured according to the ANFIS algorithm. Analysis of the response surface of a manually tuned system shows incorrect operation at intervals corresponding to intermediate values of functional membership such as constants of the output variable. To eliminate these diferences, the fuzzy system was trained using the ANFIS algorithm based on the training sample.
Training program:
Name correct almost correct partly correct rather correct probably wrong wrong zero
Type
Parameters initfis = resdfis (’correctness’); (learn, error) = anfis (tr_data, initfis, 10); where the initial parameters: learn – a tuned system of the Sugeno type, the parameters of which minimize the error on the training set; error – system error at each training iteration; input parameters: tr_data – training sample; initfis – the original fuzzy output system; number 10 is responsible for the number of training iterations.
As can be seen from figure 4b, the trained system according to the ANFIS algorithm reproduces the expert’s opinion as accurately as possible, which makes it possible, accordingly, to more accurately formulate the final assessment, taking into account the level of the tasks done correctly. The results of testing the fuzzy system are shown in table 4.
An intelligent system for assessing multilevel test tasks based on fuzzy logic modeled in this way makes it possible to consider all the above factors using fuzzy logic that are included in the assessment. This approach ensures the maximum consideration of answers to questions of all levels of complexity by formulating a base of inference rules and selection of weighting coeficients when deriving the final grade. The stability of the system is achieved by using Gaussian membership functions, as discussed in [56, p. 14]. We see the prospect of further research in the processing of the information received from the fuzzy system and the formulation of appropriate recommendations for specialists in diferent fields of knowledge for interpreting the final grade using multilevel test tasks. [21] N. Falchikov, D. Boud, Student self-assessment in higher education: A meta-analysis,
Review of Educational Research 59 (1989) 395–430. doi:10.3102/00346543059004395. [22] N. Falchikov, J. Goldfinch, Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks, Review of Educational Research 70 (2000) 287–322. doi:10.3102/00346543070003287. [23] M. Host, B. Regnell, C. Wohlin, Using students as subjects - a comparative study of students and professionals in lead-time impact assessment, Empirical Software Engineering 5 (2000) 201–214. doi:10.1023/A:1026586415054. [24] G.-J. Hwang, H.-F. Chang, A formative assessment-based mobile learning approach to improving the learning attitudes and achievements of students, Computers and Education 56 (2011) 1023–1031. doi:10.1016/j.compedu.2010.12.002. [25] D. Newble, K. Jaeger, The efect of assessments and examinations on the learning of medical students, Medical Education 17 (1983) 165–171. doi:10.1111/j.1365-2923. 1983.tb00657.x. [26] V. V. Osadchyi, K. P. Osadcha, H. B. Varina, S. V. Shevchenko, I. S. Bulakh, Specific features of the use of augmented reality technologies in the process of the development of cognitive component of future professionals’ mental capacity, Journal of Physics: Conference Series (2021). [27] C. Rust, M. Price, O. Berry, Improving students’ learning by developing their understanding of assessment criteria and processes, Assessment and Evaluation in Higher Education 28 (2003) 147–164. doi:10.1080/02602930301671. [28] K. Scouller, The influence of assessment method on students’ learning approaches: Multiple choice question examination versus assignment essay, Higher Education 35 (1998) 453–472. doi:10.1023/A:1003196224280. [29] K. Topping, Peer assessment between students in colleges and universities, Review of
Educational Research 68 (1998) 249–276. doi:10.3102/00346543068003249. [30] D. Wiliam, C. Lee, C. Harrison, P. Black, Teachers developing assessment for learning: Impact on student achievement, Assessment in Education: Principles, Policy and Practice 11 (2004) 49–65. doi:10.1080/0969594042000208994. [31] T. Barker, An automated feedback system based on adaptive testing: Extending the model, International Journal of Emerging Technologies in Learning 5 (2010) 11–14. doi:10.3991/ ijet.v5i2.1235. [32] M. Phankokkruad, K. Woraratpanya, An automated decision system for computer adaptive testing using genetic algorithms, Phuket, 2008, pp. 655–660. doi:10.1109/SNPD.2008. 118. [33] B. Buyak, I. Tsidylo, V. Repskyi, V. Lyalyuk, Stages of conceptualization and formalization in the design of the model of the neuro-fuzzy expert system of professional selection of pupils, CEUR Workshop Proceedings 2257 (2018) 112–121. [34] H. V. Tereshchuk, I. M. Tsidylo, Automated system of fuzzy identification of expert’s competence for assessing the quality of pedagogical phenomena and processes, Information Technologies and Learning Tools 64 (2018) 234–244. URL: https://journal.iitta.gov.ua/index. php/itlt/article/view/2079. doi:10.33407/itlt.v64i2.2079. [35] A. Taylor, Fuzzy Logic With Matlab: Analyzing, Designing, and Simulating Systems, CreateSpace Independent Publishing Platform, 2017. [36] I. Lutsyk, Y. Franko, V. Rak, I. Lutsyk, R. Leshchii, O. Potapchuk, Mathematical modeling of energy-eficient active ventilation modes of granary, in: 2019 9th International Conference on Advanced Computer Information Technologies (ACIT), 2019, pp. 105–108. doi:10. 1109/ACITT.2019.8780109. [37] S. Shtovba, O. Pankevich, A. Nagorna, Analyzing the criteria for fuzzy classifier learning, Automatic Control and Computer Sciences 49 (2015) 123–132. doi:10.3103/ S0146411615030098. [38] A. Rotshtein, S. Shtovba, Predicting the reliability of algorithmic processes with fuzzy input data, Cybernetics and Systems Analysis 34 (1998) 545–552. doi:10.1007/BF02666999. [39] A. Rotshtein, S. Shtovba, Fuzzy multicriteria analysis of variants with the use of paired comparisons, Journal of Computer and Systems Sciences International 40 (2001) 499–503. [40] A. Rótshtein, S. Stovba, Managing a dynamic system by means of a fuzzy knowledge base,
Automatic Control and Computer Sciences 35 (2001) 16–22. [41] A. Rotshtein, S. Shtovba, I. Mostav, Fuzzy rule based innovation projects estimation, volume 1, Vancouver, BC, 2001, pp. 122–126. [42] A. Rotshtejn, S. Shtovba, Influence of methods of defuzzification on speed of tuning the fuzzy model, Kibernetika i Sistemnyj Analiz (2002) 169–176. [43] A. Rotshtejn, S. Shtovba, Fuzzy rule based control of a dynamic system, Avtomatika i
Vychislitel’naya Tekhnika (2001) 23–31. [44] A. Rotshteina, S. Shtovbab, Identification of a nonlinear dependence by a fuzzy knowledgebase in the case of a fuzzy training set, Cybernetics and Systems Analysis 42 (2006) 176–182. doi:10.1007/s10559-006-0051-1. [45] S. Shtovba, Fuzzy identification on the basis of regression models of parametric membership function, Journal of Automation and Information Sciences 38 (2006) 36–44. doi:10.1615/ JAutomatInfScien.v38.i11.40. [46] S. Shtovba, E. Shtovba, Prediction of competitive position of brand product by fuzzy knowledge base, Journal of Automation and Information Sciences 38 (2006) 69–80. doi:10. 1615/JAutomatInfScien.v38.i8.70. [47] S. Shtovba, Ensuring accuracy and transparency of Mamdani fuzzy model in learning by experimental data, Journal of Automation and Information Sciences 39 (2007) 39–52. doi:10.1615/JAutomatInfScien.v39.i8.50. [48] S. Shtovba, Fuzzy model tuning based on a training set with fuzzy model output values, Cybernetics and Systems Analysis 43 (2007) 334–340. doi:10.1007/s10559-007-0054-6. [49] A. Rotshtein, S. Shtovba, Modeling of the human operator reliability with the aid of the Sugeno fuzzy knowledge base, Automation and Remote Control 70 (2009) 163–169. doi:10.1134/S0005117909010123. [50] S. Shtovba, O. Pankevych, Fuzzy technology-based cause detection of structural cracks of stone buildings, volume 2105, CEUR-WS, 2018, pp. 209–218. [51] I. Khvostina, V. Oliinyk, S. Semerikov, V. Solovieva, V. Yatsenko, O. Kohut-Ferens, Hazards and risks in assessing the impact of oil and gas companies on the environment, IOP Conference Series: Earth and Environmental Science 628 (2021) 012027. doi:10.1088/ 1755-1315/628/1/012027. [52] S. V. Duplik, Model of adaptive testing on fuzzy mathematics, Computer science and education (2004) 57–65. [53] S. D. Danilova, Evaluation of test results in an adaptive automated testing system, Bulletin of VSSTU (2008) 12–20. [54] E. A. Belov, Development of a method and algorithms for testing knowledge based on intelligent processing of answers in natural language, Ph.D. thesis, Bryansk, 2006. [55] P. Nechypurenko, S. Semerikov, VlabEmbed - the new plugin Moodle for the chemistry education, CEUR Workshop Proceedings 1844 (2017) 319–326. [56] I. A. Hameed, C. G. Sorensen, Fuzzy systems in education: A more reliable system for student evaluation, in: A. T. Azar (Ed.), Fuzzy Systems, IntechOpen, 2010, pp. 1–17. URL: https://www.intechopen.com/books/fuzzy-systems/ fuzzy-systems-in-education-a-more-reliable-system-for-student-evaluation. doi:10.5772/7216.