Simulation of intellectual system for evaluation of multilevel test tasks on the basis of fuzzy logic Ivan M. Tsidylo1 , Serhiy O. Semerikov2,3,4,5 , Tetiana I. Gargula6 , Hanna V. Solonetska1 , Yaroslav P. Zamora1 and Andrey V. Pikilnyak3 1 Ternopil Volodymyr Hnatiuk National Pedagogical University, 2 Maksyma Kryvonosa Str., Ternopil, 46027, Ukraine 2 Kryvyi Rih State Pedagogical University, 54 Gagarin Ave., Kryvyi Rih, 50086, Ukraine 3 Kryvyi Rih National University, 11 Vitalii Matusevych Str., Kryvyi Rih, 50027, Ukraine 4 Institute of Information Technologies and Learning Tools of the NAES of Ukraine, 9 M. Berlynskoho Str., Kyiv, 04060, Ukraine 5 University of Educational Management, 52-A Sichovykh Striltsiv Str., Kyiv, 04053, Ukraine 6 I. Horbachevsky Ternopil National Medical University, 1 Voli Sq., Ternopil, 46001, Ukraine Abstract The article describes the stages of modeling an intelligent system for evaluating multilevel test tasks based on fuzzy logic in the MATLAB application package, namely the Fuzzy Logic Toolbox. The analysis of existing approaches to fuzzy assessment of test methods, their advantages and disadvantages is given. The considered methods for assessing students are presented in the general case by two methods: using fuzzy sets and corresponding membership functions; fuzzy estimation method and generalized fuzzy estimation method. In the present work, the Sugeno production model is used as the closest to the natural language. This closeness allows for closer interaction with a subject area expert and build well-understood, easily interpreted inference systems. The structure of a fuzzy system, functions and mechanisms of model building are described. The system is presented in the form of a block diagram of fuzzy logical nodes and consists of four input variables, corresponding to the levels of knowledge assimilation and one initial one. The surface of the response of a fuzzy system reflects the dependence of the final grade on the level of difficulty of the task and the degree of correctness of the task. The structure and functions of the fuzzy system are indicated. The modeled in this way intelligent system for assessing multilevel test tasks based on fuzzy logic makes it possible to take into account the fuzzy characteristics of the test: the level of difficulty of the task, which can be assessed as “easy”, “average", “above average”, “difficult”; the degree of correctness of the task, which can be assessed as “correct”, “partially correct”, “rather correct”, “incorrect”; time allotted for the execution of a test task or test, which can be assessed as “short”, “medium”, “long”, “very long”; the percentage of correctly completed tasks, which can be assessed as “small”, “medium”, “large”, “very large”; the final mark for the test, which can be assessed as “poor”, “satisfactory”, “good”, “excellent”, which are included in the assessment. This approach ensures the maximum consideration of answers to questions of all levels of complexity by formulating a base of inference rules and selection of weighting coefficients when deriving the final estimate. The robustness of the system is achieved by using Gaussian membership functions. The testing of the controller on the test sample brings the functional suitability of the developed model. Keywords intelligent system, multilevel test tasks, fuzzy test characteristics, fuzzy assessment, Sugeno inference system, 507 1. Introduction Test control is increasingly becoming an integral part of the educational process for all types and levels of educational institutions. Having become widespread in Western European countries and the United States, it is gradually gaining new positions in the domestic higher education. There are many practical implementations of automated testing systems, both in individual disciplines, and universal knowledge assessment systems, fully or partially invariant to specific disciplines and allowing teachers to edit their information content. Analysis of the effectiveness of automated testing in educational institutions shows that the most significant disadvantages of modern approaches to automated testing include [1, p. 4]: • the need to formulate options for answers to test items on the principle of “one is absolutely correct” – “other N are absolutely wrong”; • the primitiveness and inflexibility of the procedures for calculating the final grade, which can be reduced either to determining the ratio of the number of correct answers to the number of questions asked, or to the summation of points assigned for each correct answer; • impossibility of automating various methods of knowledge control, widely used in peda- gogical practice; • significant laboriousness of manual formation of such a set of test tasks and options for answers to each of them, which makes it possible to exclude or minimize the likelihood of presenting the same task to different people while simultaneously checking their knowledge. From this it follows that it is necessary to develop an automated knowledge control system, which requires the use of fundamentally different approaches to the presentation and processing of information based on methods and models developed within the framework of the theory of intelligent computing and knowledge engineering. A lot of studies in pedagogy are devoted to the issue of assessing knowledge. In particular: monitoring the quality of education (Cherednichenko and Yangolenko [2], He and He [3], Igbape and Idogho [4], Leontev et al. [5], Li et al. [6], Muhd Nor et al. [7], Qin et al. [8], Sorour et al. [9], Wei [10], Zhi and Nan [11] and others); development of modern innovative technologies that are included in the knowledge assessment system (Anohina-Naumeca et al. [12], Anohina-Naumeca CTE 2020: 8th Workshop on Cloud Technologies in Education, December 18, 2020, Kryvyi Rih, Ukraine " tsidylo@tnpu.edu.ua (I. M. Tsidylo); semerikov@gmail.com (S. O. Semerikov); gargulavsg@gmail.com (T. I. Gargula); Homenyuk_Hanna@tnpu.edu.ua (H. V. Solonetska); zamora@bigmir.net (Y. P. Zamora); pikilnyak@gmail.com (A. V. Pikilnyak) ~ http://tnpu.edu.ua/faculty/IPF/0002.php (I. M. Tsidylo); https://kdpu.edu.ua/semerikov (S. O. Semerikov); https://osta.tdmu.edu.ua/struktura/gargula-tetana-igorivna (T. I. Gargula); http://tnpu.edu.ua/faculty/fizmat/gomenyuk-ganna-volodimir-vna.php (H. V. Solonetska); http://tnpu.edu.ua/faculty/IPF/zamora-yaroslav-petrovich.php (Y. P. Zamora); http://www.knu.edu.ua/fakultety/fakul-tet-mehanichnoi-inzhenerii-ta-transportu/dekanat (A. V. Pikilnyak)  0000-0002-0202-348X (I. M. Tsidylo); 0000-0003-0789-0272 (S. O. Semerikov); 0000-0003-3335-0501 (T. I. Gargula); 0000-0002-2527-8653 (H. V. Solonetska); 0000-0001-6470-8233 (Y. P. Zamora); 0000-0003-0898-4756 (A. V. Pikilnyak) © 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) 508 and Grundspenkis [13], Gierłowski and Nowicki [14], Grundspenkis [15], Schmuck et al. [16], Szöllosi et al. [17] and others); the use of a multi-point scale for assessing knowledge, abilities, and skills (Bespalko [18], Linn [19] and others); theoretical approaches to the assessment of students’ knowledge, their development and improvement (Clotfelter et al. [20], Falchikov and Boud [21], Falchikov and Goldfinch [22], Host et al. [23], Hwang and Chang [24], Newble and Jaeger [25], Osadchyi et al. [26], Rust et al. [27], Scouller [28], Topping [29], Wiliam et al. [30] and others); evaluation of test results in an adaptive automated testing system, taking into account the ambiguity of the formulations of answers (Barker [31], Phankokkruad and Woraratpanya [32], Rudinskiy [1] and others). In [33] we substantiated the structural model of the neuro-fuzzy system of professional selection of students for training in IT specialties by studying the psychological characteristics, personal qualities and factual knowledge, skills and abilities of students as a unity of fuzzy and stochastic data base of the intellectual system. The issue of using fuzzy logic to describe the indicators of expert competence assessment using linguistic variables instead of numerical ones or in addition to them and the development of Sugeno’s intelligent system for determining expert competence was covered by us in [34]. The process of modeling intelligent systems based on fuzzy logic in various fields and analysis of the effectiveness of systems implemented in MATLAB is disclosed in the works of: Taylor [35] – fuzzy logic methodology, which is widely used in research and engineering practice and education, Lutsyk et al. [36] – use of parametric identification and adaptive neuro-fuzzy technologies to determine energy efficient modes of production equipment, Shtovba – the theory of fuzzy identification, methods of fuzzy clustering and their application for fuzzy rule extraction, as well as the method of decision-making in fuzzy conditions based on the merger of goals and constraints, author’s package solutions for designing fuzzy classifiers, building hierarchical fuzzy systems, training of fuzzy knowledge bases such as Mamdani, as well as for logical output with fuzzy source data [37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50]. 2. Materials and methods Models based on fuzzy logic are more flexible, as they mostly allow taking into account the experience and intuition of a specialist in a particular field. They are more adequate to the simulated reality and make it possible to obtain a solution correlated in accuracy with the initial data [51]. As a rule, the following characteristics are referred to fuzzy test characteristics: 1) the level of difficulty of the task, which can be assessed as “easy”, “average”, “above average”, “difficult”; 2) the degree of correctness of the task, which can be assessed as “correct”, “partially correct”, “rather correct”, “incorrect”; 3) time allotted for the execution of a test task or test, which can be assessed as “short”, “average”, “long”, “very long”; 4) percentage of correctly completed tasks, which can be assessed as “small”, “medium”, “large”, “very large”; 5) final mark for the test, which can be assessed as “bad”, “satisfactory”, “good”, “excellent”. 509 Among the fuzzy models for evaluating test results, adaptive models are interesting. In the work of Rudinskiy [1, p. 49], an adaptive model for evaluating the results of a “fuzzy” test is described. The idea is that the set of reference answers for each test item has a fuzzy grading scale. This fuzzy scale corresponds to the normalized numerical scale (1, 𝑡1 , 𝑡2 , 𝑡3 , 0), where 𝑡𝑖 ∈ (0, 1), 𝑖 = 1..3. All answers, except for the correct one, are assigned a subsequent question with a subset of answers. If an inaccurate answer is given to question 𝐷 at the 𝑖-th step of testing, a clarifying question is asked next, and the subset of answers contains both “more correct” (“correct”, “incomplete”) and “less correct” (“uncertain”, “wrong”) answers. If this question is answered differently from the correct one, no further additional questions are asked (otherwise the laboriousness of compiling such a structure of questions with subsets of answers to them would be very great), testing goes to the next step (question). Thus, the testing process can be represented as a movement along a directed graph, where vertices are questions, and arcs are transitions from the previous question to the next. An adaptive testing model using the apparatus of fuzzy logic is considered by Duplik [52, p. 60]. As a scale for evaluating test results, a 12-point scale proposed by Bespalko [18] is used. At the same time, the author proposes a correspondence between the percent of correct answers of the student and estimates on 12-point and 5-point scales, which, in turn, correspond to fuzzy concepts. Danilova [53, p. 17] developed an adaptive fuzzy model for evaluating the results of automated testing with division of tasks according to the levels of assimilation, proposed by Bespalko [18]. The paper presents models for evaluating test results: formalization of question-answer relations in test tasks according to the levels of assimilation is carried out for recognizing the answers of the tested person and formal presentation of test results; the scaling of the value estimates of the test items was performed; the bases of rules of fuzzy productions for evaluating test items of closed and open types have been developed; in order to ensure the adaptability of testing, a base of rules for fuzzy products has been developed for ranking tasks in the test; the calculation of the integral assessment of the test performance was done based on the assessment results of each test task. The fuzzy inference for evaluating the test results, based on the Mamdani method of fuzzy inference, is described. Belov [54] considers the problem of building an automated testing system (ATS) with the analysis of the respondent’s answers in natural language (NL). To recognize the responses of the person and the reference in the automated testing system, a linguistic analyzer module has been developed, which processes text in NL. The result of the surface-syntactic analysis of the phrases of the reference and user answers are syntactic dependency trees, including the word forms of the phrase, with the definition for each of them morphological descriptors and syntactic properties that combine words into syntactic fragments and groups. A limitation of the presented comparison model is the use of well-formed sentences. A sentence that is not well-formed is discarded by the linguistic analyzer with the requirement to the respondent to reformulate the answer. Each type of response is associated with a so-called syntactic template (SynT), which determines a set of typical syntactic constructions of a sentence and their significance. The obtained result – the degree of correspondence (relevance of phrases) – is taken as the degree of “correctness” of the respondent’s answer on the scale [0; 1]. 510 3. Results Thus, all the methods for evaluating test methods that we have considered have both advantages and disadvantages, which we have summarized in table 1 for clarity. Table 1 Advantages and disadvantages of test assessment methods Author Advantages Disadvantages Rudinskiy [1] The introduction of fuzziness in the orga- When evaluating test tasks and the test, nization of the adaptive test, which allows the apparatus of fuzzy logic is not used, the compilers of the test at the stage of and the obtained linguistic values are sim- its creation for each test task to build a ply projected onto a normalized numerical hierarchical structure of questions in the scale. The values obtained on this scale form of a directed graph. determine the degree of correctness of the answers, which are substituted into a spe- cially designed formula to obtain the final grade. Duplik [52] The use of a fuzzy logic apparatus to ob- The 12-point assessment scale, proposed tain an integral assessment of test results. by V. P. Bespalko, is used only to evenly The integral assessment is influenced by distribute the traditional 5-point scale on such indistinct characteristics of the test it and is not tied to the levels of assimila- as the current level of training, the per- tion of knowledge. centage of correct answers, the complex- ity of the task, and the time it takes to complete the task. Danilova [53] The sophistication of models for assess- The set of fuzzy production rules for evalu- ing test tasks, adaptive testing, integral ating test tasks with an open-ended ques- assessment of test results. tion is applicable only to test tasks of the “Substitution” type. Belov [54] Revealed classification of question types The graph comparison method is very la- and corresponding types of answers in bor intensive and complex. Firstly, the natural language. syntactic templates of all reference an- swers must be built in advance, and sec- ondly, the proximity of two phrases is de- termined on the scale [0; 1] by means of a complex algorithm, which would be easier to do using the apparatus of fuzzy logic. The considered methods for assessing students are presented in the general case by two methods: using fuzzy sets and corresponding membership functions; fuzzy estimation method and generalized fuzzy estimation method. The assessment system should be regularly reviewed and improved to ensure its suitability to assess students impartially and fairly. It makes sense to use a fuzzy model to describe an object when we do not have its analyt- ical description, or it is too cumbersome to use, but at the same time there is a sufficiently large amount of experimental data on the behavior of an object and/or heuristic rules for its functioning. 511 In this work, the Sugeno production model is used as the closest to the natural language. This closeness allows for closer interaction with a subject area expert and build well-understood, easily interpreted inference systems. It is important for us to develop an assessment strategy based on fuzzy sets, which requires careful consideration of the factors included in the assessment. These include: the level of difficulty of the task, the degree of correctness of the task, the final mark for the test, which can be assessed as “bad”, “satisfactory”, “good”, and “excellent”. The system is presented in the form of a block diagram of fuzzy logical nodes in figure 1 and consists of four input variables, corresponding to the levels of knowledge assimilation and one initial one. With this method, the system contains two nodes. The first node takes into account the level of complexity of the task and the degree of correctness of the task, depending on the supported task type of the automated system that is used for testing, for example Moodle [55]. The next three nodes behave like a fuzzy logic controller with two inputs with corresponding weights and one output, as in figure 2. 4. Fuzzy system implementation From the subject expert, we get the value of the matrix and the dimensions that describe the degree of importance of each question in the fuzzy domain, that is, the set of all allowed atomic values of the matrix column. The clear values are given as a vector. In the first node, the resulting data will be the experimental data, while the next nodes work as a fuzzy controller, the input of which is the output of the previous node (corresponding to the levels). The output of each node can be in the form of fuzzy values or in the form of linguistic variables. Each node has weighted coefficients that can be set equal to one with the equal influence of each input parameter. The output occurs according to the inference mechanism of the Sugeno fuzzy system. Here is a description of the system. System name: Correctness. Input variables: Level 1, Level 2, Level 3, Level 4. Initial variable: Final grade. The names of the terms of input variables: correct, wrong. The names of the terms of the original variable: correct, almost correct, partly correct, rather correct, probably wrong, wrong, zero. Fuzzy membership functions of the system are defined in the interval [0; 100] (see figure 3), the parameters of the input and initial ones, respectively, are given in tables 2 and 3. Set of rules “ If . . . then”: 1. If (level1 is wrong) and (level2 is wrong) and (level3 is wrong) and (level4 is wrong) then (final grade is zero) (1) 2. If (level1 is wrong) and (level2 is wrong) and (level3 is wrong) and (level4 is correct) then (final grade is probably wrong) (1) 3. If (level1 is wrong) and (level2 is wrong) and (level3 is correct) and (level4 is wrong) then (final grade is probably wrong) (1) 4. If (level1 is wrong) and (level2 is wrong) and (level3 is correct) and (level4 is correct) then (final grade is partly correct) (1) . . . 512 Figure 1: Block diagram of a fuzzy estimation system. 14. If (level1 is correct) and (level2 is correct) and (level3 is wrong) and (level4 is correct) then (final grade is almost correct) (1) 15. If (level1 is correct) and (level2 is correct) and (level3 is correct) and (level4 is wrong) then (final grade is partly correct) (1) 16. If (level1 is correct) and (level2 is correct) and (level3 is correct) and (level4 is correct) then (final-grade is correct) (1) As a result of modeling this system in the MATLAB application package, in particular the Fuzzy Logic Toolbox package, we obtained the response surfaces of the system at constant values of the input variables level3 and level4 equal to 50: in figure 4a – manually configured by an expert; in figure 4b – configured according to the ANFIS algorithm. Analysis of the response surface of a manually tuned system shows incorrect operation at intervals corresponding to 513 Figure 2: Node of presentation in the form of a fuzzy logical controller. Figure 3: Membership functions of input linguistic variables of a fuzzy system. intermediate values of functional membership such as constants of the output variable. To eliminate these differences, the fuzzy system was trained using the ANFIS algorithm based on the training sample. Training program: 514 Table 2 Parameters of membership functions of initial variables Name Type Parameters correct constant 100 almost correct constant 80 partly correct constant 65 rather correct constant 50 probably wrong constant 35 wrong constant 20 zero constant 0 Figure 4: The surface of the system response at constant values of the input variables level3 and level4 equal to 50: a – manually configured by an expert; b – configured according to the ANFIS algorithm. initfis = resdfis (’correctness’); (learn, error) = anfis (tr_data, initfis, 10); where the initial parameters: learn – a tuned system of the Sugeno type, the parameters of which minimize the error on the training set; error – system error at each training iteration; input parameters: tr_data – training sample; initfis – the original fuzzy output system; number 10 is responsible for the number of training iterations. As can be seen from figure 4b, the trained system according to the ANFIS algorithm reproduces the expert’s opinion as accurately as possible, which makes it possible, accordingly, to more accurately formulate the final assessment, taking into account the level of the tasks done correctly. The results of testing the fuzzy system are shown in table 4. 515 Table 3 Fuzzy system testing results Difficulty level 1 Difficulty level 2 Difficulty level 3 Difficulty level 4 Final Grade 20 25 21 36 26 50 65 75 25 52 60 45 25 0 28 98 78 60 10 52 27 35 15 0 18 60 68 50 35 50 85 90 50 54 66 85 87 80 76 81 15 7 2 0 6 56 45 90 84 73 5. Conclusion An intelligent system for assessing multilevel test tasks based on fuzzy logic modeled in this way makes it possible to consider all the above factors using fuzzy logic that are included in the assessment. This approach ensures the maximum consideration of answers to questions of all levels of complexity by formulating a base of inference rules and selection of weighting coefficients when deriving the final grade. The stability of the system is achieved by using Gaussian membership functions, as discussed in [56, p. 14]. We see the prospect of further research in the processing of the information received from the fuzzy system and the formulation of appropriate recommendations for specialists in different fields of knowledge for interpreting the final grade using multilevel test tasks. References [1] I. D. Rudinskiy, Fuzzy knowledge evaluation model as a methodological basis for au- tomation of pedagogical testing, IEEE Transactions on Education 50 (2007) 68–73. doi:10.1109/TE.2006.888904. [2] O. Cherednichenko, O. Yangolenko, Towards quality monitoring and evaluation methodol- ogy: Higher education case-study, Lecture Notes in Business Information Processing 137 (2013) 120–127. doi:10.1007/978-3-642-38370-0_11. [3] L. He, W. He, Study on the construction of internal monitoring system of chinese indepen- dent colleges’ education quality, Genova, 2009, pp. 191–194. [4] E. Igbape, P. Idogho, N-dimension data visualization spaces for academic programmes quality monitoring in nigeria higher education, volume 2019-October, Newswood Limited, 2019, pp. 238–242. [5] M. Leontev, N. Bondarenko, T. Shebzuhova, S. Butko, L. Egorova, Improving the efficiency of university management: Teacher’s performance monitoring as a tool to promote the quality of education, European Research Studies Journal 21 (2018) 527–540. doi:10.35808/ ersj/1020. 516 [6] Y. Li, P. Li, F. Zhu, R. Wang, Design of higher education quality monitoring and evaluation platform based on big data, Institute of Electrical and Electronics Engineers Inc., 2017, pp. 337–342. doi:10.1109/ICCSE.2017.8085513. [7] N. Muhd Nor, M. Azlan, S. Kiong, F. Mohamad, A. Ismail, A. Kasmin, M. Ahmad, S. Yokoyama, Development of course management and monitoring system as a qual- ity tools in engineering education, Applied Mechanics and Materials 465-466 (2014) 395–400. doi:10.4028/www.scientific.net/AMM.465-466.395. [8] F. Qin, W. Zeng, L. Li, R. Zhao, Construction of big data monitoring platform for teaching quality under intelligent education, Institute of Electrical and Electronics Engineers Inc., 2020, pp. 1594–1597. doi:10.1109/IWCMC48107.2020.9148224. [9] A. Sorour, A. Atkins, C. Stanier, F. Alharbi, Comparative frameworks for monitoring quality assurance in higher education institutions using business intelligence, Institute of Electrical and Electronics Engineers Inc., 2020. doi:10.1109/ICCIT-144147971.2020.9213808. [10] C. Wei, Higher vocational education quality monitoring system, Lecture Notes in Electrical Engineering 217 LNEE (2013) 55–60. doi:10.1007/978-1-4471-4850-0_8. [11] Z. Zhi, Z. Nan, Study on the construction of teaching quality monitoring system for the undergraduate physical education majors, Jilin, 2011, pp. 758–761. doi:10.1109/HHBE. 2011.6028937. [12] A. Anohina-Naumeca, M. Strautmane, J. Grundspenkis, Development of the scoring mechanism for the concept map based intelligent knowledge assessment system, volume 471, Sofia, 2010, pp. 376–381. doi:10.1145/1839379.1839446. [13] A. Anohina-Naumeca, J. Grundspenkis, Evaluating students’ concept maps in the con- cept map based intelligent knowledge assessment system, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5968 LNCS (2010) 8–15. doi:10.1007/978-3-642-12082-4_2. [14] K. Gierłowski, K. Nowicki, A novel architecture for e-learning knowledge assessment systems, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 4823 LNCS (2008) 276–287. doi:10.1007/ 978-3-540-78139-4_25. [15] J. Grundspenkis, Intelligent knowledge assessment systems: Myth or reality, Frontiers in Artificial Intelligence and Applications 315 (2019) 31–46. doi:10.3233/ 978-1-61499-941-6-31. [16] B. Schmuck, D. Sima, S. Szöllosi, The design space of the implementation of knowledge as- sessment systems, Sydney, NSW, 2006, pp. 587–593. doi:10.1109/ITHET.2006.339672. [17] S. Szöllosi, D. Sima, B. Schmuck, The design space of the services of knowledge assessment systems, Sydney, NSW, 2006, pp. 571–578. doi:10.1109/ITHET.2006.339816. [18] V. Bespalko, Requirements of educational films for professional and technical education, Soviet Education 2 (1960) 17–19. doi:10.2753/RES1060-9393020317. [19] R. Linn, Validating inferences from national assessment of educational progress achievement-level reporting, Applied Measurement in Education 11 (1998) 23–47. doi:10.1207/s15324818ame1101_2. [20] C. Clotfelter, H. Ladd, J. Vigdor, Teacher-student matching and the assessment of teacher effectiveness, Journal of Human Resources 41 (2006) 778–820. doi:10.3368/jhr.xli.4. 778. 517 [21] N. Falchikov, D. Boud, Student self-assessment in higher education: A meta-analysis, Review of Educational Research 59 (1989) 395–430. doi:10.3102/00346543059004395. [22] N. Falchikov, J. Goldfinch, Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks, Review of Educational Research 70 (2000) 287–322. doi:10.3102/00346543070003287. [23] M. Host, B. Regnell, C. Wohlin, Using students as subjects - a comparative study of students and professionals in lead-time impact assessment, Empirical Software Engineering 5 (2000) 201–214. doi:10.1023/A:1026586415054. [24] G.-J. Hwang, H.-F. Chang, A formative assessment-based mobile learning approach to improving the learning attitudes and achievements of students, Computers and Education 56 (2011) 1023–1031. doi:10.1016/j.compedu.2010.12.002. [25] D. Newble, K. Jaeger, The effect of assessments and examinations on the learning of medical students, Medical Education 17 (1983) 165–171. doi:10.1111/j.1365-2923. 1983.tb00657.x. [26] V. V. Osadchyi, K. P. Osadcha, H. B. Varina, S. V. Shevchenko, I. S. Bulakh, Specific features of the use of augmented reality technologies in the process of the development of cognitive component of future professionals’ mental capacity, Journal of Physics: Conference Series (2021). [27] C. Rust, M. Price, O. Berry, Improving students’ learning by developing their understanding of assessment criteria and processes, Assessment and Evaluation in Higher Education 28 (2003) 147–164. doi:10.1080/02602930301671. [28] K. Scouller, The influence of assessment method on students’ learning approaches: Multiple choice question examination versus assignment essay, Higher Education 35 (1998) 453–472. doi:10.1023/A:1003196224280. [29] K. Topping, Peer assessment between students in colleges and universities, Review of Educational Research 68 (1998) 249–276. doi:10.3102/00346543068003249. [30] D. Wiliam, C. Lee, C. Harrison, P. Black, Teachers developing assessment for learning: Impact on student achievement, Assessment in Education: Principles, Policy and Practice 11 (2004) 49–65. doi:10.1080/0969594042000208994. [31] T. Barker, An automated feedback system based on adaptive testing: Extending the model, International Journal of Emerging Technologies in Learning 5 (2010) 11–14. doi:10.3991/ ijet.v5i2.1235. [32] M. Phankokkruad, K. Woraratpanya, An automated decision system for computer adaptive testing using genetic algorithms, Phuket, 2008, pp. 655–660. doi:10.1109/SNPD.2008. 118. [33] B. Buyak, I. Tsidylo, V. Repskyi, V. Lyalyuk, Stages of conceptualization and formalization in the design of the model of the neuro-fuzzy expert system of professional selection of pupils, CEUR Workshop Proceedings 2257 (2018) 112–121. [34] H. V. Tereshchuk, I. M. Tsidylo, Automated system of fuzzy identification of expert’s competence for assessing the quality of pedagogical phenomena and processes, Information Technologies and Learning Tools 64 (2018) 234–244. URL: https://journal.iitta.gov.ua/index. php/itlt/article/view/2079. doi:10.33407/itlt.v64i2.2079. [35] A. Taylor, Fuzzy Logic With Matlab: Analyzing, Designing, and Simulating Systems, CreateSpace Independent Publishing Platform, 2017. 518 [36] I. Lutsyk, Y. Franko, V. Rak, I. Lutsyk, R. Leshchii, O. Potapchuk, Mathematical modeling of energy-efficient active ventilation modes of granary, in: 2019 9th International Conference on Advanced Computer Information Technologies (ACIT), 2019, pp. 105–108. doi:10. 1109/ACITT.2019.8780109. [37] S. Shtovba, O. Pankevich, A. Nagorna, Analyzing the criteria for fuzzy classifier learn- ing, Automatic Control and Computer Sciences 49 (2015) 123–132. doi:10.3103/ S0146411615030098. [38] A. Rotshtein, S. Shtovba, Predicting the reliability of algorithmic processes with fuzzy input data, Cybernetics and Systems Analysis 34 (1998) 545–552. doi:10.1007/BF02666999. [39] A. Rotshtein, S. Shtovba, Fuzzy multicriteria analysis of variants with the use of paired comparisons, Journal of Computer and Systems Sciences International 40 (2001) 499–503. [40] A. Rótshtein, S. Stovba, Managing a dynamic system by means of a fuzzy knowledge base, Automatic Control and Computer Sciences 35 (2001) 16–22. [41] A. Rotshtein, S. Shtovba, I. Mostav, Fuzzy rule based innovation projects estimation, volume 1, Vancouver, BC, 2001, pp. 122–126. [42] A. Rotshtejn, S. Shtovba, Influence of methods of defuzzification on speed of tuning the fuzzy model, Kibernetika i Sistemnyj Analiz (2002) 169–176. [43] A. Rotshtejn, S. Shtovba, Fuzzy rule based control of a dynamic system, Avtomatika i Vychislitel’naya Tekhnika (2001) 23–31. [44] A. Rotshteina, S. Shtovbab, Identification of a nonlinear dependence by a fuzzy knowl- edgebase in the case of a fuzzy training set, Cybernetics and Systems Analysis 42 (2006) 176–182. doi:10.1007/s10559-006-0051-1. [45] S. Shtovba, Fuzzy identification on the basis of regression models of parametric membership function, Journal of Automation and Information Sciences 38 (2006) 36–44. doi:10.1615/ JAutomatInfScien.v38.i11.40. [46] S. Shtovba, E. Shtovba, Prediction of competitive position of brand product by fuzzy knowledge base, Journal of Automation and Information Sciences 38 (2006) 69–80. doi:10. 1615/JAutomatInfScien.v38.i8.70. [47] S. Shtovba, Ensuring accuracy and transparency of Mamdani fuzzy model in learning by experimental data, Journal of Automation and Information Sciences 39 (2007) 39–52. doi:10.1615/JAutomatInfScien.v39.i8.50. [48] S. Shtovba, Fuzzy model tuning based on a training set with fuzzy model output values, Cy- bernetics and Systems Analysis 43 (2007) 334–340. doi:10.1007/s10559-007-0054-6. [49] A. Rotshtein, S. Shtovba, Modeling of the human operator reliability with the aid of the Sugeno fuzzy knowledge base, Automation and Remote Control 70 (2009) 163–169. doi:10.1134/S0005117909010123. [50] S. Shtovba, O. Pankevych, Fuzzy technology-based cause detection of structural cracks of stone buildings, volume 2105, CEUR-WS, 2018, pp. 209–218. [51] I. Khvostina, V. Oliinyk, S. Semerikov, V. Solovieva, V. Yatsenko, O. Kohut-Ferens, Hazards and risks in assessing the impact of oil and gas companies on the environment, IOP Conference Series: Earth and Environmental Science 628 (2021) 012027. doi:10.1088/ 1755-1315/628/1/012027. [52] S. V. Duplik, Model of adaptive testing on fuzzy mathematics, Computer science and education (2004) 57–65. 519 [53] S. D. Danilova, Evaluation of test results in an adaptive automated testing system, Bulletin of VSSTU (2008) 12–20. [54] E. A. Belov, Development of a method and algorithms for testing knowledge based on intelligent processing of answers in natural language, Ph.D. thesis, Bryansk, 2006. [55] P. Nechypurenko, S. Semerikov, VlabEmbed - the new plugin Moodle for the chemistry education, CEUR Workshop Proceedings 1844 (2017) 319–326. [56] I. A. Hameed, C. G. Sorensen, Fuzzy systems in education: A more reliable system for student evaluation, in: A. T. Azar (Ed.), Fuzzy Systems, Inte- chOpen, 2010, pp. 1–17. URL: https://www.intechopen.com/books/fuzzy-systems/ fuzzy-systems-in-education-a-more-reliable-system-for-student-evaluation. doi:10.5772/7216. 520