=Paper=
{{Paper
|id=Vol-2879/paper30
|storemode=property
|title=Simulation of intellectual system for evaluation of multilevel test tasks on the basis of fuzzy logic
|pdfUrl=https://ceur-ws.org/Vol-2879/paper30.pdf
|volume=Vol-2879
|authors=Ivan M. Tsidylo,Serhiy O. Semerikov,Tetiana I. Gargula,Hanna V. Solonetska,Yaroslav P. Zamora,Andrey V. Pikilnyak
}}
==Simulation of intellectual system for evaluation of multilevel test tasks on the basis of fuzzy logic==
https://ceur-ws.org/Vol-2879/paper30.pdf
Simulation of intellectual system for evaluation of
multilevel test tasks on the basis of fuzzy logic
Ivan M. Tsidylo1 , Serhiy O. Semerikov2,3,4,5 , Tetiana I. Gargula6 , Hanna V. Solonetska1 ,
Yaroslav P. Zamora1 and Andrey V. Pikilnyak3
1
Ternopil Volodymyr Hnatiuk National Pedagogical University, 2 Maksyma Kryvonosa Str., Ternopil, 46027, Ukraine
2
Kryvyi Rih State Pedagogical University, 54 Gagarin Ave., Kryvyi Rih, 50086, Ukraine
3
Kryvyi Rih National University, 11 Vitalii Matusevych Str., Kryvyi Rih, 50027, Ukraine
4
Institute of Information Technologies and Learning Tools of the NAES of Ukraine, 9 M. Berlynskoho Str., Kyiv, 04060,
Ukraine
5
University of Educational Management, 52-A Sichovykh Striltsiv Str., Kyiv, 04053, Ukraine
6
I. Horbachevsky Ternopil National Medical University, 1 Voli Sq., Ternopil, 46001, Ukraine
Abstract
The article describes the stages of modeling an intelligent system for evaluating multilevel test tasks
based on fuzzy logic in the MATLAB application package, namely the Fuzzy Logic Toolbox. The analysis
of existing approaches to fuzzy assessment of test methods, their advantages and disadvantages is given.
The considered methods for assessing students are presented in the general case by two methods: using
fuzzy sets and corresponding membership functions; fuzzy estimation method and generalized fuzzy
estimation method. In the present work, the Sugeno production model is used as the closest to the
natural language. This closeness allows for closer interaction with a subject area expert and build
well-understood, easily interpreted inference systems. The structure of a fuzzy system, functions and
mechanisms of model building are described. The system is presented in the form of a block diagram
of fuzzy logical nodes and consists of four input variables, corresponding to the levels of knowledge
assimilation and one initial one. The surface of the response of a fuzzy system reflects the dependence of
the final grade on the level of difficulty of the task and the degree of correctness of the task. The structure
and functions of the fuzzy system are indicated. The modeled in this way intelligent system for assessing
multilevel test tasks based on fuzzy logic makes it possible to take into account the fuzzy characteristics
of the test: the level of difficulty of the task, which can be assessed as “easy”, “average", “above average”,
“difficult”; the degree of correctness of the task, which can be assessed as “correct”, “partially correct”,
“rather correct”, “incorrect”; time allotted for the execution of a test task or test, which can be assessed
as “short”, “medium”, “long”, “very long”; the percentage of correctly completed tasks, which can be
assessed as “small”, “medium”, “large”, “very large”; the final mark for the test, which can be assessed as
“poor”, “satisfactory”, “good”, “excellent”, which are included in the assessment. This approach ensures
the maximum consideration of answers to questions of all levels of complexity by formulating a base of
inference rules and selection of weighting coefficients when deriving the final estimate. The robustness
of the system is achieved by using Gaussian membership functions. The testing of the controller on the
test sample brings the functional suitability of the developed model.
Keywords
intelligent system, multilevel test tasks, fuzzy test characteristics, fuzzy assessment, Sugeno inference
system,
507
1. Introduction
Test control is increasingly becoming an integral part of the educational process for all types and
levels of educational institutions. Having become widespread in Western European countries
and the United States, it is gradually gaining new positions in the domestic higher education.
There are many practical implementations of automated testing systems, both in individual
disciplines, and universal knowledge assessment systems, fully or partially invariant to specific
disciplines and allowing teachers to edit their information content. Analysis of the effectiveness
of automated testing in educational institutions shows that the most significant disadvantages
of modern approaches to automated testing include [1, p. 4]:
• the need to formulate options for answers to test items on the principle of “one is absolutely
correct” – “other N are absolutely wrong”;
• the primitiveness and inflexibility of the procedures for calculating the final grade, which
can be reduced either to determining the ratio of the number of correct answers to the
number of questions asked, or to the summation of points assigned for each correct
answer;
• impossibility of automating various methods of knowledge control, widely used in peda-
gogical practice;
• significant laboriousness of manual formation of such a set of test tasks and options for
answers to each of them, which makes it possible to exclude or minimize the likelihood
of presenting the same task to different people while simultaneously checking their
knowledge.
From this it follows that it is necessary to develop an automated knowledge control system,
which requires the use of fundamentally different approaches to the presentation and processing
of information based on methods and models developed within the framework of the theory of
intelligent computing and knowledge engineering.
A lot of studies in pedagogy are devoted to the issue of assessing knowledge. In particular:
monitoring the quality of education (Cherednichenko and Yangolenko [2], He and He [3], Igbape
and Idogho [4], Leontev et al. [5], Li et al. [6], Muhd Nor et al. [7], Qin et al. [8], Sorour et al. [9],
Wei [10], Zhi and Nan [11] and others); development of modern innovative technologies that are
included in the knowledge assessment system (Anohina-Naumeca et al. [12], Anohina-Naumeca
CTE 2020: 8th Workshop on Cloud Technologies in Education, December 18, 2020, Kryvyi Rih, Ukraine
" tsidylo@tnpu.edu.ua (I. M. Tsidylo); semerikov@gmail.com (S. O. Semerikov); gargulavsg@gmail.com
(T. I. Gargula); Homenyuk_Hanna@tnpu.edu.ua (H. V. Solonetska); zamora@bigmir.net (Y. P. Zamora);
pikilnyak@gmail.com (A. V. Pikilnyak)
~ http://tnpu.edu.ua/faculty/IPF/0002.php (I. M. Tsidylo); https://kdpu.edu.ua/semerikov (S. O. Semerikov);
https://osta.tdmu.edu.ua/struktura/gargula-tetana-igorivna (T. I. Gargula);
http://tnpu.edu.ua/faculty/fizmat/gomenyuk-ganna-volodimir-vna.php (H. V. Solonetska);
http://tnpu.edu.ua/faculty/IPF/zamora-yaroslav-petrovich.php (Y. P. Zamora);
http://www.knu.edu.ua/fakultety/fakul-tet-mehanichnoi-inzhenerii-ta-transportu/dekanat (A. V. Pikilnyak)
0000-0002-0202-348X (I. M. Tsidylo); 0000-0003-0789-0272 (S. O. Semerikov); 0000-0003-3335-0501 (T. I. Gargula);
0000-0002-2527-8653 (H. V. Solonetska); 0000-0001-6470-8233 (Y. P. Zamora); 0000-0003-0898-4756 (A. V. Pikilnyak)
© 2020 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
508
and Grundspenkis [13], Gierłowski and Nowicki [14], Grundspenkis [15], Schmuck et al. [16],
Szöllosi et al. [17] and others); the use of a multi-point scale for assessing knowledge, abilities,
and skills (Bespalko [18], Linn [19] and others); theoretical approaches to the assessment of
students’ knowledge, their development and improvement (Clotfelter et al. [20], Falchikov
and Boud [21], Falchikov and Goldfinch [22], Host et al. [23], Hwang and Chang [24], Newble
and Jaeger [25], Osadchyi et al. [26], Rust et al. [27], Scouller [28], Topping [29], Wiliam et al.
[30] and others); evaluation of test results in an adaptive automated testing system, taking
into account the ambiguity of the formulations of answers (Barker [31], Phankokkruad and
Woraratpanya [32], Rudinskiy [1] and others). In [33] we substantiated the structural model
of the neuro-fuzzy system of professional selection of students for training in IT specialties
by studying the psychological characteristics, personal qualities and factual knowledge, skills
and abilities of students as a unity of fuzzy and stochastic data base of the intellectual system.
The issue of using fuzzy logic to describe the indicators of expert competence assessment using
linguistic variables instead of numerical ones or in addition to them and the development of
Sugeno’s intelligent system for determining expert competence was covered by us in [34]. The
process of modeling intelligent systems based on fuzzy logic in various fields and analysis
of the effectiveness of systems implemented in MATLAB is disclosed in the works of: Taylor
[35] – fuzzy logic methodology, which is widely used in research and engineering practice
and education, Lutsyk et al. [36] – use of parametric identification and adaptive neuro-fuzzy
technologies to determine energy efficient modes of production equipment, Shtovba – the
theory of fuzzy identification, methods of fuzzy clustering and their application for fuzzy rule
extraction, as well as the method of decision-making in fuzzy conditions based on the merger
of goals and constraints, author’s package solutions for designing fuzzy classifiers, building
hierarchical fuzzy systems, training of fuzzy knowledge bases such as Mamdani, as well as for
logical output with fuzzy source data [37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50].
2. Materials and methods
Models based on fuzzy logic are more flexible, as they mostly allow taking into account the
experience and intuition of a specialist in a particular field. They are more adequate to the
simulated reality and make it possible to obtain a solution correlated in accuracy with the initial
data [51].
As a rule, the following characteristics are referred to fuzzy test characteristics:
1) the level of difficulty of the task, which can be assessed as “easy”, “average”, “above average”,
“difficult”;
2) the degree of correctness of the task, which can be assessed as “correct”, “partially correct”,
“rather correct”, “incorrect”;
3) time allotted for the execution of a test task or test, which can be assessed as “short”, “average”,
“long”, “very long”;
4) percentage of correctly completed tasks, which can be assessed as “small”, “medium”, “large”,
“very large”;
5) final mark for the test, which can be assessed as “bad”, “satisfactory”, “good”, “excellent”.
509
Among the fuzzy models for evaluating test results, adaptive models are interesting. In the
work of Rudinskiy [1, p. 49], an adaptive model for evaluating the results of a “fuzzy” test is
described. The idea is that the set of reference answers for each test item has a fuzzy grading
scale. This fuzzy scale corresponds to the normalized numerical scale (1, 𝑡1 , 𝑡2 , 𝑡3 , 0), where
𝑡𝑖 ∈ (0, 1), 𝑖 = 1..3. All answers, except for the correct one, are assigned a subsequent question
with a subset of answers. If an inaccurate answer is given to question 𝐷 at the 𝑖-th step of
testing, a clarifying question is asked next, and the subset of answers contains both “more
correct” (“correct”, “incomplete”) and “less correct” (“uncertain”, “wrong”) answers. If this
question is answered differently from the correct one, no further additional questions are asked
(otherwise the laboriousness of compiling such a structure of questions with subsets of answers
to them would be very great), testing goes to the next step (question). Thus, the testing process
can be represented as a movement along a directed graph, where vertices are questions, and
arcs are transitions from the previous question to the next.
An adaptive testing model using the apparatus of fuzzy logic is considered by Duplik [52,
p. 60]. As a scale for evaluating test results, a 12-point scale proposed by Bespalko [18] is used.
At the same time, the author proposes a correspondence between the percent of correct answers
of the student and estimates on 12-point and 5-point scales, which, in turn, correspond to fuzzy
concepts.
Danilova [53, p. 17] developed an adaptive fuzzy model for evaluating the results of automated
testing with division of tasks according to the levels of assimilation, proposed by Bespalko [18].
The paper presents models for evaluating test results: formalization of question-answer relations
in test tasks according to the levels of assimilation is carried out for recognizing the answers of
the tested person and formal presentation of test results; the scaling of the value estimates of
the test items was performed; the bases of rules of fuzzy productions for evaluating test items
of closed and open types have been developed; in order to ensure the adaptability of testing, a
base of rules for fuzzy products has been developed for ranking tasks in the test; the calculation
of the integral assessment of the test performance was done based on the assessment results
of each test task. The fuzzy inference for evaluating the test results, based on the Mamdani
method of fuzzy inference, is described.
Belov [54] considers the problem of building an automated testing system (ATS) with the
analysis of the respondent’s answers in natural language (NL). To recognize the responses of
the person and the reference in the automated testing system, a linguistic analyzer module
has been developed, which processes text in NL. The result of the surface-syntactic analysis of
the phrases of the reference and user answers are syntactic dependency trees, including the
word forms of the phrase, with the definition for each of them morphological descriptors and
syntactic properties that combine words into syntactic fragments and groups.
A limitation of the presented comparison model is the use of well-formed sentences. A
sentence that is not well-formed is discarded by the linguistic analyzer with the requirement to
the respondent to reformulate the answer. Each type of response is associated with a so-called
syntactic template (SynT), which determines a set of typical syntactic constructions of a sentence
and their significance. The obtained result – the degree of correspondence (relevance of phrases)
– is taken as the degree of “correctness” of the respondent’s answer on the scale [0; 1].
510
3. Results
Thus, all the methods for evaluating test methods that we have considered have both advantages
and disadvantages, which we have summarized in table 1 for clarity.
Table 1
Advantages and disadvantages of test assessment methods
Author Advantages Disadvantages
Rudinskiy [1] The introduction of fuzziness in the orga- When evaluating test tasks and the test,
nization of the adaptive test, which allows the apparatus of fuzzy logic is not used,
the compilers of the test at the stage of and the obtained linguistic values are sim-
its creation for each test task to build a ply projected onto a normalized numerical
hierarchical structure of questions in the scale. The values obtained on this scale
form of a directed graph. determine the degree of correctness of the
answers, which are substituted into a spe-
cially designed formula to obtain the final
grade.
Duplik [52] The use of a fuzzy logic apparatus to ob- The 12-point assessment scale, proposed
tain an integral assessment of test results. by V. P. Bespalko, is used only to evenly
The integral assessment is influenced by distribute the traditional 5-point scale on
such indistinct characteristics of the test it and is not tied to the levels of assimila-
as the current level of training, the per- tion of knowledge.
centage of correct answers, the complex-
ity of the task, and the time it takes to
complete the task.
Danilova [53] The sophistication of models for assess- The set of fuzzy production rules for evalu-
ing test tasks, adaptive testing, integral ating test tasks with an open-ended ques-
assessment of test results. tion is applicable only to test tasks of the
“Substitution” type.
Belov [54] Revealed classification of question types The graph comparison method is very la-
and corresponding types of answers in bor intensive and complex. Firstly, the
natural language. syntactic templates of all reference an-
swers must be built in advance, and sec-
ondly, the proximity of two phrases is de-
termined on the scale [0; 1] by means of a
complex algorithm, which would be easier
to do using the apparatus of fuzzy logic.
The considered methods for assessing students are presented in the general case by two
methods: using fuzzy sets and corresponding membership functions; fuzzy estimation method
and generalized fuzzy estimation method. The assessment system should be regularly reviewed
and improved to ensure its suitability to assess students impartially and fairly.
It makes sense to use a fuzzy model to describe an object when we do not have its analyt-
ical description, or it is too cumbersome to use, but at the same time there is a sufficiently
large amount of experimental data on the behavior of an object and/or heuristic rules for its
functioning.
511
In this work, the Sugeno production model is used as the closest to the natural language. This
closeness allows for closer interaction with a subject area expert and build well-understood,
easily interpreted inference systems.
It is important for us to develop an assessment strategy based on fuzzy sets, which requires
careful consideration of the factors included in the assessment. These include: the level of
difficulty of the task, the degree of correctness of the task, the final mark for the test, which
can be assessed as “bad”, “satisfactory”, “good”, and “excellent”. The system is presented in the
form of a block diagram of fuzzy logical nodes in figure 1 and consists of four input variables,
corresponding to the levels of knowledge assimilation and one initial one. With this method,
the system contains two nodes. The first node takes into account the level of complexity of
the task and the degree of correctness of the task, depending on the supported task type of the
automated system that is used for testing, for example Moodle [55].
The next three nodes behave like a fuzzy logic controller with two inputs with corresponding
weights and one output, as in figure 2.
4. Fuzzy system implementation
From the subject expert, we get the value of the matrix and the dimensions that describe the
degree of importance of each question in the fuzzy domain, that is, the set of all allowed atomic
values of the matrix column. The clear values are given as a vector. In the first node, the
resulting data will be the experimental data, while the next nodes work as a fuzzy controller,
the input of which is the output of the previous node (corresponding to the levels). The output
of each node can be in the form of fuzzy values or in the form of linguistic variables. Each
node has weighted coefficients that can be set equal to one with the equal influence of each
input parameter. The output occurs according to the inference mechanism of the Sugeno fuzzy
system. Here is a description of the system.
System name: Correctness.
Input variables: Level 1, Level 2, Level 3, Level 4.
Initial variable: Final grade.
The names of the terms of input variables: correct, wrong.
The names of the terms of the original variable: correct, almost correct, partly correct, rather
correct, probably wrong, wrong, zero.
Fuzzy membership functions of the system are defined in the interval [0; 100] (see figure 3),
the parameters of the input and initial ones, respectively, are given in tables 2 and 3.
Set of rules “ If . . . then”:
1. If (level1 is wrong) and (level2 is wrong) and (level3 is wrong) and (level4 is wrong) then
(final grade is zero) (1)
2. If (level1 is wrong) and (level2 is wrong) and (level3 is wrong) and (level4 is correct) then
(final grade is probably wrong) (1)
3. If (level1 is wrong) and (level2 is wrong) and (level3 is correct) and (level4 is wrong) then
(final grade is probably wrong) (1)
4. If (level1 is wrong) and (level2 is wrong) and (level3 is correct) and (level4 is correct) then
(final grade is partly correct) (1) . . .
512
Figure 1: Block diagram of a fuzzy estimation system.
14. If (level1 is correct) and (level2 is correct) and (level3 is wrong) and (level4 is correct) then
(final grade is almost correct) (1)
15. If (level1 is correct) and (level2 is correct) and (level3 is correct) and (level4 is wrong) then
(final grade is partly correct) (1)
16. If (level1 is correct) and (level2 is correct) and (level3 is correct) and (level4 is correct)
then (final-grade is correct) (1)
As a result of modeling this system in the MATLAB application package, in particular the
Fuzzy Logic Toolbox package, we obtained the response surfaces of the system at constant
values of the input variables level3 and level4 equal to 50: in figure 4a – manually configured by
an expert; in figure 4b – configured according to the ANFIS algorithm. Analysis of the response
surface of a manually tuned system shows incorrect operation at intervals corresponding to
513
Figure 2: Node of presentation in the form of a fuzzy logical controller.
Figure 3: Membership functions of input linguistic variables of a fuzzy system.
intermediate values of functional membership such as constants of the output variable. To
eliminate these differences, the fuzzy system was trained using the ANFIS algorithm based on
the training sample.
Training program:
514
Table 2
Parameters of membership functions of initial variables
Name Type Parameters
correct constant 100
almost correct constant 80
partly correct constant 65
rather correct constant 50
probably wrong constant 35
wrong constant 20
zero constant 0
Figure 4: The surface of the system response at constant values of the input variables level3 and level4
equal to 50: a – manually configured by an expert; b – configured according to the ANFIS algorithm.
initfis = resdfis (’correctness’);
(learn, error) = anfis (tr_data, initfis, 10);
where the initial parameters: learn – a tuned system of the Sugeno type, the parameters of
which minimize the error on the training set; error – system error at each training iteration;
input parameters: tr_data – training sample; initfis – the original fuzzy output system;
number 10 is responsible for the number of training iterations.
As can be seen from figure 4b, the trained system according to the ANFIS algorithm reproduces
the expert’s opinion as accurately as possible, which makes it possible, accordingly, to more
accurately formulate the final assessment, taking into account the level of the tasks done
correctly. The results of testing the fuzzy system are shown in table 4.
515
Table 3
Fuzzy system testing results
Difficulty level 1 Difficulty level 2 Difficulty level 3 Difficulty level 4 Final Grade
20 25 21 36 26
50 65 75 25 52
60 45 25 0 28
98 78 60 10 52
27 35 15 0 18
60 68 50 35 50
85 90 50 54 66
85 87 80 76 81
15 7 2 0 6
56 45 90 84 73
5. Conclusion
An intelligent system for assessing multilevel test tasks based on fuzzy logic modeled in this
way makes it possible to consider all the above factors using fuzzy logic that are included in
the assessment. This approach ensures the maximum consideration of answers to questions
of all levels of complexity by formulating a base of inference rules and selection of weighting
coefficients when deriving the final grade. The stability of the system is achieved by using
Gaussian membership functions, as discussed in [56, p. 14]. We see the prospect of further
research in the processing of the information received from the fuzzy system and the formulation
of appropriate recommendations for specialists in different fields of knowledge for interpreting
the final grade using multilevel test tasks.
References
[1] I. D. Rudinskiy, Fuzzy knowledge evaluation model as a methodological basis for au-
tomation of pedagogical testing, IEEE Transactions on Education 50 (2007) 68–73.
doi:10.1109/TE.2006.888904.
[2] O. Cherednichenko, O. Yangolenko, Towards quality monitoring and evaluation methodol-
ogy: Higher education case-study, Lecture Notes in Business Information Processing 137
(2013) 120–127. doi:10.1007/978-3-642-38370-0_11.
[3] L. He, W. He, Study on the construction of internal monitoring system of chinese indepen-
dent colleges’ education quality, Genova, 2009, pp. 191–194.
[4] E. Igbape, P. Idogho, N-dimension data visualization spaces for academic programmes
quality monitoring in nigeria higher education, volume 2019-October, Newswood Limited,
2019, pp. 238–242.
[5] M. Leontev, N. Bondarenko, T. Shebzuhova, S. Butko, L. Egorova, Improving the efficiency
of university management: Teacher’s performance monitoring as a tool to promote the
quality of education, European Research Studies Journal 21 (2018) 527–540. doi:10.35808/
ersj/1020.
516
[6] Y. Li, P. Li, F. Zhu, R. Wang, Design of higher education quality monitoring and evaluation
platform based on big data, Institute of Electrical and Electronics Engineers Inc., 2017, pp.
337–342. doi:10.1109/ICCSE.2017.8085513.
[7] N. Muhd Nor, M. Azlan, S. Kiong, F. Mohamad, A. Ismail, A. Kasmin, M. Ahmad,
S. Yokoyama, Development of course management and monitoring system as a qual-
ity tools in engineering education, Applied Mechanics and Materials 465-466 (2014)
395–400. doi:10.4028/www.scientific.net/AMM.465-466.395.
[8] F. Qin, W. Zeng, L. Li, R. Zhao, Construction of big data monitoring platform for teaching
quality under intelligent education, Institute of Electrical and Electronics Engineers Inc.,
2020, pp. 1594–1597. doi:10.1109/IWCMC48107.2020.9148224.
[9] A. Sorour, A. Atkins, C. Stanier, F. Alharbi, Comparative frameworks for monitoring quality
assurance in higher education institutions using business intelligence, Institute of Electrical
and Electronics Engineers Inc., 2020. doi:10.1109/ICCIT-144147971.2020.9213808.
[10] C. Wei, Higher vocational education quality monitoring system, Lecture Notes in Electrical
Engineering 217 LNEE (2013) 55–60. doi:10.1007/978-1-4471-4850-0_8.
[11] Z. Zhi, Z. Nan, Study on the construction of teaching quality monitoring system for the
undergraduate physical education majors, Jilin, 2011, pp. 758–761. doi:10.1109/HHBE.
2011.6028937.
[12] A. Anohina-Naumeca, M. Strautmane, J. Grundspenkis, Development of the scoring
mechanism for the concept map based intelligent knowledge assessment system, volume
471, Sofia, 2010, pp. 376–381. doi:10.1145/1839379.1839446.
[13] A. Anohina-Naumeca, J. Grundspenkis, Evaluating students’ concept maps in the con-
cept map based intelligent knowledge assessment system, Lecture Notes in Computer
Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in
Bioinformatics) 5968 LNCS (2010) 8–15. doi:10.1007/978-3-642-12082-4_2.
[14] K. Gierłowski, K. Nowicki, A novel architecture for e-learning knowledge assessment
systems, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial
Intelligence and Lecture Notes in Bioinformatics) 4823 LNCS (2008) 276–287. doi:10.1007/
978-3-540-78139-4_25.
[15] J. Grundspenkis, Intelligent knowledge assessment systems: Myth or reality,
Frontiers in Artificial Intelligence and Applications 315 (2019) 31–46. doi:10.3233/
978-1-61499-941-6-31.
[16] B. Schmuck, D. Sima, S. Szöllosi, The design space of the implementation of knowledge as-
sessment systems, Sydney, NSW, 2006, pp. 587–593. doi:10.1109/ITHET.2006.339672.
[17] S. Szöllosi, D. Sima, B. Schmuck, The design space of the services of knowledge assessment
systems, Sydney, NSW, 2006, pp. 571–578. doi:10.1109/ITHET.2006.339816.
[18] V. Bespalko, Requirements of educational films for professional and technical education,
Soviet Education 2 (1960) 17–19. doi:10.2753/RES1060-9393020317.
[19] R. Linn, Validating inferences from national assessment of educational progress
achievement-level reporting, Applied Measurement in Education 11 (1998) 23–47.
doi:10.1207/s15324818ame1101_2.
[20] C. Clotfelter, H. Ladd, J. Vigdor, Teacher-student matching and the assessment of teacher
effectiveness, Journal of Human Resources 41 (2006) 778–820. doi:10.3368/jhr.xli.4.
778.
517
[21] N. Falchikov, D. Boud, Student self-assessment in higher education: A meta-analysis,
Review of Educational Research 59 (1989) 395–430. doi:10.3102/00346543059004395.
[22] N. Falchikov, J. Goldfinch, Student peer assessment in higher education: A meta-analysis
comparing peer and teacher marks, Review of Educational Research 70 (2000) 287–322.
doi:10.3102/00346543070003287.
[23] M. Host, B. Regnell, C. Wohlin, Using students as subjects - a comparative study of students
and professionals in lead-time impact assessment, Empirical Software Engineering 5 (2000)
201–214. doi:10.1023/A:1026586415054.
[24] G.-J. Hwang, H.-F. Chang, A formative assessment-based mobile learning approach to
improving the learning attitudes and achievements of students, Computers and Education
56 (2011) 1023–1031. doi:10.1016/j.compedu.2010.12.002.
[25] D. Newble, K. Jaeger, The effect of assessments and examinations on the learning of
medical students, Medical Education 17 (1983) 165–171. doi:10.1111/j.1365-2923.
1983.tb00657.x.
[26] V. V. Osadchyi, K. P. Osadcha, H. B. Varina, S. V. Shevchenko, I. S. Bulakh, Specific features
of the use of augmented reality technologies in the process of the development of cognitive
component of future professionals’ mental capacity, Journal of Physics: Conference Series
(2021).
[27] C. Rust, M. Price, O. Berry, Improving students’ learning by developing their understanding
of assessment criteria and processes, Assessment and Evaluation in Higher Education 28
(2003) 147–164. doi:10.1080/02602930301671.
[28] K. Scouller, The influence of assessment method on students’ learning approaches: Multiple
choice question examination versus assignment essay, Higher Education 35 (1998) 453–472.
doi:10.1023/A:1003196224280.
[29] K. Topping, Peer assessment between students in colleges and universities, Review of
Educational Research 68 (1998) 249–276. doi:10.3102/00346543068003249.
[30] D. Wiliam, C. Lee, C. Harrison, P. Black, Teachers developing assessment for learning:
Impact on student achievement, Assessment in Education: Principles, Policy and Practice
11 (2004) 49–65. doi:10.1080/0969594042000208994.
[31] T. Barker, An automated feedback system based on adaptive testing: Extending the model,
International Journal of Emerging Technologies in Learning 5 (2010) 11–14. doi:10.3991/
ijet.v5i2.1235.
[32] M. Phankokkruad, K. Woraratpanya, An automated decision system for computer adaptive
testing using genetic algorithms, Phuket, 2008, pp. 655–660. doi:10.1109/SNPD.2008.
118.
[33] B. Buyak, I. Tsidylo, V. Repskyi, V. Lyalyuk, Stages of conceptualization and formalization
in the design of the model of the neuro-fuzzy expert system of professional selection of
pupils, CEUR Workshop Proceedings 2257 (2018) 112–121.
[34] H. V. Tereshchuk, I. M. Tsidylo, Automated system of fuzzy identification of expert’s
competence for assessing the quality of pedagogical phenomena and processes, Information
Technologies and Learning Tools 64 (2018) 234–244. URL: https://journal.iitta.gov.ua/index.
php/itlt/article/view/2079. doi:10.33407/itlt.v64i2.2079.
[35] A. Taylor, Fuzzy Logic With Matlab: Analyzing, Designing, and Simulating Systems,
CreateSpace Independent Publishing Platform, 2017.
518
[36] I. Lutsyk, Y. Franko, V. Rak, I. Lutsyk, R. Leshchii, O. Potapchuk, Mathematical modeling of
energy-efficient active ventilation modes of granary, in: 2019 9th International Conference
on Advanced Computer Information Technologies (ACIT), 2019, pp. 105–108. doi:10.
1109/ACITT.2019.8780109.
[37] S. Shtovba, O. Pankevich, A. Nagorna, Analyzing the criteria for fuzzy classifier learn-
ing, Automatic Control and Computer Sciences 49 (2015) 123–132. doi:10.3103/
S0146411615030098.
[38] A. Rotshtein, S. Shtovba, Predicting the reliability of algorithmic processes with fuzzy input
data, Cybernetics and Systems Analysis 34 (1998) 545–552. doi:10.1007/BF02666999.
[39] A. Rotshtein, S. Shtovba, Fuzzy multicriteria analysis of variants with the use of paired
comparisons, Journal of Computer and Systems Sciences International 40 (2001) 499–503.
[40] A. Rótshtein, S. Stovba, Managing a dynamic system by means of a fuzzy knowledge base,
Automatic Control and Computer Sciences 35 (2001) 16–22.
[41] A. Rotshtein, S. Shtovba, I. Mostav, Fuzzy rule based innovation projects estimation,
volume 1, Vancouver, BC, 2001, pp. 122–126.
[42] A. Rotshtejn, S. Shtovba, Influence of methods of defuzzification on speed of tuning the
fuzzy model, Kibernetika i Sistemnyj Analiz (2002) 169–176.
[43] A. Rotshtejn, S. Shtovba, Fuzzy rule based control of a dynamic system, Avtomatika i
Vychislitel’naya Tekhnika (2001) 23–31.
[44] A. Rotshteina, S. Shtovbab, Identification of a nonlinear dependence by a fuzzy knowl-
edgebase in the case of a fuzzy training set, Cybernetics and Systems Analysis 42 (2006)
176–182. doi:10.1007/s10559-006-0051-1.
[45] S. Shtovba, Fuzzy identification on the basis of regression models of parametric membership
function, Journal of Automation and Information Sciences 38 (2006) 36–44. doi:10.1615/
JAutomatInfScien.v38.i11.40.
[46] S. Shtovba, E. Shtovba, Prediction of competitive position of brand product by fuzzy
knowledge base, Journal of Automation and Information Sciences 38 (2006) 69–80. doi:10.
1615/JAutomatInfScien.v38.i8.70.
[47] S. Shtovba, Ensuring accuracy and transparency of Mamdani fuzzy model in learning
by experimental data, Journal of Automation and Information Sciences 39 (2007) 39–52.
doi:10.1615/JAutomatInfScien.v39.i8.50.
[48] S. Shtovba, Fuzzy model tuning based on a training set with fuzzy model output values, Cy-
bernetics and Systems Analysis 43 (2007) 334–340. doi:10.1007/s10559-007-0054-6.
[49] A. Rotshtein, S. Shtovba, Modeling of the human operator reliability with the aid of
the Sugeno fuzzy knowledge base, Automation and Remote Control 70 (2009) 163–169.
doi:10.1134/S0005117909010123.
[50] S. Shtovba, O. Pankevych, Fuzzy technology-based cause detection of structural cracks of
stone buildings, volume 2105, CEUR-WS, 2018, pp. 209–218.
[51] I. Khvostina, V. Oliinyk, S. Semerikov, V. Solovieva, V. Yatsenko, O. Kohut-Ferens, Hazards
and risks in assessing the impact of oil and gas companies on the environment, IOP
Conference Series: Earth and Environmental Science 628 (2021) 012027. doi:10.1088/
1755-1315/628/1/012027.
[52] S. V. Duplik, Model of adaptive testing on fuzzy mathematics, Computer science and
education (2004) 57–65.
519
[53] S. D. Danilova, Evaluation of test results in an adaptive automated testing system, Bulletin
of VSSTU (2008) 12–20.
[54] E. A. Belov, Development of a method and algorithms for testing knowledge based on
intelligent processing of answers in natural language, Ph.D. thesis, Bryansk, 2006.
[55] P. Nechypurenko, S. Semerikov, VlabEmbed - the new plugin Moodle for the chemistry
education, CEUR Workshop Proceedings 1844 (2017) 319–326.
[56] I. A. Hameed, C. G. Sorensen, Fuzzy systems in education: A more reliable
system for student evaluation, in: A. T. Azar (Ed.), Fuzzy Systems, Inte-
chOpen, 2010, pp. 1–17. URL: https://www.intechopen.com/books/fuzzy-systems/
fuzzy-systems-in-education-a-more-reliable-system-for-student-evaluation.
doi:10.5772/7216.
520