The evaluation of educational quality is a complex and multifaceted endeavour, often involving the assessment of non-numerical characteristics that are challenging to formalise. While certain aspects of higher education institutions, such as the number of computers, students, or the area of educational premises, are quantifiable, the evaluation of educational programmes and institutional performance is typically conducted using qualitative criteria. In the context of self-assessment and accreditation processes, institutions and expert reviewers are required to assess compliance with established criteria using a four-level scale: A, B, E, and F.
Consequently, there is a pressing need for the development of methods that enable the quantitative description of processes and subjects related to the assessment of educational programme quality and institutional performance. The concept of educational quality is of particular significance, as it represents a comprehensive indicator that reflects both the outcomes of an educational institution and its alignment with societal needs and expectations in terms of individual competency development. The application of quantitative evaluation methods for educational programmes and institutional activities can empower higher education institutions to identify existing deficiencies and potential issues, providing an opportunity to address them proactively before accreditation examinations.
However, assessing the quality of educational programmes and institutional performance is complicated by the fact that the value of this indicator is influenced by numerous factors, potentially with an unknown nature of influence. Moreover, the "product" of education -a graduate of an educational institution -should be considered as a complex system. While various methods and algorithms exist for assessing the quality of educational activities, this study proposes a novel approach based on the neuro-fuzzy paradigm, leveraging the rapid advancements in artificial intelligence-based analytical systems. Among the most well-established and effective AI technologies are neural networks, which have demonstrated success in addressing a wide range of "fuzzy" tasks, such as prediction, classification, handwritten text recognition, language processing, image analysis [1,2,3], and often serve as the sole effective solution in scenarios where traditional technologies are inadequate. In this work, artificial neural networks are employed to tackle the challenge of evaluating the quality of educational programmes and institutional performance.
A prerequisite for accreditation is the compliance of educational programmes and institutional activities with legally established criteria. Specifically, the forms and methods of teaching should effectively contribute to the attainment of the stated objectives of the educational programme and the intended learning outcomes.
Given that educational programmes and institutional activities must adhere to the principles of student-centredness and academic freedom, the hypothesis of this study posits that a sample of current students and recent graduates can provide an adequate comprehensive assessment of the quality of educational programmes and institutional performance.
The intelligent processing of data using neural networks enables the generation of probabilistic forecasts of future accreditation examination results in higher education institutions, which can facilitate the improvement of measures aimed at enhancing educational programmes. These predictive insights can serve as informative and advisory resources for faculty and department leaders. Furthermore, educational programme coordinators can leverage these forecasts to plan activities and individualised work with educators to positively influence the predicted outcomes. The analysis of the obtained data can also reveal weaknesses in the educational process, providing opportunities for modernisation.
In light of these considerations, this article aims to substantiate, develop, and implement a mathematical model for the comprehensive assessment of educational programme quality and institutional performance based on neuro-fuzzy approaches.
The assessment of educational activity quality based on well-defined criteria and methodologies is a crucial aspect of the accreditation process for educational programmes in Ukraine. During the preparation for accreditation and the compilation of materials for self-assessment, institutions often encounter challenges in determining the objectivity of their self-evaluation and identifying potential problems and shortcomings in their educational activities. To address this issue, there is an urgent need for mathematical tools that can assist higher education managers in evaluating the quality of the educational services they provide.
The shift in educational philosophy and practice has led to a heightened focus on student learning outcomes. The educational process should be results-oriented, emphasising what students actually know and can do. Consequently, student-centred learning has emerged as an approach in which students influence the content, activities, materials, and pace of their learning, placing them at the heart of the learning process [4].
European Union initiatives emphasise the importance of increasing the efficiency, international attractiveness, and competitiveness of higher education institutions. Wächter et al. [5] examines various approaches to quality, quality assurance, and ratings, analysing recent research and providing recommendations and policy options for parliament from a comparative perspective.
The challenge of identifying a set of effective indicators that are easily measurable and applicable to diverse institutions, from large public universities to small regional private colleges, and from university programmes to alternative programmes, is also relevant in the United States [6].
Cherniak et al. [7] explored the possibility of assessing the quality of qualimetry objects using a graphanalytical method, applying the principle of determining the area and volume under curved surfaces, both in the plane and in space, created by combining estimates of individual quality indicators on a dimensionless scale. The research demonstrates that mathematical dependencies are typically nonlinear, and their investigation involves the development of universal methods applicable to qualimetry objects, regardless of their nature, complexity, or importance. By representing unit quality indicators on a single (dimensionless) rating scale, the authors propose determining a single comprehensive quality indicator for a qualimetry object using the integration method, which takes into account the evaluation of unit quality indicators.
Parvu and Ipate [8] propose a mathematical model based on a set of indicators adapted to the globally recognised classification structure of intellectual capital, namely the external structure, internal structure, and employee competence. The Rompedet method, an original product of the Romanian school of management [9], is employed as a mathematical calculation tool.
When assessing the quality of education, we encounter a vast array of criteria, each potentially consisting of numerous sub-criteria. Consequently, the task of evaluating educational quality in its mathematical formulation is inherently multi-criteria. Problem situations modelled and described by linear models that depend on multiple factors play a significant role, and solving multi-criteria decision-making problems often involves solving multi-criteria linear programming problems, also known as vector optimisation problems.
Considering these challenges, mathematical models of integrated quality assessment using methods based on the convolution of criteria were also of interest for this study. Models and methods of multicriteria optimisation are discussed in [10], particularly the method of additive convolution of criteria and the method of multiplicative and minimax convolution of criteria. The method of multiplicative convolution of partial criteria to a single generalised indicator, which utilises the maximum (minimum) values of partial criteria as a normalised divisor, is considered in [11]. Chervak [12] employs one of the methods for solving the Paretian multi-criteria optimisation problem as a mathematical tool in the decision-making process. To organise selection problems on the same admissible set of alternatives, the concept of a super criterion for any criterion is introduced; if one criterion is a super criterion of another on a given set, the latter is a sub-criterion of the former. The solution of the multi-criteria selection problem by Paretian convolution is shown to be reducible to the solution of scalar or lexicographic optimisation problems.
The theory of artificial neural networks and deep learning models is explored in fundamental works [13,14,15], while system design based on the neuro-fuzzy approach is discussed in [16,17,18,19,20].
Lesinski et al. [21] consider the use of neural networks to classify the status of higher education graduates based on selected academic, demographic, and other indicators. A multi-layer neural network with feedback is employed as a model, trained on over 5,000 records from entrance exams and university databases. The nine input variables consisted of categorical and numerical data containing information about high school education, test results, high school teacher assessments, parental assessments, and more. Based on these inputs, the multi-layer neural network predicted the success of university entrants, achieving a classification accuracy exceeding 95%. Black et al. [4], examining the relationship between quality and high school student success in college, found no convincing evidence that exposure characteristics of high school diminish over time in teaching students.
To address the issue of determining the quality of educational training, Mahapatra and Khan [22] developed the EduQUAL methodology and proposed an integrative approach using neural networks to assess educational quality. Four neural network models based on a feedback algorithm are used to predict educational quality for different stakeholders, with the P-E Gap model identified as the best model for all stakeholders.
The need to introduce neural network technology in educational courses is highlighted by Semerikov et al. [23]. Educational neural networks are often used for forecasting purposes. For example, students must choose courses of interest for the upcoming semester, but due to limitations such as insufficient resources and the overhead of offering multiple courses, some universities may not be able to teach all courses selected by students. Universities need to know each student's course requirements for optimal course planning each semester. Kardan et al. [24] used a neural network to model student choice behaviour and apply the resulting function to predict the final enrolment of students for each course, demonstrating high prediction accuracy based on experimental data. Arsad et al. [25], Osadchyi et al. [26], and Okubo et al. [27] prove that the use of neural networks in predicting educational processes allows obtaining results with significantly higher accuracy and in less time. According to Abu Naser et al. [28], an artificial neural network can correctly predict the success of more than 80% of future students. Chaban and Kukhtiak [29] analyse the problem of the social system consisting of many higher education students and teachers to create effective "teacher-student" learning pairs, using elements of artificial intelligence theory based on artificial neural networks to form these learning pairs. Okubo et al. [30] propose the use of a recurrent neural network (RNN) to predict students' final grades using journal data stored in educational systems.
Liu et al. [31] propose a method for assessing the quality of postgraduate education preparation based on the neural network backpropagation algorithm and stress testing. This method creates a publicly available list of indicators consisting of 19 criteria in 4 groups: attitudes towards teaching, teaching content, teaching approach, and key teacher characteristics. After the neural network algorithm determines the optimal parameters of the evaluation model, a sensitivity test identifies indicators that significantly impact educational quality. Additionally, scenario analysis is used to study the impact of educational quality in predefined situations, providing theoretical and empirical support for assessing the quality of postgraduate teaching, improving educational quality, and fostering teachers' professional growth.
Educational institutions continuously strive to improve their services, aiming to have the best teaching staff, enhance teaching quality, and boost students' academic success. Understanding the factors influencing student learning can help universities and learning centres adapt their curricula and teaching methods to meet students' needs. One of the first measures taken by educational institutions in response to the COVID-19 pandemic was the creation of virtual learning environments [32]. To understand the factors influencing the university learning process in virtual learning environments, Rivas et al. [33] applied several automatic learning methods, including tree-like models and various types of artificial neural networks, to publicly available datasets.
The availability of educational data supported by learning platforms [34] provides opportunities to study student behaviour and solve problems in higher education, optimise the educational environment, and ensure decision-making using artificial neural networks [35].
Cader [36] uses a deep neural network to assess students' acquisition of knowledge and skills, noting that the relatively small amount of available assessment data required for neural network training is an obstacle to the application of the method in teaching. A new data augmentation method -asynchronous data augmentation through pre-categorisation -is proposed to address this problem, enabling neural network training even for small datasets. Do and Chen [37] present a neuro-fuzzy classifier that uses the results of previous exams and other related factors as input variables to classify students based on their expected learning outcomes. The results showed that the proposed approach achieved high accuracy compared to other known classification approaches, such as Naive Bayes and neural networks.
Fazlollahtabar and Mahdavi [38] proposed a neuro-fuzzy approach based on evolutionary techniques to obtain the optimal learning pathway for both teachers and students. The neuro-fuzzy approach provides recommendations to teachers for making pedagogical decisions based on students' learning styles, while the neural network approach is used for students to create personalised curriculum profiles based on their individual needs in a fuzzy environment.
Taylan and Karagözoğlu [39] use a systematic approach to designing a fuzzy inference system based on a class of neural networks to assess student achievement. The developed method uses a fuzzy system, supplemented by neural networks, to enhance characteristics such as flexibility, speed, and adaptability, referred to as the adaptive neuro-fuzzy inference system (ANFIS). The results of the ANFIS model are as reliable as statistical methods but encourage a more natural way of interpreting student learning outcomes.
In comparison with these works, this study fills a gap in the methods of comprehensive assessment of educational programme quality and institutional performance based on a neuro-fuzzy approach.
This study employed methods of mathematical modelling and computational experimentation based on the statistical processing of data assessments of educational programme quality and institutional performance. The essence of the mathematical modelling methodology is to replace the original object with its mathematical model and study it using computer technology. The processing, analysis, and interpretation of calculation results were carried out through constant comparison with the results of statistical processing of expert estimates. Throughout the research, refinements were made, the mathematical model was revised, and the cycle of the computational experiment was repeated.
The methodology for assessing the quality of the curriculum and educational activities is built using artificial intelligence methods and tools, implemented in the Fuzzy Logic Toolbox of the MATLAB system in the form of an adaptive neuro-fuzzy output (ANFIS).
A fuzzy inference system can be represented as a neuro-fuzzy network -a special type of direct signal propagation neural network, or ANFIS model. The architecture of a neuro-fuzzy network is isomorphic to a fuzzy knowledge base. Neuro-fuzzy networks use differentiated implementations of triangular norms (multiplication and probabilistic OR) and smooth membership functions. This enables the use of fast training algorithms for neural networks based on the backpropagation method to tune neuro-fuzzy networks.
ANFIS implements the Sugeno fuzzy inference system through a five-layer feed-forward neural network. The purpose of each network layer is as follows:
• First layer -terms of input variables;
• Second layer -antecedents (parcels) of fuzzy rules;
• Third layer -normalisation of the degree of implementation of the rules;
• Fourth layer -conclusion of the rules;
• Fifth layer -aggregation of the result obtained according to different rules.
The network inputs are not allocated to a separate layer. Figure 1 shows an ANFIS network with two input variables (𝑥 1 and 𝑥 2 ) and four fuzzy rules. Three terms are used for the linguistic evaluation of the input variable, and two terms for the variable. We will use the following notation:
• 𝑎 𝑖,𝑟 -fuzzy term with a membership function 𝜇 𝑟 (𝑥 𝑖 ) used for linguistic evaluation of a variable 𝑥 𝑖 in the 𝑟-th rule (𝑟 = 1, 𝑚, 𝑖 = 1, 𝑛); • 𝑏 𝑞,𝑟 are the conclusion coefficients of the 𝑟-th rule (𝑟 = 1, 𝑚, 𝑞 = 0, 𝑛).
The ANFIS network operates as follows.
Layer 1. Each node of the first layer represents one term with a bell membership function. The network inputs are connected only to their terms. The number of nodes in the first layer is equal to the sum of the cardinalities of the term set of input variables. The degree of belonging of the value of the input variable to the corresponding fuzzy term is fed to the output of the node:
where 𝑎, 𝑏 and 𝑐 are membership function parameters that can be tuned. Layer 2. The number of nodes in the second layer is 𝑚. Each node of this layer corresponds to one fuzzy rule. The node of the second layer is connected to the nodes of the first layer, which form the antecedents of the corresponding rule. Therefore, each node of the second layer can receive from 1 to 𝑛 signals. The output of the node is the degree of execution of the rule, calculated as the product of the input signals. Let us denote the outputs of the nodes of this layer as 𝜏 𝑟 , 𝑟 = 1, 𝑚. Layer 3. The number of nodes in the third layer is also 𝑚. Each node of this layer calculates the relative level of execution of the fuzzy rule according to the formula:
Layer 4. The number of nodes in the fourth layer is also 𝑚. Each node is connected to one node of the third layer, as well as to all inputs of the network (figure 1 connections to the inputs are not shown). The node of the fourth layer calculates the contribution of one fuzzy rule to the network output by the formula:
Layer 5. A single node of this layer sums up the contributions of all rules:
Typical neural network training procedures can be applied to tune an ANFIS network, as it uses only differentiated features. A combination of gradient descent as a backpropagation algorithm and the least-squares method is commonly used. The error backpropagation algorithm regulates the parameters of rule antecedents, i.e., membership functions. The least-squares method evaluates the rule inference coefficients since they are linearly related to the network output.
Each iteration of the tuning procedure is performed in two steps.
In the first stage, a training sample is fed to the inputs, and based on the discrepancy between the desired and actual behaviour of the network, the optimal parameters of the nodes of the fourth layer are determined using the least-squares method.
In the second stage, the residual mismatch is transmitted from the network output to the inputs, and the parameters of the nodes of the first layer are modified by the backpropagation of the error. At the same time, the rule inference coefficients found at the previous stage do not change. The iterative tuning procedure continues as long as the mismatch exceeds a predetermined value. To tune the membership functions, in addition to the error backpropagation method, other optimisation algorithms can be used, such as the Levenberg-Marquardt method.
The ANFIS editor in MATLAB allows the automatic synthesis of a neuro-fuzzy network from experimental data. In this case, the accessories of the synthesised systems are tuned (trained) in such a way as to minimise the deviations between the results of fuzzy modelling and experimental data. The ANFIS editor is loaded using the anfisedit command.
The ANFIS editor contains 3 top menus -File, Edit and View, a visualisation area, ANFIS properties area, data loading area, source fuzzy inference system generation area, training area, testing area, current information output area, as well as Help and Close buttons, which allow calling the help window and closing the ANFIS editor, respectively.
Participants in the experiment were 22 full-time master's students and 32 graduates of higher education institutions of the previous term studying the same specialities -a total of 54 people. This number of respondents is due to the number of indicators of quality criteria because the data format of the artificial network in MATLAB supports square matrices, in this case, 54x54. Before the accreditation examination, students were offered questionnaires with a proposal to assess the quality of the educational programme and educational activities of the speciality on an assessment scale covering four levels: F, E, B, A. Student assessments were used to form the vector of artificial neural network inputs. After the accreditation examination, the expert assessments were used to check the quality of the prediction of the artificial neural network.
The experience of European countries demonstrates the expediency of involving students in accreditation examinations. For example, the Polish Accreditation Commission consists of 80-90 members appointed by the Minister of Science and Higher Education among the candidates nominated by the Senates of higher education institutions, the conferences of rectors of scientific schools and universities in Poland, and the Parliament of Students of Poland (the President of the Student Parliament is a member of the Polish Accreditation Commission). In Slovakia, the Academic Ranking and Rating Agency is a civic association founded in 2004 on the initiative of former student leaders and academics. The Slovenian Quality Assurance Agency for Higher Education SQAA-NAKVIS appoints at least three members of each expert group, of which at least one foreign expert, an expert in the field of quality assessment of higher education, and one representative from among students [40].
To ensure the representativeness of the sample, the study of its design was carried out based on randomisation. The decision on the statistical deviation of the null hypothesis regarding the differences between the averages was also associated with the procedure of random sampling.
The rating scale covers four levels of compliance with the requirements of the legislation (F, E, B, A) [41]. The legislation also establishes 10 criteria for assessing the quality of the educational programme [41]:
1) Design and objectives of the educational programme (4); 2) Structure and content of the educational programme (9); 3) Access to the educational programme and recognition of learning outcomes (4); 4) Teaching and learning according to the educational programme (5); 5) Control measures, evaluation of applicants for higher education and academic integrity (4); 6) Human resources (6); 7) Educational environment and material resources (6); 8) Internal quality assurance of the educational programme (7); 9) Transparency and publicity (3); 10) Learning through research (6).
In turn, each of these criteria has from 3 to 9 indicators (the number is indicated in parentheses). Together, all 10 criteria contain 54 indicators.
In the first stage of the study, data on the results of the assessment of students and graduates of higher education regarding educational programmes and educational activities for each criterion were collected and statistically processed.
In the second stage, a computational experiment was performed. The cycle of the computational experiment was carried out in several stages: 1) Choice of approximation and mathematical formulation of the problem (construction of a mathematical model of the phenomenon under study); 2) Development of a computational algorithm for solving the problem; 3) Implementation of the algorithm in the form of a PC program; 4) Calculations on the PC; 5) Processing, analysis and interpretation of calculation results, comparison with the results of statistical processing of expert estimates and, if necessary, refinement or revision of the mathematical model, i.e., return to the first stage and repeat the cycle of the computational experiment.
Assessing the quality of the curriculum and learning activities is complicated by the fact that each of the 10 criteria, in turn, consists of several indicators (3)(4)(5)(6)(7)(8)(9) and is due to many factors, possibly with an unknown nature of influence, which is also non-numerical. To assess the quality of the curriculum and training activities, a two-tier system based on the ANFIS package and artificial neural networks is proposed to predict assessment scores.
The ANFIS hybrid system is a combination of the Sugeno neuro-fuzzy inference method with the ability to train a five-layer artificial neural network (ANN) of direct propagation with a single output and multiple inputs, which are fuzzy linguistic variables. As input variables of the ANFIS system, we use the criteria for evaluating the quality of the educational programme of 10 groups of factors 𝑉 𝑖 (𝑖 = 1, ..., 10).
The output variable of the ANFIS system is a numerical assessment of the quality of the curriculum and training activities and is defined as a function 𝑦 = 𝑓 (𝑉 1 , 𝑉 2 , 𝑉 3 , 𝑉 4 , 𝑉 5 , 𝑉 6 , 𝑉 7 , 𝑉 8 , 𝑉 9 , 𝑉 1 0).
Layer 1 of the ANFIS system for the linguistic evaluation of input parameters uses the term set of all possible values of the linguistic variable. 𝐴 𝑉 𝑖 = {"𝐹 ", "𝐸", "𝐵", "𝐴"}. In symbolic form we write: 𝐴 𝑉 𝑖 = {𝐹 < 𝑖 >, 𝐸 < 𝑖 >, 𝐵 < 𝑖 >, 𝐴 < 𝑖 >}. The term set of the original linguistic variable y is the set of values of quality assessments of the curriculum and educational activities: 𝑇 𝑦 = {𝐹, 𝐸, 𝐵, 𝐴}. The outputs of the nodes of layer 1 are the values of the membership functions at specific values of the input variables.
Layer 2 is non-adaptive and defines the preconditions of fuzzy production rules. Production rules -a form of representation of human knowledge in the form of a sentence type -if (condition), then (action). The rules provide a formal way to present recommendations, guidance, or strategies. They are ideal in cases where the knowledge of the subject area arises from the empirical associations accumulated during the work on solving problems in a particular field.
Each node of this layer is connected to those nodes of layer 1, which form the prerequisites of the corresponding rule. To solve this problem, four fuzzy production rules are formulated: 𝑃 = {𝑝 1 , 𝑝 2 , 𝑝 3 , 𝑝 4 }, because according to the features of the ANFIS network, the number of network rules must correspond to the dimension of the term set of the source variable 𝑦.
Nodes perform a fuzzy logical operation "I" (min). The outputs of the nodes of this layer are the degree of truth (fulfilment) of the preconditions of each of the four fuzzy production rules, which are calculated by the formulas:
.
(
Layer 3 normalises the degree of implementation of each of the fuzzy production rules (calculation of the relative degree of implementation of the rules) as follows:
where ℎ = 1, ..., 4 is the production rule number. Layer 4 calculates the contribution of each fuzzy production rule to the output of the network according to the formula.
ℎ 𝑉 1 + 𝑣
ℎ 𝑉 2 + 𝑣
(3)
where 𝑣
ℎ -coefficients of the initial function (𝑖 = 0, ..., 5). Layer 5 summarises the contributions of all the rules:
Training of the ANFIS network was carried out for 24 epochs by a hybrid method. During training, the type of membership functions, the type of initial function, and their coefficients are selected. As a result of training a fuzzy network for four rules, Gaussian functions were adopted as membership functions, and a linear function was adopted as the initial function. As a result of training, membership functions and their coefficients were also obtained.
To assess each of the 10 groups of factors that affect the quality of the curriculum and educational activities by the evaluation criteria, 10 modules are used, which are implemented using artificial neural networks. Thus, it is necessary to design neural networks, a mathematical model of a comprehensive assessment of the quality of the educational program and educational activities based on the methods of the neuro-fuzzy approach. For this purpose, the Neural Network Toolbox was used. To form neural networks, it is necessary to determine their topology, learning mechanism, and testing procedure. Also, the training of an artificial neural network requires input data -a sample of answers of students and graduates with reliable quality indicators, determined based on these criteria.
A standard 𝐿-layer feedforward neural network consists of a layer of input nodes (we will stick to the position that it is not contained in the network as an independent layer), (𝐿 − 1) hidden layers, and an output layer that is connected in series in the forward direction and does not contain a connection between elements within a layer and feedback between layers. The most popular class of multilayer feed-forward networks is formed by multilayer perceptrons, where each computational element uses a limit or sigmoidal activation function. A multilayer perceptron can form arbitrarily complex decision limits and implement arbitrary Boolean functions. The development of a backpropagation algorithm for determining weights in a multilayer perceptron has made these networks the most popular among researchers and users of neural networks. The vast majority of programs involve the use of such multilayer perceptrons. Networks consisting of successive layers of neurons are more commonly used. Although any network without feedback can be represented as successive layers, the presence of many neurons in each layer can significantly speed up calculations using matrix accelerators.
The popularity of perceptrons is due to a wide range of available tasks that can be solved with their help. In the general case, they solve the problem of approximating multidimensional functions, that is, constructing a multidimensional mapping 𝐹 : 𝑥 ⇒ 𝑦 that generalizes a given set of parameters {𝑥 𝛼 ⇒ 𝑦 𝛼 }.
Depending on the type of output variables (the type of input variables is not critical), the approximation of functions can take the form of classification (discrete set of initial values), or regression (continuous initial values).
Many practical problems of pattern recognition, noise filtering, time series prediction, etc. come down to basic settings. The reason for the popularity of perceptrons is that, for their range of tasks, they are, firstly, universal, and secondly, they are efficient in terms of the computational complexity of devices.
As a result of the development of neurocomputing, a large number of efficient models of neural networks have been created, focused on solving various problems. Due to this, artificial neural networks are successfully used to solve a wide class of practical problems. Therefore, when solving a specific problem, it is necessary to solve the issue of choosing the most appropriate neural network model, its parameters, and the training method.
Typically, a network consists of many sensor elements (input nodes or source nodes) that form an input layer; one or more hidden layers of computational neurons, and one output layer of neurons. The input signal propagates through the network in a forward direction from layer to layer. Such networks are usually called multilayer perceptrons. They are a generalization of a single layer perceptron.
Multilayer perceptrons are successfully used to solve various problems. At the same time, supervised learning is performed using such a popular algorithm as the error back-propagation algorithm. This method consists of error correction (error-correction learning rule). It can be thought of as a generalization of the equally popular adaptive filtering algorithm, the mean squared error minimization (LMS) algorithm.
Multilayer perceptrons have three characteristic features.
1. Each neuron of the network has a non-linear activation function. It should be noted that this non-linear function is smooth (that is, differentiated everywhere), in contrast to the hard threshold function used in the Rosenblatt perceptron. The most popular form of a function that satisfies this requirement is the sigmoidal nonlinearity, defined by the logistic function
where 𝑣 𝑗 is the induced local field (i.e., the weighted sum of all synaptic inputs plus the limit value) of neuron 𝑗; 𝑦 𝑗 is the output of the neuron. The presence of non-linearity plays a very important role, since otherwise the "input-output" mapping of the network can be reduced to a conventional single-layer perceptron. Moreover, the use of the logistic function is biologically motivated, since it takes into account the recovery phase of a real neuron. 2. The network contains one or more layers of hidden neurons that are not part of the input or output of the network. These neurons allow the network to learn how to solve complex problems by sequentially extracting the most important features of the input image (vector).
The network has a high degree of connectivity (connectivity), implemented using synaptic connections. Changing the level of network connectivity requires changing the plurality of synaptic connections or their weights.
The combination of all these properties, along with learning-by-doing, provides the computational power of a multilayer perceptron. However, these same qualities are the reason for the incompleteness of modern knowledge about the behaviour of such networks. First, the distributed form of nonlinearity and the high connectivity of the network significantly complicate the theoretical analysis of a multilayer perceptron. Second, the presence of hidden neurons makes the learning process more difficult to visualise. It is in the learning process that it is necessary to determine which signs of the input signal should be given by hidden neurons. Then the learning process becomes even more difficult, since the search must be performed in a wide range of possible functions, and the choice must be made among alternative representations of the input images.
The emergence of the backpropagation algorithm was a landmark event in the development of neural networks, since it implements a computationally efficient method for training a multilayer perceptron. The backpropagation algorithm does not offer a truly optimal solution to all potential problems, but it is most effective in learning multilayer machines.
An artificial neural network for the analysis of indicators of the quality of the educational program and educational activities will have the number of input neurons (according to the number of indicators for all criteria) 54; output neurons -54. Input signals were determined based on students' assessments of each indicator of this quality criterion, while the scale F, E, B, A were translated into numerical 1; 2; 3; 4 respectively. Part of the data is given in table 1.
Input signals (T) based on students' assessments of quality criteria.
Student grades quality criteria 1 2 3 4 5 6 7 ... It is important that the neural network can predict expert assessments if student and graduate assessments are to be ranked in ascending order based on the determination of the grade point average. According to the hypothesis, we assume that students with higher academic performance are better acquainted with the goals, structure, and content of the educational program, the process and characteristics of teaching and learning according to the educational program, control measures, assessment system, and all other aspects of educational activities. assessments of the quality of the educational program and educational activities will be more objective.
The Neural Network Toolbox application package Matlab Mathematical Modeling Environment (version R2014a) was used in the work. After starting the Matlab system, enter the nntool command on the command line, which opens the window for entering data and creating a neural network (Neural Network / Data Manager) (figure 2). To create a new network, we chose New, to view the data you need to select Import. The data is contained in the P.mat file. This file is a matrix of two lines, in which the numbers 1-54 are indicators of quality criteria, and 55-108 -are the evaluation of students and graduates on the indicators of quality criteria. Its contents are stored in the P.txt file. The next step is to import the data (figure 3). The next step was to create data ("T") -goals, which are an array of size 54x54, which contains information about the grades given by the participants of the experiment -full-time master's students (22 people) and graduates of higher education institutions there are specialties (32 people) -a total of 54 people. This number of respondents is due to the number of indicators of quality criteria because the data format of the artificial network in Matlab supports square matrices, in this case, 54x54. The data is stored in a T.mat file. Its contents can be viewed using a text editor.
We import data in the same way as for the array P.
In the next step, a neural network was created (figure 4). An artificial neural network for the analysis of indicators of the quality of the educational program and educational activities will have the number of input neurons (according to the number of indicators for all criteria) 54; output neurons -54. Input signals were determined based on students' assessments for each indicator of this quality criterion, while the scales F, E, B, A were converted to numerical 1; 2; 3; 4 respectively.
The configuration of the neural network of direct propagation is chosen based on a heuristic rule: the number of neurons of the hidden layer is equal to half of the total number of input and output neurons. The artificial neural network for the analysis of quality indicators of the educational program and educational activity will have the number of input neurons 2 (according to the dimensionality of the data -indicators of quality criteria and student evaluation); source neurons 54, therefore, the number of hidden neurons is 28. The View button allows you to view the network structure (figure 5).
In our case, 2 is the number of input neurons, which is known to be selected based on the dimension of the input data (1 -indicators of quality criteria; 2 -student assessments). Output neurons -54. The configuration of the neural network of direct propagation (feed-forward backdrop) is chosen based on the heuristic rule: the number of neurons in the hidden layer is equal to half the total number of input and output neurons, so the hidden layer has 28 neurons.
The next stage is network training and coaching. Double-clicking with the left mouse button on the created neural network network1 in the window of the Neural Network / Data Manager opens a window with the network.
The View tab presents the neural network itself. Go to the Reinitialize Weights tab, where the Input Ranges column selects the P input from the Get from the input list. Then press the Set Input Ranges and Initialize Weights buttons in succession allowing us to initialize the scales needed to initialize the entire network.
The next step is network learning. Learning the backpropagation method involves two passes through all layers of the network: forward and backward. In a forward pass, the image (incoming vector) is fed to the sensor nodes of the network, after which it propagates through the network from layer to layer. As a result, a set of output signals is generated, which is the actual response of the network to a given input image. In forward traversal, all synaptic weights of the network are fixed. In a backward pass, all synaptic weights are adjusted according to the error correction rule, namely: the actual output of the network is subtracted from the desired (target) response, resulting in an error signal. This signal subsequently propagates through the network in the opposite direction of the synaptic connections. Hence the name -backpropagation algorithm. The synaptic weights are tuned to bring the network output as close as possible to the desired statistical meaning. The back-propagation algorithm is sometimes referred to as the simplified back-propagation algorithm. The learning process using this algorithm is called back-propagation learning.
Going to the Train tab opens a learning window in which P and T are selected instead of input data and targets, respectively (figure 6 You can calculate that the average network error is 0.0321, which indicates the efficiency of the system.
After learning the network, you can proceed to data forecasting. Returning to the Neural Network / Next, you need to return to the Network window. In the Simulate tab of the input values house, the P1 array is selected, and the Outputs output value is renamed to forecast (figure 7). After clicking the Simulate Network button, you can return to the Neural Network / Data Manager window and, by clicking the Export button, copy the source forecast array to the Matlab workspace. After receiving the table in the workspace, pay attention to the last column, which is responsible for forecasting (figure 8).
The data obtained in the study can be viewed in the forecast.mat file.
Comparing the data issued by the system and the real data, we can see that the neural network does make predictions that are quite close to reality. Compared with expert estimates, the average absolute error is 0.0321, the relative error is 7.08%.
In the second part of the experiment, forecasting was carried out using a different type of neural network -a neuro-fuzzy network, or ANFIS-model.
Expert estimates are used as validation data. Create data files: training.dat, testing.dat, checking.dat. It should be noted that attempts to consider large data volumes lead to a reduction in the number of observations in the training sample and its simultaneous unjustified growth, which can negatively affect the network's ability to learn. So, first you need to turn the available information into a form that is understandable and meaningful for the neuro-fuzzy network. Consider the average value of the assessment of each of the 10 criteria for assessing the quality of the educational program. For training, we use the average scores of all students for each of the 10 criteria. For testing, the marks of students numbered from 12 to 30 are used, for verification -the marks that were put by 31 students.
We preliminarily transpose the data, so the numbers of students will be in the rows, and the grades according to the quality criteria will be in the columns. The data in the files contains 10 columns -9 grades (incoming) and 1 grade (source). The first file contains 54 lines and 10 columns. The second has 18 rows and 10 columns. The third has one row and 10 columns. Anfis Editor is used to building MATLAB fuzzy neural networks. Run the editor with the anfisedit command. In the Load data menu, select Training, and From disk, click the load data button. In the window that opens, select the previously created training.dat file. In the Load data menu, select Testing and From disk, click the load data button. In the window that opens, select the previously created testing.dat. In the Load data menu, select Checking and From disk, and click the load data button. In the window that opens, select the previously created checking.dat. The visualization area contains two types of information: when training the system, the learning curve in the form of a graph of the dependence of the learning error on the iteration ordinal number; when loading data and testing the system -experimental data and simulation results.
Experimental data and simulation results are displayed as a set of points in two-dimensional space. In this case, the serial number of the data line in the sample (training, test, or control) is plotted along the abscissa axis, and the value of the initial variable of this sample line is plotted along the ordinate axis. The following markers are used: blue dot (.) -test set; blue circle (o) -training sample; blue plus (+) -control sample; a red asterisk (*) -simulation results.
Then, having set the Generate FIS menu switch to the Grid partition position, you should press the Generate FIS button. In this case, the model has 10 input variables, each of which corresponds to 9 terms of the gaussmf type. The original variable is determined by a linear function. Let's generate a Sugeno-type fuzzy inference system by pressing the Generate FIS button. In the window that opens, set 3 membership functions of the gaussmf type for each input variable. The choice of the property function here is because we assume a normal distribution for a random variable, defined by a Gaussian function according to probability theory. For the output variable, we set the membership function const.
To train the hybrid network, we will choose the backdrop method (error backpropagation) with an error level of 0 and a number of cycles of 10. Let's start training the hybrid network (figure 10).
As can be seen from figure 10, according to the training results, the average error is approximately 0.007.
We test the fuzzy inference system first on the training set. Now let's test the resulting fuzzy inference system on the known values of expert estimates. Now we download this sample in testing mode in the Anfis editor. The results are shown in Figure 12. The mean score of the experts is 3.99; network prediction of the neural fuzzy network is 3.51. The relative forecast error is 12.57%.
Comparing the prediction errors of the neuro-fuzzy network (12.57%) and the L-layer feed-forward Table 4 also shows that the quality of the program and educational activities is at a fairly high level, which reflects the average score of the peer review.
102-124
The study aimed to demonstrate that the challenge of predicting the assessment of educational programme quality and institutional performance can be adequately addressed through an artificial neural network, obtaining a comprehensive evaluation based on a neuro-fuzzy approach. The mathematical model involves the use of neural networks and is based on the technology of analytical processing of statistical data. Standard methods of mathematical statistics are used to analyse the estimates received from respondents.
The proposals for using students as experts in evaluating educational programmes and institutional activities are debatable; it would be more appropriate to involve teachers from other educational institutions. However, in the process of preparing for self-assessment, this approach can be considered quite suitable.
The results of the neural network should be considered not as final, but as a test. As noted, for more detailed conclusions, it is necessary to train the network on a larger amount of experimental data.
The network structure has room for further improvement and customisation in future studies.
The assumption that a sample of students and graduates can prepare a dataset for setting up and teaching an artificial neural network to evaluate the quality of educational programmes and activities is confirmed by ordering the quality assessments of students and graduates in ascending order of their grade point average. In practice, this allows predicting the results and identifying existing shortcomings to eliminate them before the accreditation examination. However, the difficulty of this method lies in choosing the architecture of the neural network and preparing a training sample to configure it. In particular, future plans include increasing the volume of the input vector of the artificial neural network, with the form based on estimates of teachers, stakeholders, and experts.
As a result of developing a mathematical model for the comprehensive evaluation of educational programme quality and institutional performance based on neuro-fuzzy approaches, we have managed to achieve two key outcomes. Firstly, we have devised a mechanism for obtaining a quantitative assessment of educational programmes and activities that will enable higher education institutions to detect shortcomings and potential problems, and address them prior to accreditation examinations. Secondly, we have demonstrated that a sample of students and graduates can be used to prepare a training dataset for configuring and training an artificial neural network capable of adequately performing a comprehensive assessment of educational programmes and institutional activities. This can be accomplished by arranging the assessments of programme quality and educational activities provided by students and graduates in ascending order based on their grade point average. It is emphasised that these methods are effective provided they adhere to the principles of student-centredness and academic freedom.
By preparing a training sample for setting up and teaching an artificial neural network based on a sample of students and graduates, we were able to evaluate the quality of educational programmes and activities. A comparison of the results produced by an artificial neural network of direct propagation with one output and several inputs with real data shows that the neural network generates predictions close to reality. Compared with expert estimates, the average absolute error was 0.0321, and the relative error was 7.08%.
The results of this study can be applied in the practice of higher education institutions to predict outcomes, identify existing shortcomings, and eliminate them before accreditation examinations.
We see prospects for further research in the application of software products based on neural network theory to automate the processes of organisation, control, and analysis of the educational process, as well as the introduction of neural network software for the direct training of students in certain disciplines.
Declaration on Generative AI: The authors have not employed any Generative AI tools.