Expert evaluation of electronic learning resources (ELRs) has been considered, based on the theory of fuzzy logic with the use of the Analytic Hierarchy Process (AHP) method. The concept of fuzzy logic has been used for quantitative evaluation of qualitative data in real decision-making tasks. The development of the recommender system based on fuzzy logic methods for expert evaluation of e-learning resources is proposed, including computer mathematics systems, and making decisions for choosing the most effective resources for use in the learning process. The term of the recommender system for choosing the most effective e-learning resources is considered and analyzed. A scientific publications review was conducted on the problems of expert evaluation, the use of fuzzy logic methods and the use of recommender systems. The general structure of the recommender system along with a description of all subsystems is given. The features of using of fuzzy logic theory in the selection of computer mathematics systems has been considered. The main criteria for evaluating computer mathematics system (CMS) have been presented, and the paired comparison method has been used to calculate the importance of criteria. The process of digital learning objects evaluation using fuzzy logic methods has been described in detail, and the algorithm of this approach has been presented. As a result of the expert assessment, a list of recommended alternatives for e-learning resourses that correspond to the specified criteria, has been obtained. The general structure of the recommender system, for selecting digital learning objects, has been presented. Using the UML language, the diagram of options, sequence diagram, and activity diagram of the recommendation system for selecting DLO have been designed. The main stages of user interaction with the recommendation system, which contributes to the choice of digital learning objects, have been described.
learning objects and significant aspects of their effectiveness in transferring knowledge from authors to users.
To ensure quality control of the assessment of e-learning resources, it is important to formulate a list of requirements for electronic learning resources, select experts for quality assessment of electronic resources, and conduct expert evaluation of the quality of e-learning resources. In the past, classical methods of evaluation were based on traditional logic and binary mathematics, but new approaches, such as fuzzy logic, have appeared. The concept of fuzzy logic and fuzzy sets was first introduced by Lotfi A. Zadeh. They represent a mathematical tool that allows mathematical justification of tasks that don’t have complete statistics, or need to resolve the problem of reconciling conflicting criteria for determining preferences.
Fuzzy logic makes it possible to widely use expert knowledge in generation of advantages. Based on the fuzzy logic tool it is possible to design decision support systems that can function effectively in conditions where there is information about the studied object that has a qualitative nature.
A large number of domestic and foreign scientists engage the questions of e-learning resources implementation in the learning process. Special attention is paid to the quality of such resources. Researcher Reem Almazyad [19] created an online survey to collect empirical data on quality assurance points in the following areas: infrastructure, institutional vision, development of electronic learning resources, content and learning support.
Based on the quantitative empirical study by Carla Reinken [
Issues of quality research of software using fuzzy logic are angaged by Cai Yakun, Juan Jyanchan,
and others [24]. The presented model is aimed at rapid evaluation of software quality and can
accurately display the relationship between internal and external properties of software. Audrey
Romero-Pelaez and Renee Solano [
The process of evaluating electronic learning resources is a complex procedure that involves functional capabilities analysis, accessibility, architecture, and other criteria. Currently, it is relevant to develop a certain algorithm that contributes to effective evaluation of e-learning resources. The effective option of such algorithm is recommender system.
The most important scientific works in the field of research, development and improvement of
recommender systems are works of foreign and domestic scientists, such as: Aurora Esteban, Amelia
Zafra [
In the field of educational services using a recommender system, learning resources are selected based on the learning style and knowledge level of students, thereby ensuring the effectiveness of the educational process.
In other words, participants in the educational process can be provided with personalized learning
content [
Regardless of the application field and implementation specifics, the purpose of the recommendation system is to provide the user with the most relevant information, which can take the form of various products.
In our research, recommender systems are a class of intelligent systems that generate recommendations on fuzzy logic methods by forming a rated list of electronic learning resources that meet certain requirements and criteria.
The aim of the article is to model and develop the recommender system based on fuzzy logic methods for expert evaluation of e-learning resources that allows us to choose of the most effective ELRs for use in the learning process on the basis of generated recommendations.
The development of a recommender system for evaluating and selecting electronic learning resources aims to make the selection of alternatives process fast, accurate and correct.
Creating a conceptual model of the recommendation system allows to identify its different components and possible ways of their interaction. Conceptual model developing of the recommender system ensures the identification of its different entities and their potential interaction. Identifying these requirements at this stage allows to save resources in subsequent stages of the development life cycle, when adding new components of the recommendation system requires more effort and resources.
UML diagrams are a powerful tool for visualizing and simplifying the understanding of the conceptual model of the system. They allow to describe various aspects of the recommender system such as its structure, behavior and interaction with users.
In particular, UML diagrams include class diagrams, sequence diagrams, activity diagrams and others. The use of UML diagrams simplifies understanding between developers and customers, and also promotes more efficient process of information system development.
For describing the functions of the recommender system, the use case diagram (Fig. 1) which shows the interaction between the user and the recommender system, i.e., the relationship between the user and various use cases in which the user is involved, has been developed.
There are two actors in this system: «User» and «Administrator». «User» is a person who has authorization in the system and full access to all functions and capabilities.
A registered user can create his profile, based on which initial recommendations will be provided. The «User» can form a request for the selection of electronic learning resources (ELR), specifying a number of selection criteria, evaluate suggestions, change the selection criteria if necessary, and receive a list of recommended alternatives.
In addition, «User» can make changes to the parameters of his own profile, that is, manage his account.
«Administrator» is such User who is responsible for setting up the recommender system, managing users, and can receive analytics related to user interaction with the system. The set of use cases are: «Manage users»; «Delete user»; «Add new suggestions of electronic learning resources»; «Manage ELR selection criteria», «See the analytics of system usage»; «Generate report».
Figure 2 shows the sequence diagram of the recommender system for selecting electronic learning resources. First, the user makes a request for a recommendation according to the ELR selection criteria set by him. The controller checks the metadata and descriptions available in the database of elearning resources, forms a matrix of pairwise comparisons and transforms the data into a fuzzy matrix. Then, the system, that uses fuzzy logic, determines a list of recommended alternatives. Electronic learning resources are recommended to users if they have high indicators according to the specified criteria, in particular, taking into account the importance of the criteria. all elements and compares criteria or alternatives with respect to the criterion in a natural, pairwise mode.
The process begins with collecting quantitative and qualitative data, as well as linguistic decisions from decision-making experts. Then, the necessary calculations are performed using fuzzy logic methods.
A fuzzy subset ̃ of the universal set U is characterized by a membership function , -, which assigns to each element a number ( ) from the interval , - which characterizes the degree of membership of the element in the subset A. The degree of membership is a number from the range , - An element of the universal set with a higher degree of membership corresponds more to the properties of the fuzzy set.
The carrier of the fuzzy set A is the set of points in U for which the value of ( ) is positive. The height of the fuzzy set A is the value of ( ).
The membership function is a function that allows to calculate the degree of membership of any element of the universal set in the fuzzy set.
If the universal set is finite * +, then the fuzzy set ̃ is expressed as follows [26]: In the case of the continuous set the following notation is used: ̃ ∫
( )
The set A can be defined by the characteristic function ( ), which takes one of two values: 0 – if u doesn’t belong to the set ( ), and 1 - if it does ( ). Thus, a crisp set can be considered as a limiting case of a fuzzy set, which membership function aquires only binary values [17].
The practical application of fuzzy set theory involves the existence of membership functions that are described by linguistic terms such as «low», «medium», «high», and others. The task of constructing membership functions is as follows. There are two sets: a set of terms * + and a universal set * +. A fuzzy set ̃to specify the linguistic term on the universal set U is defined as [26]: ̃ ( ( ) ( ) ( ) ) ̅̅̅̅̅̅
It is necessary to determine the degrees of elements membership of the set to elements of the set , that is, to find ( ) for all ̅̅̅̅̅̅ і ̅̅̅̅̅.
There are two methods for constructing membership functions. The first method is based on the statistical processing of decisions made by a group of experts. The second method is based on pairwise comparisons made by a single expert. Analytical expressions are provided for approximating membership functions that are constructed with the help of expert data.
In the first method of constructing membership functions, each expert fills out a questionnaire which indicates his decisions about the presence in the elements ( ̅̅̅̅̅) of properties of the fuzzy set ̃ ( ̅̅̅̅̅̅).
Let
be the number of experts; properties in the element assessments are binary, i.e., of the fuzzy set ̃ * + , where 1 indicates the presence in the element - the decision of the k-th expert about the presence of ̅̅̅̅̅ ̅̅̅̅̅ ̅̅̅̅̅̅ We assume that expert properties of the fuzzy set ̃ , and 0 indicates their absence. According to the results of the survey, the degrees of membership of the fuzzy set ̃ are calculated as follows: ( ) ∑ ̅̅̅̅̅ ̅̅̅̅̅ ( )
In the construction of the membership function using the second method, for each pair of elements in the universal set, an expert assesses the advantages of one element over the other regarding the properties of the fuzzy set. Such pairwise comparisons can be conveniently represented by the following matrix: [ ] where is the level of preference of the element over ̅̅̅̅̅, that is determined by the nine-point Saaty scale: 1 – if there is no preference of element over ; 3 – if the preference of element over is weak; 5 – if the preference of element over is significant; 7 – if the preference of element over is obvious; 9 – if the preference of element over is absolute; 2, 4, 6, 8 – intermediate comparative estimates: 2 – almost weak preference, 4 – almost significant preference, 6 – almost obvious preference, 8 – almost absolute preference. where – the maximum eigenvalue of the matrix A.
If * + – the set of options that need to be analyzed.
* + – the set of criteria used to evaluate the options.
The task of multicriteria analysis is to rank elements of the set P according to the criteria from the set .
Let ( ) be a number in the range [
̅̅̅̅̅ ̅̅̅̅̅ { ( ) ( ) ( ) } where ( ) – is the degree of element membership in the fuzzy set ̃
Finding the membership degrees of a fuzzy set is conveniently done using the pairwise comparison method to construct a membership function based on pairwise comparisons. Using this method, it is necessary to form pairwise comparison matrices of options for each criterion. The total number of such matrices should be equal to the number of criteria.
The best option is the one that is simultaneously the best according to all criteria. The fuzzy solution ̃ is found as the intersection of criteria [26]: ̃ ̃ ̃ ̃ { ̅̅̅̅̅ ( ) ̅̅̅̅̅ ( ) ̅̅̅̅̅ ( ) }
According to the obtained fuzzy set ̃ the best option should be considered the one with the highest degree of membership. ( ) ( ) ( ) ( ) ( ) The pairwise comparison matrix is diagonal ( ̅̅̅̅̅) and inversely symmetric
The degrees of membership are considered equal to the corresponding coordinates of the
eigenvector ( ) the matrix of pairwise comparisons A:
( )
̅̅̅̅̅
The eigenvector is found from the following system of equations [
( )) ( ) ̅̅̅̅̅ .
( )/ ̅̅̅̅̅
In case of unevenly important criteria, the degrees of membership of the fuzzy set ̃ can be found as follows: where – represents the coefficient of relative importance of the criterion
The exponent in the formula concentrates the fuzzy set ̃ according to the importance of criterion . The coefficients of relative importance of criteria will be determined using pairwise comparisons by the Saaty scale [22].
As an example of decision-making in fuzzy conditions using the Bellman-Zadeh scheme, will be considered comparison of four computer mathematics systems (CMSs): GeoGebra, SMathStudio, SageMath, Wolfram Mathematica ( ) for effective use in the mathematical training of IT specialists. For evaluating the Computer Mathematics Systems (CMSs), we will use the following criteria [28]: – interactivity; – multimodality; – modification capability; – cross-platform compatibility; – open source availability; – architecture; – functionality; – number of topics for processing; – relevance to the subject area; – compliance with the educational standards.
The expert conclusions correspond to the following pairwise comparison matrices: Each matrix has six elements that correspond to pairwise comparisons from the table. Other elements are found, taking into account that the pairwise comparison matrix is diagonal and inversely symmetric.
Applying formulas (2) and (3) to the pairwise comparison matrices (7), we obtain the following fuzzy sets: ( ) ( ) ( ) ( ) ( ) (7) ( )
According to (8) follows, that CMS is better according to the criteria , , , , - ; CMS – according to criteria - , . Therefore the choice of CMS will depend on the importance of the criteria.
To calculate the coefficients of relative importance of criteria, we will use the expert pairwise comparison method. The following pairwise comparison matrix corresponds to expert judgments: [ ] Using formulas (2) and (3), we can calculate the coefficients of relative importance of criteria that means the highest importance for decision-making is given to functionality ( ), the number of topics for processing ( ) and relevance to the subject area ( )
Using formula (6), we obtain the following fuzzy sets: ̃ ̃ ̃ ̃ ̃ ̃ ̃ ̃ ̃ ̃ { { { { { { { { { { } } } } } } } } } { { { { { { { { { } { } } } } } } } } } }
The intersection of these fuzzy sets gives the following degrees of membership for the fuzzy decision ̃ : ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) As a result, we obtain a fuzzy set ̃ { } ̃ ̃ ̃ ̃ which indicates the superiority of CMS which indicates the superiority of SCM over others. Thus, SCM is better than the others, which simultaneously satisfy all criteria, taking into account their importance. Fuzzy sets, that show how fully the CMS meet the criteria , are written as follows:
The recommended CMS according to criteria . is shown in Figure 4. This CMS was obtained by calculating the unfilled area of a circle, that is, the part of the ELR quality that still needs to be achieved for its one hundred percent completeness [25].
Based on the conducted research, a list of recommended CMSs has been obtained in the following sequence: GeoGebra, Wolfram Mathematica, SMathStudio, SageMath. According to the research, the recommender system will suggest alternative CMS in such sequence that corresponds to the criteria mentioned above. The main task of the recommender system is to provide personalized recommendations to the user, taking into account their preferences while is choosing electronic learning resources.
As a result of the study, a recommendation system that generates recommendations based on fuzzy logic methods for expert evaluation of electronic training resources including computer mathematics systems is obtained. This system could be used for choosing alternatives to choose the most effective ELR for use in the learning process. The process of selecting ELRs is based on fuzzy logic theory using the Analytic Hierarchy Process method. A methodology has been developed for working with qualitative and quantitative criteria in conditions of uncertainty. The applicability of these methods is illustrated by the example of the problem of choosing electronic learning resources.
The analysis of recommender systems led to the introduction of the concept of that generates recommendation system for choosing the most effective electronic learning resources. Through expert evaluation of electronic learning resources, using fuzzy logic methods, a list of recommended resources, that meet specified criteria, was obtained. The results show that the use of the demonstrated recommendation system is an effective means of decision support for selecting electronic learning resources.
[15] O. Artemenko, O. Kunanets and V. Pasichnyk “E-tourism recommender systems: a survey and development perspectives” Econtechmod, vol. 6(2), (2017), pp. 91-95. [16] O. Artemenko, V. Pasichnyk and N. Kunanets “Using Mobile Location-based Recommender Systems for Providing Real Time Recommendations for Social Distancing Urban Route Planning” In: 2020 IEEE 15th International Conference on Computer Sciences and Information Technologies (CSIT), 2020, vol. 2, pp. 305-308. [17] O. Dudnyk and Z. Sokolovska “Forecasting Development Trends in the Information Technology Industry Using Fuzzy Logic” Eastern-European Journal of Enterprise Technologies, vol. 1(13), 2023. doi:10.15587/1729-4061.2023.267906. [18] P. Veres, O. Kots, Y. Levus, and O. Vlasenko. “Recommendation System for Leisure TimeManagement in Quarantine Conditions” In: Modern Machine Learning Technologies and Data Science Workshop (vol. 3312, pp. 263-282). MoMLeT&DS 2022. [19] R. Almazyad, “Enhancing the Quality and Reliability of OER Content” in: Eighth International Conference on Educational Innovation through Technology (EITT), 2019, pp. 35-38. doi:10.1109/EITT.2019.00016. [20] S. Aguilar “A research-based approach for evaluating resources for transitioning to teaching online” Information and Learning Sciences. vol. 121(5/6) (2020), pp. 301-310. doi:10.1108/ILS04-2020-0072. [21] S. Wan, Z. Niu “An e-learning recommendation approach based on the self-organization of learning resource”. Knowledge-Based Systems, vol. 160, pp. 71-87, 2018. doi:10.1016/j.knosys.2018.06.014. [22] S. Yatsyuk, V. Yunchyk, S. Mukutuyk, O. Duda, O., and A. Fedonuyk. Application of the hierarchy analysis method for the choice of the computer mathematics system for the IT-sphere specialists preparation. Journal of Physics: Conference Series, vol. 1840(1), 2021. doi:10.1088/1742-6596/1840/1/012065. [23] S. Yatsyuk, V. Yunchyk, T. Cheprasova, and A. Fedonuyk. “The Models of Data and Knowledge Representation in Educational System of Mathematical Training of IT-specialists”. IEEE 15th International Conference on Computer Sciences and Information Technologies, vol. 2, pp. 269272, 2020. doi:10.1109/CSIT49958.2020.9321899. [24] T. Sun, X. Lv, Y. Cai, Y. Pan, and J. Huang. “Software test quality evaluation based on fuzzy mathematics”. J. Intell. Fuzzy Syst., 40, pp. 6125-6135, 2021. doi:10.3233/JIFS-189451. [25] V. Pasichnyk, V. Yunchyk, and A. Fedoniuk “Procedures for assessing the quality of electronic learning resources using petal diagrams” Herald of the National University Lviv Polytechnic.
Information systems and networks, (11), 2022, pp. 87-102. [26] V. Pasichnyk, V. Yunchyk, N. Kunanets, and A. Fedoniuk “Using fuzzy logic in the process of expert evaluation of elearning resources”. Scientific Bulletin of UNFU, 32(4), pp. 66-76. 2022. doi:10.36930/40320411 [27] V. Yunchyk, A. Fedonuyk, M. Khomyak, and S. Yatsyuk, S. Cognitive modeling of the learning process of training IT specialists. In CEUR Workshop Proceedings, pp. 141-150, 2021. [28] V. Yunchyk, and A. Fedonyuk, A. A. Comparative characteristics of the functionality of the system of computer mathematics in the process of task solving. Herald of the National University Lviv Polytechnic. Information systems and networks, vol. 6, pp. 90-103, 2019. doi:10.23939/sisn2019.02.090. [29] Ye. Meleshko, S. Semenov, and V. Khokh. “Research of methods of building advisory systemson the internet” Control, Navigation and Communication Systems. Academic Journal. 1. pp. 131-136, 2018. doi:10.26906/SUNZ.2018.1.131.