In the modern world, the importance of practical evaluation of educational content is increasing due to the rapid development of information technologies and access to many educational resources. Consequently, there is a need to develop and implement recommendation systems for assessing educational content. This paper provides an overview of a project to create a recommendation system for evaluating educational content. This project aims to develop models, methods, and algorithms for automated analysis and recommendations regarding the quality of educational materials. The methodology of working on the project, the tools and technologies used, and the results and areas of application of the recommendation system are described. Potential advantages of implementing a recommendation system in the educational process and methods of interaction with users are considered. The selection of the project's recommendation system lifecycle model has been justified. All stages of the cyclical development process are described. The recommendation system project is developed based on a three-tier architecture. Resources for the implementation of the recommendation system project have been identified. The visualization of the results of the evaluation of the education content is considered using the method of petal diagrams. An example of evaluating methodological guidelines at the faculty's scientific-methodological commission meeting is provided. Criteria for assessing educational content are outlined. Aggregated expert ratings based on evaluation criteria for educational materials are presented. A series of petal diagrams have been constructed to visualize the evaluation of educational content by groups of experts.
Given the factors and circumstances of today, there is a significantly increased necessity for adapting the educational process to an online format. The rapid development of information technologies demands constant updating of methods and approaches to creating and disseminating educational content that should meet the current requirements of society and the job market.
The growing dynamics and volume of new educational content create a decrease in the quality of educational materials. Educational information resources are usually formed without proper verification and testing, which can pose problems in determining their credibility and quality. The increase in the volume of educational content generates the need for expert evaluation of its quality and alignment with academic goals.
Evaluating educational content is a procedure that education experts should follow. In higher education institutions, these experts typically include faculty members, groups responsible for curriculum development, pedagogical teams of scientific and methodological commissions of faculties, scientific and methodological councils of institutes and universities, scientific and technical councils of institutes and universities, academic councils of faculties, institutes, and universities, where the evaluation of educational content is collegially discussed and conducted within expert environments.
A project for a recommendation system for evaluating educational content that implements the appropriate evaluation methodology and a sequence of steps to be taken professionally, promptly, and proficiently is needed.
The research aims to analyze, design, develop, and validate information technologies' models, methods, and components for building a recommendation system for evaluating educational content.
The procedures for selecting and evaluating educational content and electronic resources
are complex processes that involve analyzing a wide range of criteria. It is pertinent to
develop information technology tools that facilitate the practical assessment of such
resources [
The essential research in the development and improvement of recommendation
systems includes contributions from both domestic and foreign researchers: in [
In [
In a study [
According to research [11], an information system of a recommendation type is a specialized information system that facilitates the implementation of basic information processes aimed at providing personalized recommendations to users.
The goal is to develop a recommendation system for evaluating educational content. The educational process is the sphere of application of the information technology components collected in the information system. The intended recommendation system is for expert environments of subject departments in secondary educational institutions, pedagogical councils, cycle commissions, pedagogical collectives of departments, program provision groups, scientific-methodical commissions of faculties, scientific-methodical councils of institutes and universities, scientific-technical councils of institutes and universities, academic councils of faculties, institutes, and universities, overall for all expert communities involved in decision-making regarding the selection and evaluation of educational and educational content [12].
Figure 1 depicts the functional requirements of information technology components used in building the recommendation system prototype as a UML use case diagram, highlighting the main actors in the process of working with the recommendation system [13]. For experts, it is possible to create a user profile based on which initial recommendations are provided. This profile forms a request for selecting resources and educational content, taking into account the type and ratings for each criterion and receiving a list of recommended alternatives. Additionally, the expert can modify the parameters of their profile, thus managing their account.
The "Administrator" is responsible for configuring the prototype of the recommendation system, managing users, and receiving analytical data on user interaction with the system. The set of use cases includes: "Account Management"; "Management of Selection Criteria"; "Management of Electronic Learning Resources and Educational Content"; "Generation of Analytics for Selection of Electronic Learning Resources and Educational Content."
To verify the reliability of the results of the functioning of information technology components that were used as the basis for building the prototype of the recommendation system, a series of experimental studies were conducted from 2020 to 2023 at Lesya Ukrainka Volyn National University, particularly at the Faculty of Information Technology and Mathematics.
One example of using the developed recommendation system project illustrates the evaluation of educational content at a meeting of the scientific-methodical commission of the Faculty of Information Technology and Mathematics. To conduct the experimental study, participants of the scientific-methodical commission were asked to evaluate the teaching material submitted for recommendation for publication. The expert community was also asked to assess the guidelines for practical exercises in the "Computer Discrete Mathematics" course. Participants of the scientific-methodical commission were briefed on using the proposed software product and provided with necessary technical support. The expert community evaluated each unit of educational content (developed practical exercise) using the proposed toolkit. Since the participants of the scientific-methodical commission are lecturers from various departments, the averaged values of all expert ratings were considered. As a result of the evaluation, it was found that in two practical exercises, the ratings were lowest for the criteria of the presence of solved examples and the availability of necessary literature sources. This allowed the participants of the scientific-methodical commission to indicate to the developer the need for revisions to the practical exercises according to the specified requirements.
The recommendation system project was implemented using the spiral model. This is an iterative model that combines elements of both sequential and iterative approaches. It proposes a cyclic development process, where each cycle consists of 4 stages [14]: - Planning - defining the goals and tasks of the cycle, risk assessment, and resource planning.
- Analysis - gathering and analyzing information necessary for implementing the cycle. - Design - developing detailed design of system components to be developed within the cycle.
- Implementation - coding, testing, and deployment of system components.
The choice of the spiral model for the development lifecycle of the recommendation system is driven by the fact that creating a recommendation system is a complex and undefined process that requires a flexible approach. Early testing and obtaining user feedback are critical success factors for the project. A recommendation system is a system that must constantly evolve to meet changing user needs [15].
The recommendation system project is developed based on a three-tier architecture (see Fig. 2). This allowed for separating the system into interconnected parts, distributing system functions among them, and isolating the user interface from the data. The three-tier architecture includes: - Presentation layer – the level at which the user perceives information.
- Application layer – the level where tools for managing the recommendation system are located, as well as components such as setting the type of educational content and objectives, searching for educational content and objectives, displaying results, and generating reports.
- Data management layer – the level where data is physically stored, with subsystems for determining the type of educational content and objectives, analyzing educational content and objectives, generating results, and generating user reports.
An approach utilizing petal diagrams has been considered to visualize the assessment of educational content. The methodological guidelines were evaluated based on the following criteria: - Relevance of the topic to the educational component syllabus. - Adequacy of necessary theoretical educational material. - Structuring of the material. - Presence of solved task examples. - Provision of tasks for independent completion. - Availability of required literary sources.
The assessment of educational content is considered within a polar coordinate system, where an irregular polygon is formed [16]. The area of this polygon reflects qualitative and quantitative aspects of assessing educational content across all its characteristics simultaneously. The shape of the polygons represents qualitative characteristics of academic content across all criteria simultaneously, while the shape of sectoral polygons indicates compliance with specific criteria. The difference between the circle's area and the polygon's area is a fraction that needs to be achieved at a certain point to improve performance [17].
Expert ratings are obtained through surveys, utilizing a ranked scale to assess each criterion. Expert groups determine corresponding ratings, which are then considered using appropriate weighting coefficients (Table 1). The influence of each criterion on the overall score varies according to each expert's individual determination of values. The authority coefficients considered in the calculations differ depending on the qualifications of the experts (Table 2). The values of importance coefficients are expressed in absolute and relative units (Table 3). These values adjust the aggregated indicators of educational materials assessed by
material
solved tasks
6 9 8 9 experts. Considering their significance, the initial values of experts' authority coefficients are determined empirically.
To determine the comprehensive indicators of educational materials, a set of ratings provided by respective experts is used (Tables 3 and 4) [18]. ̃ = { , = , ∙ , ∙ , = ̅1̅,̅̅̅, = ̅1̅̅,̅̅̅}, where gi,k is the comprehensive indicator of educational materials, xi,k is the rating of educational materials, wi,k is the weighting coefficients of educational materials, and qk is the coefficient of importance of experts [19].
Experts evaluate educational materials on a 10-point scale, and the weighting coefficients of evaluation criteria are also on a 10-point scale. The coefficient of experts' importance is assessed from 0 to 1; hence, the comprehensive indicator of educational materials is assessed with values from 0 to 100. were considered to construct a radar diagram, which serves as segments delayed from the origin of the coordinate system. Using the lengths of these segments that meet the criteria, the formula for obtaining a radar chart is derived: 1 2
0,7 44,1 33,6 63 44,8 63 63 0,9 54 64,8 64,8 63 50,4 57,6 0,8 57,6 39,2 57,6 50,4 64 72
(2) Average ratings
0,8 51,90 45,87 61,80 52,73 59,13 64,20 (3) zk =
k Sпд , πr2
k ∈ K + 1 During the division of the area of the obtained polygon by the area of the circle, the quotient representing the quality ratio of educational materials according to expert ratings was obtained: where r is the radius of the circle, zk is the proportion of the available conformity of educational materials to the specified criteria. As identified, the radius of the circle will be equal to 100, since the comprehensive indicator of conformity of educational materials (gi⬚,k) also equals 100 in the case of maximum value. The unfilled portion of the sector's area indicates the need for improvement of educational materials according to the specified criterion. Figures 3-6 record the assessments of conformity of educational materials to the specified criteria using radar charts [20].
In the recommendation system project, radar charts are used to visualize the results of assessing educational content. The criteria values are calculated using the aggregated ratings of experts, the weighting coefficients of the requirements, and the weighting coefficients of the experts. The unfilled portion of the sector indicates the need to improve educational materials according to the specified criterion [21].
The components of information technology used as the basis for building the recommendation system are implemented in the form of a multi-page web application, which offers several advantages: - A feature-rich interface. - Fast interface responsiveness since all actions do not require server access. - Significant reduction in server load. - Personalization and fast data transmission speed.
The software implementation is carried out using the following tools: 1. Node.js is used for scripting the web application.
2. Express is a widely used Node.js framework for developing web applications and APIs.
3. Twig template engine (for HTML) - Twig is a template engine for developing HTML templates in PHP applications.
4. CSS (Cascading Style Sheets) - a style sheet language used for styling and presenting the appearance of web pages written using markup languages such as HTML or XML. 5. MongoDB - a document-oriented database that falls under NoSQL databases. 6. npm (Node Package Manager) is one of the most popular package managers in Node.js and JavaScript environments. It allows developers to manage and use third-party libraries and modules developed by other programmers and publish their packages for use by other users.
The functional purpose of the recommendation system project in evaluating educational content is objectively assessing the developed methodological materials. This system facilitates convenient and efficient interaction among experts with their perspectives on content evaluation, with the tools helping objectively consider the criteria' multi-aspect nature.
By utilizing the assessment scores based on established criteria and activating computations, the recommendation system project assists experts in conducting responsible and well-founded evaluations of educational content. With the help of a database, the recommendation system can store many resources and related information, facilitating efficient selection and quick access to recommended ratings. Visualization of recommended results using radar charts promotes easy understanding and comparison of content considering its characteristics.
The research successfully addressed a pressing scientific task by designing and developing a recommendation system for assessing educational content within educational expert environments tasked with making decisions regarding creating highquality educational materials. [11] V. Yunchyk, Information Technologies for Educational Content Formation for ELearning Systems, Doctoral Dissertation in Philosophy, Specialty 126 "Information Systems and Technologies", Lviv Polytechnic National University, Lviv, 2023. [12] V. Pasichnyk et al., Model of the Recommender System for the Selection of Electronic Learning Resources, CEUR Workshop Proceedings 5th International Workshop on Modern Machine Learning Technologies and Data Science (MoMLeT+DS 2023), Vol. 3426, pp. 344-355. ISSN 1613-0073. [13] V. Yunchyk and Y. Fedoniuk, Results of developing the recommendation system for electronic educational resource selection, Manažérska Inform. Vedecký Časopis O Inform., vol. 1, no. 1, 2023, [Online]. Available: https://manazerskainformatika.sk/results-of-developing-the-recommendationsystem-for-electronic-educational-resource-selection/ [14] K. Qian and S. Jain, Digital Content Creation: An Analysis of the Impact of
Recommendation Systems, Manag. Sci., 2024, doi: 10.1287/mnsc.2022.03655. [15] N. Cespedes Garcia, and P. Cespedes Garcia, Life Cycle Models of Software Development, Young Scientist, 2023. vol. 2 (114), pp. 17-20. doi: 10.32839/23045809/2023-2-114-4.. [16] B. Sadeghi Bigham and A. Mohades, The Dual of Polar Diagram and its Extraction 1, Recent Progress in Computational Sciences and Engineering, 1st ed., T. Simos and G.
Maroulis, Eds., CRC Press, 2019, pp. 451–454. doi: 10.1201/9780429070655-108. [17] K. Nazemi, D. Burkhardt, D. Hoppe, M. Nazemi, and J. Kohlhammer, Web-based Evaluation of Information Visualization, Procedia Manuf., vol. 3, pp. 5527–5534, 2015, doi: 10.1016/j.promfg.2015.07.718. [18] B. Saket, A. Endert, and C. Demiralp, Task-Based Effectiveness of Basic Visualizations, IEEE Trans. Vis. Comput. Graph., vol. 25(7), pp. 2505–2512, 2019, doi: 10.1109/TVCG.2018.2829750. [19] Yu. Grytsiuk and V. Daliavskyi, Usage of Petal Diagrams for Visualizing the Results of Expert Assessment of Software Quality, Scientific Bulletin of NLTU of Ukraine, vol. 28(9), pp. 95–104, 2018, doi: 10.15421/40280919. [20] C. Xiong, V. Setlur, B. Bach, K. Lin, E. Koh, and S. Franconeri, Visual Arrangements of Bar Charts Influence Comparisons in Viewer Takeaways, 2021, doi: 10.48550/ARXIV.2108.06370. [21] A. Fedonuyk, V. Yunchyk, T. Cheprasova, and S. Yatsyuk, The Models of Data and Knowledge Representation in Educational System of Mathematical Training of ITspecialists. IEEE 15th International Conference on Computer Sciences and Information Technologies, vol. 2, pp. 269-272, 2020. doi:10.1109/CSIT49958.2020.9321899.