In today's world, the importance of effective evaluation of educational content is increasing due to the rapid pace of development in information technology and access to a large number of educational resources. The study provides an example of an expert assessment of educational content in the process of training IT specialists. Using the example of the educational program "Computer Science and Information Technologies" at the bachelor's degree level, the main types of educational content were identified. Quantitative characteristics of the educational content for IT specialist s training were compiled into a table. Expert communities for assessing educational content were identified. Calculations were made for expert assessments needed at each stage of the training course. In this context, there is a need for the development and implementation of recommendation systems for evaluating educational content. An overview of the recommendation system for evaluating educational content is proposed. The functional purpose of the recommendation system in the context of evaluating educational content is to ensure objective assessment of developed methodological materials. Potential advantages of implementing the recommendation system in the educational process and methods of interaction with users are considered. A prototype of the recommendation system is developed based on a three-tier architecture. Information technology components used as the basis for building the recommendation system are implemented as a multipage web application. To visualize the results of evaluating educational content, an approach using radar charts is considered. The study addresses the relevant scientific task of developing a recommendation system for evaluating educational content for educational expert environments that need to make decisions regarding the formation of quality educational content.
Currently, one of the pertinent areas of information technology application is in the field of education. The desire to enhance learning processes and their increasing dependence on various information resources generate the need for the development and implementation of innovative teaching methods and tools, particularly electronic learning systems. A fundamental component of modern electronic learning systems is educational content, which requires active updating and adaptation to constantly evolving needs.
The availability of Internet network resources and the expansion of electronic learning system capabilities significantly emphasize the task of selecting quality and effective educational resources. In the context of a vast array and saturation of educational materials, the problem of choosing the most effective and high-quality educational resources becomes particularly relevant.
The widespread use of electronic learning systems in educational processes actively encourages developers of educational resources to create diverse, high-quality, and up-to-date educational content. The increase in the dynamics and volumes of creating new educational content often leads, in many cases, to a decrease in the quality of educational materials. Educational information resources are typically formed without proper verification and testing, which can pose challenges in determining the credibility and quality of resources. The increase in the volume of educational materials generates the need for professional assessment of quality and alignment with educational goals.
The assessment of educational content is a procedure typically carried out by experts in the field of education. In higher education institutions, these experts usually include faculty members, curriculum development groups, and pedagogical teams of faculty-level scientific and methodological commissions. They also include scientific and technical councils of institutes and universities, academic councils of faculties, institutes, and universities in expert environments where the evaluation of educational content is collegial discussed and conducted.
There is a need to analyze the possibilities of expert assessment of educational content and to develop a recommendation system for assessing educational content. This recommendation system implements the appropriate assessment methodology and the sequence of steps to be taken professionally, promptly, and competently.
The purpose of the research is to analyze expert assessment of educational content, develop and test models, methods, and components of information technology for building a recommendation system for assessing educational content.
The utilization of modern information technologies in the processes of educational content
formation and evaluation is explored by scholars such as A. Burden [
Research on the development of information technologies for educational content
formation based on artificial intelligence methods is dedicated to the works of I. Viznyuk [
Studies on the development and improvement of recommendation systems are addressed
by many domestic and foreign researchers, such as O. Veres [
In the extensive spectrum of analyzed research results, the utilization of recommendation systems in various fields of activity is presented. However, insufficient attention is given to the issues related to the creation and utilization of information technologies for building recommendation systems for expert evaluation of electronic learning resources and educational content. Such systems are necessary and can be effectively used by expert communities, which regularly need to make decisions regarding the selection and provision of recommendations for the use of new high-quality electronic learning resources.
According to research [20], an information system of a recommendation type is a specialized information system that facilitates the implementation of basic information processes to provide personalized recommendations to users.
At Lesya Ukrainka Volyn National University, IT specialists are trained under the Computer Science and Information Technologies educational program. The bachelor's degree curriculum includes 6 educational components in the general training cycle, 30 educational components in the professional training cycle, and 12 educational components in the elective cycle. The professional training cycle of education seekers was analyzed for the necessary types of educational content.
For the proper preparation of IT specialists within the Computer Science and Information Technologies educational program at the bachelor's degree level, the following types of educational content were identified:
K1 Lecture outlines on the topic; K2 Guidelines for practical work; K3 Guidelines for laboratory work; K4 Module control work task sets; K5 Test task sets; K6 Guidelines for independent study; K7 Guidelines for independent work; K8 Guidelines for writing a term paper; K9 Guidelines for internships; K10 Guidelines for writing a qualification paper.
According to the curriculum of the Computer Science specialty (122), a different number of units of educational content needs to be evaluated for each educational component. This takes into account the specifics of the educational components and the number of hours allocated for study (see Table 1).
The mechanism of recommending educational publications for printing and use in the educational process involves consideration at the department level, during the meeting of the faculty scientific and methodological commission. It also involves consideration at the meeting of the university scientific and methodological council.
Faculty members from the department where the author works and which deals with this issue review, discuss, evaluate, and recommend publications for printing.
Participants of the faculty scientific and methodological commission oversee the educational publication to ensure compliance with current requirements and the quality of publications intended for printing, as well as recommend them for use in the educational process e s r u o c 1 e s r u o c 2 e s r u o c 3 e s r u o c 4 39 18 18 26 18 13 17 34 18 18 22 18 14 15 15 17 20 20 15 15 15 15 15 15 18 35 18 17 18 18 40 18 18 17 44 23 19 17 19 19 23 20 24 18 18 20 20 20 18 18 2 1 4 3 3 4 2 3 2 1 2 2 2 2 1 2 2 1 2 2 2 1 2 2 1 3 2 3 3 2 1 1 2 4 2 2 1 2 4 1 1 1 1 1 1 1 1 1 1 1 1
The university scientific and methodological council reviews and recommends (or rejects) materials submitted by authors to the educational department for printing. The process of recommending educational publications must undergo evaluation by three expert communities, with the number of individuals specified in Table 2.
Expert Assessment Communities
The composition of the graduating department Faculty Scientific and Methodological Commission University Scientific and Methodological Council
Let * + be the set of expert communities, where { } is the set of participants in the faculty scientific and methodological commission, { } is the set of faculty members from the graduating department, { } is the set of participants in the university scientific and methodological council.
Let ( ̅̅̅̅̅̅) be educational components, and { } be the set of types of educational content. Then, the sum of units of educational content for is given by: Let be the sum of units of educational content to be evaluated for the 1st year, i.e., , be the sum for the 2nd year, i.e., , be the sum for the 3rd year, i.e., , be the sum for the 4th year, i.e., .
Then, for the first year, ∑ expert evaluations are needed, for the second year ∑ , for the third year ∑ and for the fourth year ∑ . In total, over the entire study period evaluations are required, where:
Tables 3-6 provide quantitative characteristics of expert evaluation of educational content for IT specialist training for each year, and Figures 1-4 illustrate the quantitative expert evaluation.
Therefore, for evaluating the educational content in the first year of IT specialists' training 14364 expert procedures are required. In the second year, this number slightly increases to 15204, while there is a decrease in the third year to 8736, as it includes coursework and internships. On the fourth year, the number decreases further to 7980 and the total number of expert procedures required for the entire training period is 46284 (Table 7, Figure 5).
Overall, it can be argued that the process of evaluating educational content is quite laborintensive and time-consuming. In the majority of situations, the processes of evaluating electronic educational resources and educational content are usually carried out improperly or not conducted at all. The developed and proposed information technology toolkit is aimed at significantly improving, simplifying, and expediting the implementation of expert evaluation processes for electronic educational resources and educational content in expert educational environments.
The purpose of the prototype of the recommendation system for evaluating educational content and resources is to provide users with personalized recommendations for selecting the best electronic educational resources and educational content in specific educational situations. The system aims to provide users with quality and relevant resources that meet their needs and enhance the effectiveness of learning. [21].
The sphere of application of the recommendation system is the educational process. The intended recommendation system is for expert environments of subject departments of secondary educational institutions, pedagogical councils, cyclical commissions, pedagogical collectives of departments, support groups for educational programs, scientific and methodological commissions of faculties, scientific and methodological councils of institutes and universities, scientific and technical councils of institutes and universities, academic councils of faculties, institutes, and universities, overall for all expert communities that need to make decisions regarding the selection and evaluation of electronic educational resources and educational content.
The prototype of the recommendation system is developed based on a three-tier architecture (Figure 6). This allowed for dividing the system into interconnected parts, distributing system functions among them, and separating the user interface from the data.
The three-tier architecture includes: - Presentation layer: This is the level at which the user perceives information. - Application layer: This is the level where the tools for managing the recommendation system are located, as well as components such as setting the type of educational resources (EER) and educational components (OK), searching for EER and OK, displaying results, and generating reports.
- Data management layer: This is the level where data is physically stored, with subsystems for determining the type of EER and OK, analyzing EER and OK, generating results, and generating user reports.
The subsystem for determining the type of electronic educational resources and educational content allows the user to specify the type and select criteria that educational resources should meet. The subsystem includes a module for processing type assignment results, a database of types of educational resources, and a database of educational content criteria.
The subsystem for analyzing electronic educational resources and educational content consists of an OLAP (Online Analytical Processing) warehouse, databases of electronic educational resources and educational content, a data loading module, and a module for analyzing EER and OK.
The subsystem for generating results is intended for generating a recommendation ranking of EER and data visualization. It contains modules for calculating the recommendation ranking, building radar charts, and generating results.
The subsystem for generating reports is intended for generating reports on the analytics of queries for electronic resources and educational content. It includes a user profile database, a user query database, and a report generation module. The information technology functioning of the prototype educational content recommendation system involves the use of various technologies, algorithms, and methods for collecting, processing, and providing personalized recommendations to users.
The main idea of the prototype recommendation system is to collect expert evaluations for EER and educational content based on certain criteria. Based on these evaluations, the system builds a recommendation ranking of resources, ordered from most to least recommended.
Additionally, the system provides data visualization by creating a radar chart for each resource, where each segment represents the value of a criterion based on its importance. This allows users to assess which specific aspects each recommended resource corresponds to and make a more informed choice.
The functional purpose of the recommendation system in the context of evaluating educational content is to ensure objective assessment of developed methodological materials. This system facilitates convenient and efficient interaction between experts who have their own views on content evaluation and the toolkit that helps objectively consider multifaceted criteria.
By utilizing the evaluation scores based on established criteria and activating computations, the prototype recommendation system assists experts in conducting responsible and well-founded assessments of educational content. The recommendation system contains a database where many resources and associated information are stored, facilitating efficient selection and quick access to recommended rankings. Visualization of results through radar charts promotes understanding and comparison of content considering its characteristics. [13] Ye. Meleshko, S. Semenov, V. Khokh, Research of methods of building advisory systemson the internet Control, Navigation and Communication Systems 1 (2018) 131136. doi:10.26906/SUNZ.2018.1.131. [14] A. Esteban, A. Zafra, C. Romero, Helping university students to choose elective courses by using a hybrid multi-criteria recommendation system with genetic optimization, Knowledge-Based Systems 194 (2019). doi:10.1016/j.knosys.2019.105385. [15] M. P. L. Perera, A Review: Artificial Intelligent Approach for Enhancing Adaptability in an Adaptive E-Learning Environment, International Journal of Engineering and Advanced Technology 10(4) (2021) 37 42. doi: 10.35940/ijeat.D2297.0410421. [16] M. Elias, A. Oelen, M. Tavakoli, G. Kismihók, S. Auer, Quality Evaluation of Open
Educational Resources. EC-TEL 12315 (2020). doi:10.1007/978-3-030-57717-9_36. [17] J. Lin, H. Pu, Y. Li, J. Lian, Intelligent Recommendation System for Course Selection in Smart Education, Procedia Computer Science 129 (2018) 449-453. doi: 10.1016/j.procs.2018.03.023. [18] R. Zhang, Personalized Course Recommendation Method Based on Learner Interest Mining in Educational Big Data Environment, Scientific Programming 2022 (2022) 1 8. doi: 10.1155/2022/9943965. [19] H. Slimani, O. Hamal, N.-E. El Faddouli, S. Bennani, N. Amrous, Semantic recommendation system of digital educational resources, in: Proceedings of the 12th. International Conference on Intelligent Systems: Theories and Applications, Rabat Morocco, 2018, pp. 1 6. doi: 10.1145/3289402.3289513. [20] V. Yunchyk, Information Technologies for Educational Content Formation for E-Learning
Systems, Ph.D, thesis, Lviv Polytechnic National University, Lviv, Ukraine, 2023. [21] V. Pasichnyk et al., Model of the Recommender System for the Selection of Electronic Learning Resources, in: Proceedings of the 5th International Workshop on Modern Machine Learning Technologies and Data Science (MoMLeT+DS 2023), Ukraine, 2023, pp. 344-355. [22] S. Yatsyuk, V. Yunchyk, S. Mukutuyk, O. Duda, O., A. Fedonuyk, Application of the hierarchy analysis method for the choice of the computer mathematics system for the ITsphere specialists preparation, Journal of Physics: Conference Series 1840 (2021). doi:10.1088/1742-6596/1840/1/012065. [23] V. Yunchyk, A. Fedonuyk, M. Khomyak, S. Yatsyuk, S. Cognitive modeling of the learning process of training IT specialists, in: Proceedings of the 3rd International Workshop on Modern Machine Learning Technologies and Data Science (MoMLeT+DS 2021), Ukraine, 2021, pp. 141-150. [24] S. Yatsyuk, V. Yunchyk, T. Cheprasova, A. Fedonuyk. The Models of Data and Knowledge Representation in Educational System of Mathematical Training of IT-specialists, in: Proceedings of the 15th International Conference on Computer Sciences and Information Technologies (CSIT), 2020, pp. 269-272. doi:10.1109/CSIT49958.2020.9321899. [25] V. Yunchyk, Y. Fedoniuk, Results of developing the recommendation system for electronic educational resource selection, informatike 1 (2023), [Online]. Available: https://manazerskainformatika.sk/results-ofdeveloping-the-recommendation-system-for-electronic-educational-resource-selection/.