=Paper=
{{Paper
|id=Vol-3723/paper8
|storemode=property
|title=Expert assessment of educational content in IT specialists training process
|pdfUrl=https://ceur-ws.org/Vol-3723/paper8.pdf
|volume=Vol-3723
|authors=Volodymyr Pasichnyk,Nataliia Kunanets,Valentyna Yunchyk,Maria Khomyak,Anatolii Fedonyuk,Yurii Knysh
|dblpUrl=https://dblp.org/rec/conf/modast/PasichnykKYKFK24
}}
==Expert assessment of educational content in IT specialists training process==
https://ceur-ws.org/Vol-3723/paper8.pdf
Expert assessment of educational content in IT
specialists training process
Volodymyr Pasichnyk1, , Nataliia Kunanets1, , Valentyna Yunchyk2 ,
Maria Khomyak2, , Anatolii Fedonyuk2, and Yurii Knysh2,
1 Lviv Polytechnic National University, Stepana Bandery str. 12, Lviv, 79013, Ukraine
2 Lesya Ukrainka Volyn National University, 13 Volya Avenue, Lutsk, 43025, Ukraine
Abstract
In today's world, the importance of effective evaluation of educational content is increasing due to
the rapid pace of development in information technology and access to a large number of
educational resources. The study provides an example of an expert assessment of educational
content in the process of training IT specialists. Using the example of the educational program
"Computer Science and Information Technologies" at the bachelor's degree level, the main types of
educational content were identified. Quantitative characteristics of the educational content for IT
specialist s training were compiled into a table. Expert communities for assessing educational
content were identified. Calculations were made for expert assessments needed at each stage of the
training course.
In this context, there is a need for the development and implementation of recommendation
systems for evaluating educational content. An overview of the recommendation system for
evaluating educational content is proposed. The functional purpose of the recommendation system
in the context of evaluating educational content is to ensure objective assessment of developed
methodological materials. Potential advantages of implementing the recommendation system in the
educational process and methods of interaction with users are considered. A prototype of the
recommendation system is developed based on a three-tier architecture. Information technology
components used as the basis for building the recommendation system are implemented as a multi-
page web application. To visualize the results of evaluating educational content, an approach using
radar charts is considered. The study addresses the relevant scientific task of developing a
recommendation system for evaluating educational content for educational expert environments
that need to make decisions regarding the formation of quality educational content.
Keywords
Recommendation System, Educational Content, Expert Assessment, Electronic Learning Resources
1. Introduction
Currently, one of the pertinent areas of information technology application is in the field of
education. The desire to enhance learning processes and their increasing dependence on
various information resources generate the need for the development and implementation of
innovative teaching methods and tools, particularly electronic learning systems. A
MoDaST-2024: 6th International Workshop on Modern Data Science Technologies, May, 31 - June, 1, 2024, Lviv-Shatsk,
Ukraine
Corresponding author.
These authors contributed equally.
vpasichnyk@gmail.com (V. Pasichnyk); nek.lviv@gmail.com (N. Kunanets); uynchik@gmail.com (V. Yunchyk);
polekha@ukr.net (M. Khomyak); fedonyukanatan@gmail.com (A. Fedonuyk); yra.vasuliovu4@gmail.com
(Yu. Knysh)
0000-0001-9434-563X (V. Pasichnyk); 0000-0003-3007-2462 (N. Kunanets); 0000-0003-3500-1508 (V. Yunchyk);
0000-0002-9245-7993 (M. Khomyak); 0000-0003-0942-227X (A. Fedonuyk); 0009-0000-6237-6888 (Yu. Knysh)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
fundamental component of modern electronic learning systems is educational content, which
requires active updating and adaptation to constantly evolving needs.
The availability of Internet network resources and the expansion of electronic learning
system capabilities significantly emphasize the task of selecting quality and effective
educational resources. In the context of a vast array and saturation of educational materials,
the problem of choosing the most effective and high-quality educational resources becomes
particularly relevant.
The widespread use of electronic learning systems in educational processes actively
encourages developers of educational resources to create diverse, high-quality, and up-to-date
educational content. The increase in the dynamics and volumes of creating new educational
content often leads, in many cases, to a decrease in the quality of educational materials.
Educational information resources are typically formed without proper verification and
testing, which can pose challenges in determining the credibility and quality of resources. The
increase in the volume of educational materials generates the need for professional
assessment of quality and alignment with educational goals.
The assessment of educational content is a procedure typically carried out by experts in
the field of education. In higher education institutions, these experts usually include faculty
members, curriculum development groups, and pedagogical teams of faculty-level scientific
and methodological commissions. They also include scientific and technical councils of
institutes and universities, academic councils of faculties, institutes, and universities in expert
environments where the evaluation of educational content is collegial discussed and
conducted.
There is a need to analyze the possibilities of expert assessment of educational content and
to develop a recommendation system for assessing educational content. This recommendation
system implements the appropriate assessment methodology and the sequence of steps to be
taken professionally, promptly, and competently.
The purpose of the research is to analyze expert assessment of educational content,
develop and test models, methods, and components of information technology for building a
recommendation system for assessing educational content.
2. Analysis of literature sources
The utilization of modern information technologies in the processes of educational content
formation and evaluation is explored by scholars such as A. Burden [1], An. A. Galang [2],
R. C. Clark [3], R. Mayer [4], H. Kilinc [5], and others.
Research on the development of information technologies for educational content
formation based on artificial intelligence methods is dedicated to the works of I. Viznyuk [6],
K. Mamchur [7], M. Maryenko [8], M. Shyshkina [9], S. A. D. Popenici [10], A. Haleem [11],
and others.
Studies on the development and improvement of recommendation systems are addressed
by many domestic and foreign researchers, such as O. Veres [12], Ye. Meleshko [13],
C. Romero [14], R. Peres-Rodriguez [15], M. Elias [16], J. Lin [17], J. Zhang [18], H. Slimani
[19], and others.
In the extensive spectrum of analyzed research results, the utilization of recommendation
systems in various fields of activity is presented. However, insufficient attention is given to
the issues related to the creation and utilization of information technologies for building
recommendation systems for expert evaluation of electronic learning resources and
educational content. Such systems are necessary and can be effectively used by expert
communities, which regularly need to make decisions regarding the selection and provision of
recommendations for the use of new high-quality electronic learning resources.
According to research [20], an information system of a recommendation type is a
specialized information system that facilitates the implementation of basic information
processes to provide personalized recommendations to users.
3. Presentation of the main material
At Lesya Ukrainka Volyn National University, IT specialists are trained under the Computer
Science and Information Technologies educational program. The bachelor's degree curriculum
includes 6 educational components in the general training cycle, 30 educational components
in the professional training cycle, and 12 educational components in the elective cycle. The
professional training cycle of education seekers was analyzed for the necessary types of
educational content.
For the proper preparation of IT specialists within the Computer Science and Information
Technologies educational program at the bachelor's degree level, the following types of
educational content were identified:
K1 Lecture outlines on the topic;
K2 Guidelines for practical work;
K3 Guidelines for laboratory work;
K4 Module control work task sets;
K5 Test task sets;
K6 Guidelines for independent study;
K7 Guidelines for independent work;
K8 Guidelines for writing a term paper;
K9 Guidelines for internships;
K10 Guidelines for writing a qualification paper.
According to the curriculum of the Computer Science specialty (122), a different number of
units of educational content needs to be evaluated for each educational component. This takes
into account the specifics of the educational components and the number of hours allocated
for study (see Table 1).
The mechanism of recommending educational publications for printing and use in the
educational process involves consideration at the department level, during the meeting of the
faculty scientific and methodological commission. It also involves consideration at the
meeting of the university scientific and methodological council.
Faculty members from the department where the author works and which deals with this
issue review, discuss, evaluate, and recommend publications for printing.
Participants of the faculty scientific and methodological commission oversee the
educational publication to ensure compliance with current requirements and the quality of
publications intended for printing, as well as recommend them for use in the educational
process
Table 1
Quantitative characteristics of educational content for IT specialists training at Lesya
Ukrainka Volyn National University
Types of educational content Total units of
educational
content
39 40 2 2 4 87
18 18 1 1 2 40
18 18 1 37
1 course
26 35 4 2 67
18 18 3 39
13 17 3 1 34
17 17 2 2 38
34 44 4 2 4 88
18 18 2 1 39
18 18 3 39
22 23 2 1 48
2 course
18 19 1 3 1 42
14 17 2 33
15 19 2 36
15 19 2 36
1 1
17 23 2 3 45
20 20 2 42
20 24 1 3 48
3 course
15 18 2 35
15 18 2 35
1 1
1 1
1 1
15 20 2 1 38
15 20 1 36
15 20 2 1 38
15 18 2 1 36
4 course
18 18 2 38
1 1
1 1
1 1
1 1
The university scientific and methodological council reviews and recommends (or rejects)
materials submitted by authors to the educational department for printing. The process of
recommending educational publications must undergo evaluation by three expert
communities, with the number of individuals specified in Table 2.
Table 2
Expert Assessment Communities of Educational Content
Expert Assessment Communities Number of Individuals
The composition of the graduating department 12
Faculty Scientific and Methodological Commission 6
University Scientific and Methodological Council 24
Let * + be the set of expert communities, where { } is the set
of participants in the faculty scientific and methodological commission, { }
is the set of faculty members from the graduating department, { } is the set
of participants in the university scientific and methodological council.
Let ( ̅̅̅̅̅̅) be educational components, and { } be the set of types
of educational content. Then, the sum of units of educational content for is given by:
∑
Let be the sum of units of educational content to be evaluated for the 1st year, i.e.,
, be the sum for the 2nd year, i.e.,
, be the sum for the 3rd year, i.e.,
, be the sum for the 4th year, i.e.,
.
Then, for the first year, ∑ expert evaluations are needed, for the second year
∑ , for the third year ∑ and for the fourth year ∑ . In total, over
the entire study period evaluations are required, where:
∑ ∑
Tables 3-6 provide quantitative characteristics of expert evaluation of educational content
for IT specialist training for each year, and Figures 1-4 illustrate the quantitative expert
evaluation.
Table 3
The number of expert assessments conducted for the 1st year
Total Expert Communities
Educational
Evaluation
Components
Units 6 12 24
87 522 1044 2088
40 240 480 960
37 222 444 888
67 402 804 1608
39 234 468 936 Total Number of
34 204 408 816 Expert Evaluations
38 228 456 912 Conducted for 1st Year
Total 342 2052 4104 8208 14364
Figure 1: Expert evaluations conducted for the 1st year
Table 4
The number of expert assessments conducted for the 2nd year
Total Expert Communities
Educational
Evaluation
Components
Units 6 12 24
88 528 1056 2112
39 234 468 936
39 234 468 936
48 288 576 1152
42 252 504 1008
33 198 396 792 Total Number of
36 216 432 864 Expert Evaluations
36 216 432 864 Conducted for 2nd
1 6 12 24 Year
Total 362 2172 4344 8688 15204
Figure 2: Expert evaluations conducted for the 2nd year
Table 5
The number of expert assessments conducted for the 3rd year
Total Expert Communities
Educational
Evaluation
Components
Units 6 12 24
45 270 540 1080
42 252 504 1008
48 288 576 1152
35 210 420 840
35 210 420 840
1 6 12 24 Total Number of Expert
1 6 12 24 Evaluations Conducted
1 6 12 24 for 3rd Year
Total 208 1248 2496 4992 8736
Figure 3: Expert evaluations conducted for the 3rd year
Table 6
The number of expert assessments conducted for the 4th year
Total Expert Communities
Educational
Evaluation
Components
Units 6 12 24
38 228 456 912
36 216 432 864
38 228 456 912
36 216 432 864
38 228 456 912
1 6 12 24 Total Number of
1 6 12 24 Expert Evaluations
1 6 12 24 Conducted for 4th
1 6 12 24 Year
Total 190 1140 2280 4560 7980
Figure 4: Expert evaluations conducted for the 4th year
Therefore, for evaluating the educational content in the first year of IT specialists' training
14364 expert procedures are required. In the second year, this number slightly increases to
15204, while there is a decrease in the third year to 8736, as it includes coursework and
internships. On the fourth year, the number decreases further to 7980 and the total number of
expert procedures required for the entire training period is 46284 (Table 7, Figure 5).
Table 7
Number of expert evaluations conducted throughout the study period
Total Expert communities Number of
Courses evaluation expert
units 6 12 24 evaluations
I 342 2052 4104 8208 14364
II 362 2172 4344 8688 15204
III 208 1248 2496 4992 8736
IV 190 1140 2280 4560 7980
Total number of expert evaluations for the entire study period 46284
Figure 5: Expert evaluations conducted over the entire study period
Overall, it can be argued that the process of evaluating educational content is quite labor-
intensive and time-consuming. In the majority of situations, the processes of evaluating
electronic educational resources and educational content are usually carried out improperly or
not conducted at all. The developed and proposed information technology toolkit is aimed at
significantly improving, simplifying, and expediting the implementation of expert evaluation
processes for electronic educational resources and educational content in expert educational
environments.
The purpose of the prototype of the recommendation system for evaluating educational
content and resources is to provide users with personalized recommendations for selecting the
best electronic educational resources and educational content in specific educational
situations. The system aims to provide users with quality and relevant resources that meet
their needs and enhance the effectiveness of learning. [21].
The sphere of application of the recommendation system is the educational process. The
intended recommendation system is for expert environments of subject departments of
secondary educational institutions, pedagogical councils, cyclical commissions, pedagogical
collectives of departments, support groups for educational programs, scientific and
methodological commissions of faculties, scientific and methodological councils of institutes
and universities, scientific and technical councils of institutes and universities, academic
councils of faculties, institutes, and universities, overall for all expert communities that need
to make decisions regarding the selection and evaluation of electronic educational resources
and educational content.
The prototype of the recommendation system is developed based on a three-tier
architecture (Figure 6). This allowed for dividing the system into interconnected parts,
distributing system functions among them, and separating the user interface from the data.
The three-tier architecture includes:
- Presentation layer: This is the level at which the user perceives information.
- Application layer: This is the level where the tools for managing the recommendation
system are located, as well as components such as setting the type of educational resources
(EER) and educational components (OK), searching for EER and OK, displaying results, and
generating reports.
- Data management layer: This is the level where data is physically stored, with
subsystems for determining the type of EER and OK, analyzing EER and OK, generating
results, and generating user reports.
The subsystem for determining the type of electronic educational resources and
educational content allows the user to specify the type and select criteria that educational
resources should meet. The subsystem includes a module for processing type assignment
results, a database of types of educational resources, and a database of educational content
criteria.
The subsystem for analyzing electronic educational resources and educational content
consists of an OLAP (Online Analytical Processing) warehouse, databases of electronic
educational resources and educational content, a data loading module, and a module for
analyzing EER and OK.
The subsystem for generating results is intended for generating a recommendation ranking
of EER and data visualization. It contains modules for calculating the recommendation
ranking, building radar charts, and generating results.
The subsystem for generating reports is intended for generating reports on the analytics of
queries for electronic resources and educational content. It includes a user profile database, a
user query database, and a report generation module.
Figure 6: Structure of the educational content recommendation system
The information technology functioning of the prototype educational content
recommendation system involves the use of various technologies, algorithms, and methods for
collecting, processing, and providing personalized recommendations to users.
The main idea of the prototype recommendation system is to collect expert evaluations for
EER and educational content based on certain criteria. Based on these evaluations, the system
builds a recommendation ranking of resources, ordered from most to least recommended.
Additionally, the system provides data visualization by creating a radar chart for each
resource, where each segment represents the value of a criterion based on its importance. This
allows users to assess which specific aspects each recommended resource corresponds to and
make a more informed choice.
4. Conclusions
The functional purpose of the recommendation system in the context of evaluating
educational content is to ensure objective assessment of developed methodological materials.
This system facilitates convenient and efficient interaction between experts who have their
own views on content evaluation and the toolkit that helps objectively consider multifaceted
criteria.
By utilizing the evaluation scores based on established criteria and activating
computations, the prototype recommendation system assists experts in conducting
responsible and well-founded assessments of educational content. The recommendation
system contains a database where many resources and associated information are stored,
facilitating efficient selection and quick access to recommended rankings. Visualization of
results through radar charts promotes understanding and comparison of content considering
its characteristics.
References
[1] G. Stebbings, C. Mackintosh, A. Burden, D. Sims, Improving Student Progression in
Distance Learning Using Synchronous Webinars, Bridges and Mediation in Higher
Distance Education 1344 (2021) 315 323. doi: 10.1007/978-3-030-67435-9_24.
[2] R. Elcullada Encarnacion, A. A. Galang, B. J. Hallar, The Impact and Effectiveness of E-
Learning on Teaching and Learning, International Journal of Computing Sciences
Research 5 (2021) 383 397. doi: 10.25147/ijcsr.2017.001.1.47.
[3] R. C. Clark, R. E. Mayer, E-Learning and the Science of Instruction: Proven Guidelines for
Consumers and Designers of Multimedia Learning, Canada, 2016. doi:
10.1002/9781119239086.
[4] R. Mayer, Multimedia Learning (3rd. ed.), Cambridge University Press, 2020. doi:
10.1017/9781316941355.
[5] H. Kilinc, H. Altinpulluk, Discussion Forums as a Learning Material in Higher Education
Institutions, International Journal of Higher Education Pedagogies 2 (2021). doi:
10.33422/ijhep.v2i1.25.
[6] I. Viznyuk, N. Buglay, A. Polishchuk, V. Kylyvnyk, The Use of Artificial Intelligence in
Education, Modern Information Technologies and Innovative Teaching Methods in
Training of Specialists Methodology Theory Experience Problems 59 (2021) 14 22.
[7] K. Mamchur, Features of Using ChatGPT for the Development of Professional Competence
of Scientific and Pedagogical Workers of Higher Military Educational Institutions,
Military Education 47 (2023) 154 160. doi: 10.33099/2617-1783/2023-47/154-160.
[8] M. Maryenko, V. Kovalenko, Artificial Intelligence and Open Science in Education, Phisical
and Mathematical Education 38 (2023) 48 53. doi: 10.31110/2413-1571-2023-038-1-007.
[9] M. Shyshkina, Y. Nosenko, Promising Technologieswith Elements of AI for Professional
Developmentof Teaching, Phisical and Mathematical Education 38 (2023) 66 71. doi:
10.31110/2413-1571-2023-038-1-010.
[10] S. A. D. Popenici, S. Kerr, Exploring the impact of artificial intelligence on teaching and
learning in higher education, Research and Practice in Technology Enhanced Learning 12
(2017). doi: 10.1186/s41039-017-0062-8.
[11] M. Javaid, A. Haleem, R. P. Singh, S. Khan, I. H. Khan, Unlocking the opportunities
through ChatGPT Tool towards ameliorating the education system, BenchCouncil
Transactions on Benchmarks, Standards and Evaluations 3(2) (2023). doi:
10.1016/j.tbench.2023.100115.
[12] P. Veres, O. Kots, Y. Levus, O. Vlasenko, Recommendation System for Leisure Time-
Management in Quarantine Conditions, in: Proceedings of the 4th International
Workshop on Modern Machine Learning Technologies and Data Science (MoMLeT+DS
2021), Ukraine, 2022, pp. 263-282.
[13] Ye. Meleshko, S. Semenov, V. Khokh, Research of methods of building advisory
systemson the internet Control, Navigation and Communication Systems 1 (2018) 131-
136. doi:10.26906/SUNZ.2018.1.131.
[14] A. Esteban, A. Zafra, C. Romero, Helping university students to choose elective courses
by using a hybrid multi-criteria recommendation system with genetic optimization,
Knowledge-Based Systems 194 (2019). doi:10.1016/j.knosys.2019.105385.
[15] M. P. L. Perera, A Review: Artificial Intelligent Approach for Enhancing Adaptability in
an Adaptive E-Learning Environment, International Journal of Engineering and
Advanced Technology 10(4) (2021) 37 42. doi: 10.35940/ijeat.D2297.0410421.
[16] M. Elias, A. Oelen, M. Tavakoli, G. Kismihók, S. Auer, Quality Evaluation of Open
Educational Resources. EC-TEL 12315 (2020). doi:10.1007/978-3-030-57717-9_36.
[17] J. Lin, H. Pu, Y. Li, J. Lian, Intelligent Recommendation System for Course Selection in
Smart Education, Procedia Computer Science 129 (2018) 449-453. doi:
10.1016/j.procs.2018.03.023.
[18] R. Zhang, Personalized Course Recommendation Method Based on Learner Interest
Mining in Educational Big Data Environment, Scientific Programming 2022 (2022) 1 8.
doi: 10.1155/2022/9943965.
[19] H. Slimani, O. Hamal, N.-E. El Faddouli, S. Bennani, N. Amrous, Semantic
recommendation system of digital educational resources, in: Proceedings of the 12th.
International Conference on Intelligent Systems: Theories and Applications, Rabat
Morocco, 2018, pp. 1 6. doi: 10.1145/3289402.3289513.
[20] V. Yunchyk, Information Technologies for Educational Content Formation for E-Learning
Systems, Ph.D, thesis, Lviv Polytechnic National University, Lviv, Ukraine, 2023.
[21] V. Pasichnyk et al., Model of the Recommender System for the Selection of Electronic
Learning Resources, in: Proceedings of the 5th International Workshop on Modern
Machine Learning Technologies and Data Science (MoMLeT+DS 2023), Ukraine, 2023, pp.
344-355.
[22] S. Yatsyuk, V. Yunchyk, S. Mukutuyk, O. Duda, O., A. Fedonuyk, Application of the
hierarchy analysis method for the choice of the computer mathematics system for the IT-
sphere specialists preparation, Journal of Physics: Conference Series 1840 (2021).
doi:10.1088/1742-6596/1840/1/012065.
[23] V. Yunchyk, A. Fedonuyk, M. Khomyak, S. Yatsyuk, S. Cognitive modeling of the learning
process of training IT specialists, in: Proceedings of the 3rd International Workshop on
Modern Machine Learning Technologies and Data Science (MoMLeT+DS 2021), Ukraine,
2021, pp. 141-150.
[24] S. Yatsyuk, V. Yunchyk, T. Cheprasova, A. Fedonuyk. The Models of Data and Knowledge
Representation in Educational System of Mathematical Training of IT-specialists, in:
Proceedings of the 15th International Conference on Computer Sciences and Information
Technologies (CSIT), 2020, pp. 269-272. doi:10.1109/CSIT49958.2020.9321899.
[25] V. Yunchyk, Y. Fedoniuk, Results of developing the recommendation system for
electronic educational resource selection,
informatike 1 (2023), [Online]. Available: https://manazerskainformatika.sk/results-of-
developing-the-recommendation-system-for-electronic-educational-resource-selection/.