Evaluating the Quality of Education: Application of Black, Gray, and White Box Methods Artem Artyukhov1,2,3, , Iurii Volk3, , Oleksandr Dluhopolskyi2,4, , Nadiia Artyukhova1,3, , Olena Zelikovska5, 1 University of Economics in Bratislava, Dolnozemskรก str. 1, 85-235, Bratislava, Slovakia 2 WSEI University, Projektowa str. 4, 20-209, Lublin, Poland 3 Sumy State University, Kharkivska str. 116, 40-000, Sumy, Ukraine 4 West Ukrainian National University, Lvivska str. 11, 46-027, Ternopil, Ukraine 5 Taras Shevchenko National University of Kyiv, Volodymyrska str. 60, 01-033, Kyiv, Ukraine Abstract that includes initial data, system parameters, control parameters, disturbing parameters, and output parameters. Utilizing analogies to describe such a dynamic system enables a comprehensive understanding of the internal and external factors that influence the system and allows for the prediction of its response to changes in power dynamics. To accurately assess the quality of the education system, it is crucial to consider the relationship between its socio-economic impact and several key criteria: the system s purposefulness, hierarchical structure, interdependence with the external environment, level of autonomy and openness, reliability, and dimensional characteristics. An algorithm was developed, informed by data from the black, gray, and white box methods, to facilitate the transition between models aimed at enhancing the quality of education. The description of the system advances sequentially from the black box model to the gray box model, and ultimately to the white box model. Each model is capable of independently representing a specific set of input and output parameters, but the level of determinism in the descriptive process increases as one progresses from one model to the next. However, the degree of determinism in the descriptive process increases with the transition from the previous model to the next one. Keywords 1 box model, quality of education, black box method, gray box method, white box method 1. Introduction In the field of engineering, physics, and other scientific has been widely employed to characterize material systems. A technical system is defined as a system that exhibits the ability to dynamically change over time in response to external & control parameters, and other influencing factors. Such systems are purposefully designed tasked with executing a predetermined set of functions. The functioning of such systems is described using a set of mathematical formulas, known as a mathematical model that can be supplemented by experimental findings in some instances. A system can be deterministic when external influences, especially those that are random and lack a discernible pattern, are absent. However, applying the concept of a technical system to intangible entities presents certain challenges: the principles governing such entities can be subjective and stochastic, and they are often subject to numerous external disturbances. Despite these challenges, drawing parallels between dynamic systems and intangible entities enables a comprehensive description of both internal and external influences on a dynamic system, facilitating the prediction explaining the operation of dynamic education quality systems. Additionally, various methodologies Information Technology and Implementation (IT&I-2024), November 20-21, 2024, Kyiv, Ukraine Corresponding author. These authors contributed equally. a.artyukhov@pohnp.sumdu.edu.ua (A. Artyukhov); y.volk@mss.sumdu.edu.ua (I. Volk); dlugopolsky77@gmail.com (O. Dluhopolskyi); n.artyukhova@biem.sumdu.edu.ua (N. Artyukhova); o.zelikovska@knu.ua (O. Zelikovska) 0000-0003-1112-6891 (A. Artyukhov); 0000-0002-0262-762X (I. Volk); 0000-0002-2040-8762 (O. Dluhopolskyi); 0000- 0002-2408-5737 (N. Artyukhova); 0000-0002-6559-9101 (O. Zelikovska) 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop ceur-ws.org ISSN 1613-0073 242 Proceedings can be employed to explore causal relationships within the dynamic education system, each differing in the openness of initial data, system parameters, control parameters, disturbances, and output parameters. These methodologies are consistent with the theory of testing technical systems and software products, commonly referred to as black, gray, and white box methods. When assessing the quality of an education system, it is crucial to examine the interplay between its socio-economic impact and several key criteria: the system s purposefulness, its hierarchical structure, the interdependence between the system and its external environment, the degree of autonomy and openness, the system s reliability, and its dimensional characteristics. 2. Literature review The application of black, gray, and white box models for describing dynamic systems extends beyond technical engineering or computer systems and has been effectively utilized in various fields, including cybernetics, economics, and other areas of knowledge. Andersson & Johansson [1] model the market interaction when each production and distribution unit have a positive effect on quality improvement and corresponding pricing opportunities due to information about market variables at individual locations. Boumans [2] & Kasianiuk [3] present two models of system identification white box and black box as useful tools that help to understand the processes of self- organization inside and outside organizations. Ji & Luo [4] analyzed the phenomenon of ecological economics as a black box. In articles [5-11] authors provide technique of black box vs. white box testing based on latest advancements in different domains. Although the box method has been employed in the education sector, its use has primarily focused on specific applications within the educational process, such as training, rather than on analyzing management (control) processes. Shkarlet et al. [12] describe the development of the Quadruple Helix model for solving the problems of the information economy. The studies [13-15] focus attention on SDG (Sustainable Development Goals) in the context of environmental education. Papers [16-20] are devoted to the analysis of the impact of modern technologies on the quality of higher education in Ukraine. within the educational sphere [21-30] and the increasing emphasis on achieving the SDG in education [31-38]. Figure 1: Results of b Source: composed by authors by VOSviewer in Scopus database 243 Figure 2: Results of b , Source: composed by authors by VOSviewer in Scopus database A bibliometric analysis of literature sources obtained from the scientometric Scopus database (Figures 1-2, using the VOSviewer bibliometric analysis tool) reveals a growing interest in applying approaches that describe intangible systems by drawing analogies with technical systems. [9-11]. It is important to clarify that in this context, the term quality assurance refers to software testing, not education [19-22]. The framework of unit testing quality control quality assurance can effectively establish a causal relationship between -economic development. Simultaneously, it is crucial to clearly define the prerequisites for maintaining the quality of the education system, particularly concerning the input of initial data and the prediction of outcomes. Thus, on the basis of bibliometric analysis and literature review, it is possible to formulate the purpose of the article evaluation the quality of the education system by conceptualizing it through the black, gray, and white box methods, drawing analogies between physical (engineering) systems and intangible educational systems. In this context, it is imperative to assess how the design of the education system impacts the socio-economic development of an organization, region, or country. 3. Methodology Scopus scientometric database [39] was used for bibliometric analysis. Bibliometric analysis tool VOSviewer [40]. The main method of the study was bibliometric analysis using the VOSviewer software tool to analyze the relationships between different categories and build maps to visualize the interconnectedness between them in publications indexed by the Scopus database. The bibliographic analysis was carried out using the keywords: technical system, gray box, black box, white box for 1991-2023. An algorithm for developing the facilitation of the transition between various models is used to assess the quality of the education system was constructed based on data obtained from the analysis of the black, gray, and white box methods. The algorithm is presented in the diagram in Figure 3. The diagram (Figure 3) is accompanied by the following detailed description of each level within the three-level box model framework, specifically in relation to education system quality: 1) Black box: only inputs (๐‘‹) and outputs (๐‘Œ); 2) Gray box: inputs (๐‘‹), control parameters (๐‘ˆ), and outputs (๐‘Œ); 244 3) White box: inputs (๐‘‹), control parameters (๐‘ˆ), system parameters (๐พ), disturbing parameters (๐‘), and outputs (๐‘Œ). Figure 3: Box model framework of quality of education system: X input parameters; Y output parameters; Z disturbing parameters; U control parameters; system parameters Source: original research The system is characterized by a sequential progression from the black box model to the gray box model, and ultimately to the white box model. Each of these models can operate autonomously and effectively representing a specific set of input and output parameters. However, as the transition is made from the simpler to the more complex models, there is a corresponding increase in the determinism of the description process. Such layered approach enables progressively deeper insights into the internal structure and influencing factors of the system. 4. Results s economic growth and the surrounding region. Output parameters (Y) include: the ranking of the university s educational programs among employers; the average salary of graduates from the educational programs; the career advancement of graduates of the educational programs. Input parameters (X) consist of: the range of educational programs offered; the availability of state- funded training opportunities for students; the cost of tuition; the presence of competing educational institutions within the region; the availability of practical training and internships opportunities; the list of potential employers. 245 Disturbing parameters (Z) refer to external influences, including: changes in the policies of the Ministry of Education and Science of Ukraine, particularly those related to the financing of the activities; the impact of military activities on the national economy, particularly in war-affected regions. The parameters of the K system can be represented as characteristics of the educational program and its surrounding environment, especially when evaluated against similar educational programs through benchmarking (Figure 4) or by using comparative data from rating agencies (Figure 5) such as QS World University Rankings [41], World University Rankings [42], Academic Ranking of World Universities [43] etc. To evidence the functionality of the proposed model works, we will apply assumptions, the results of which can be obtained through a survey. 100 100 100 100 100 100 100 100 100 100 90 90 80 71 70 57 60 50 44 35 Fact Target Figure 4: Outline of the educational program and its environment (benchmarking of educational programs, visualization, random data) Source: original research Key observation from Figure 4: 1. Learning through research: the target reached 90% (good result). 2. Transparency and publicity: the target reached the lower level of 57%. 3. Internal quality assurance of the educational program: the target reached 71%, indicating strong internal quality assurance overall, though one perspective sees room for improvement. 4. Educational environment and material resources: the target reached a lower level of 35%, suggesting significant concerns about the educational environment and resources. 5. Human resources: the target reached 44%, indicating a perceived difference in the adequacy of human resources. 6. Control measures, evaluation of students, and academic integrity: the target reached 50%, showing a moderate gap. 246 7. Teaching and learning in the educational program: the target reached 60%, pointing to a discrepancy in perceived effectiveness in teaching and learning practices. 8. Access to the educational program and recognition of learning outcomes: the target reached 90% (good result). 9. The structure and content of the educational program: the target reached 80%, showing that the structure and content are viewed positively overall, though one perspective notes some areas for enhancement. 10. Design and objectives of the educational program: the target reached 70%. Considering the above, some aspects such as transparency, human resources, and the educational environment being perceived as needing improvement. 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 3 2 2 Fact Target Figure 5: Outline of the educational program and its environment (0 the parameter is not applied; 1 parameter mismatch; 2 critical remarks that can be eliminated; 3 compliance of the parameter; 4 compliance of the innovative parameter) Source: original research Key observations from the Figure 5: 1. Educational environment and material resources: high or maximum rating (4 from 4). 2. Design and objectives of the educational program: high or maximum rating (4 from 4). 3. Transparency and publicity: high or maximum rating (4 from 4). 4. Control measures, evaluation of students and academic integrity: high or maximum rating (4 from 4). 5. Internal quality assurance of the educational program: 3 of 4 points (good level). 6. Teaching and learning in the educational program: 3 of 4 points (good level). 247 7. Access to the educational program and recognition of learning outcomes: 3 of 4 points (good level). 8. The structure and content of the educational program: 3 of 4 points (good level). 9. Human resources: 2 of 4 points (average level). 10. Learning through research: 2 of 4 points (average level). The evaluation of various aspects of the educational program reveals that several areas, including the educational environment and material resources, design and objectives of the educational program, transparency and publicity, and control measures and academic integrity, received the highest possible rating. Other areas, such as internal quality assurance, teaching and learning, access to the program and recognition of learning outcomes, and the structure and content of the program, were rated at a good level. However, human resources and learning through research were rated at an average level. Parameters of control U can be displayed on a radial diagram with an assessment of the degree of the impact on the output parameter on a scale from 1 to 10 points (Figure 6). government funding 10 8 quality of infrastructure grant funding 6 4 2 0 funding from the creation of popularity of the educational developments and the program provision of services ranking position international partnership Figure 6: Evaluation of the impact level of the control parameter on the output parameter (visualization, random data) Source: original research Key observation form Figure 6: 1. Government funding: the score for government funding is at 5, indicating a moderate level of government support. 2. Grant funding: this parameter is also rated at 5, suggesting a similar level of support from grants as from government funding. 3. Funding from the creation of developments and provision of services: this parameter has a 7 score. 4. International partnership: this parameter is rated very low 4, suggesting that international partnerships are limited. 6. Ranking position: the ranking position is rated around 6. 7. Popularity of the educational program: this parameter has an 8 score (the higher result). 8. Quality of infrastructure: this parameter has a higher score of 5. Overall, while the program shows strengths in areas such as service-based funding and popularity, it has moderate government and grant support, and there is a clear need for improvement in international partnerships. 5. Conclusions The proposed algorithm for the sequential description of the quality of education system and its socio-economic impact allows for several key outcomes: 1) establishing a clear set of indicators that 248 define the influence of input, control, and system parameters on output parameters, thereby enabling the prediction of their changes; 2) developing mechanisms to enhance the system by adjusting control parameters to increase the value of output parameters; 3) evaluating the current state of system parameters and identifying potential areas for improvement; 4) predicting the system's behavior under the influence of external disturbances; 5) creating a roadmap to achieve desired output parameters at operational (situational), tactical, and strategic levels. The novelty of this study lies in the application of the technical system approach typically used in engineering and physical sciences to the assessment of intangible systems, specifically the quality of education. Through the deployment of the black, gray, and white box models, the study innovatively conceptualizes the education system as a dynamic entity with inputs, control parameters, and external disturbances, allowing for a systematic evaluation of its impact on socio- economic development. Aknowledgments -03-V01- -03-V04-00522/2024/VA) and a grant from the Ministry of Education and Science of Ukraine under project Modeling and forecasting of socio- economic consequences of higher education and science reforms in wartime (No. 0124U000545). Declaration on Generative AI The authors have not employed any Generative AI tools. References [1] A. E. Andersson, B. Johansson, Inside and outside the black box: organization of interdependencies, The Annals of Regional Science 61 (2018) 501 516. doi:10.1007/s00168-018- 0886-1. [2] M. J. Boumans, Understanding in economics: gray-box models, in: H. W. de Regt,S. Leonelli, K. Eigner (Eds.), Scientific Understanding: Philosophical Perspectives, University of Pittsburgh Press, 2009, pp. 210 229. doi:10.2307/j.ctt9qh59s.14. [3] K. Kasianiuk, White box, black box and self-organization. A system-to-environment approach to leadership, Kybernetes 45(1) (2016) 1 16. doi:10.1108/K-02-2015-0057. [4] X. Ji, Z. Luo, Opening the black box of economic processes: ecological economics from its biophysical foundation to a sustainable economic institution, The Anthropocene Review 7(3) (2020) 231 247. doi:10.1177/2053019620940753. [5] R. Bierig. S. Brown, E. Galvรกn, J. Timoney, Black-Box and White-Box Testing, in: Essentials of Software Testing, Cambridge University Press, Cambridge, UK, 2021, pp. 141 164. doi:10.1017/9781108974073. [6] I. Komargodski, M. Naor, E. Eylon Yogev, White-box vs. black-box complexity of search problems: Ramsey and graph property testing, Journal of the ACM 34 (2019). doi:10.1145/3341106. [7] D. Wintana, D. Pribadi, M. Y. Nurhadi, Analisis Perbandingan Efektifitas White-Box Testing dan Black-Box Testing. Jurnal Larik Ladang Artikel Ilmu Komputer 2(1) (2022) 8 16. doi:10.31294/larik.v2i1.1382. [8] M. M. Syaikhuddin, C. Anam, A. R. Rinaldi, M. E. Conoras, Conventional software testing using white box method, Kinetik 3(1) (2018) 65 72. doi:10.22219/kinetik.v3i1.231. [9] M. Trucano, Open data, closed algorithms, and the black box of education, 2016. URL: https://blogs.worldbank.org/en/education/open-data-closed-algorithms-and-black-box- education. [10] X. D. Yu, M. Y. Zhang, M. Q. Zhu, K. H. Xu, Q. C. Xiang, Research on gray-box testing methods for software fault injection, Applied Mechanics and Materials (2014) 543 547. doi:10.4028/www.scientific.net/AMM.543-547.3360. 249 [11] V. Hassija, V. Chamola, A. Mahapatra, A. Singal, D. Goel, K. Huang, S. Scardapane, I. Spinelli, M. Mahmud, A. Hussain, Interpreting black-box models: a review on explainable artificial intelligence. Cognitive Computation 16(1) (2024) 45 74. doi:10.1007/s12559-023-10179-8. [12] S. Shkarlet, N. Kholiavko, M. Dubyna, O. Zhuk, Innovation, education, research components of the evaluation of information economy development (as exemplified by Eastern partnership countries), Marketing and Management of Innovations 1 (2019) 70 83. doi:10.21272/mmi.2019.1- 06. [13] W. T. Fang, A. Hassan, B. LePage, The living environmental education: Sound science toward a cleaner, safer, and healthier future. Springer, 2023. doi:10.1007/978-981-19-4234-1. [14] M. Mazurkiewicz, O. Liuta, K. Kyrychenko, Internal quality assurance system for the higher education: experience of Ukraine and Poland, Business Ethics and Leadership 1(4) (2017) 74 83. doi:10.21272/bel.1(4).74-83.2017. [15] A. Artyukhov, I. Volk, A. Surowiec, S. Skrzypek-Ahmed, K. Bliumska-Danko, O. Dluhopolskyi, V. Shablystyi, Quality of education and science in the context of Sustainable Development Goals from Millennium Goals to Agenda 2030: factors of innovation activity and socio-economic impact, Sustainability 14 (2022) 11468. doi:10.3390/su141811468. [16] A. Artyukhov, I. Volk, O. Dluhopolskyi, Immersive university model: a tool to increase higher education competitiveness, Sustainability 15 (2023) 7771. doi:10.3390/su15107771. [17] I. Skliar, Towards the assurance of transparency and quality of higher education in Ukraine: National Qualification Framework, Business Ethics and Leadership 2(1) (2018) 96 105. doi:10.21272/bel.2(1).96-105.2018. [18] L. Savga, O. Krykliy, K. Kyrychenko, The role of internal and external stakeholders in higher education system in Ukraine, Business Ethics and Leadership 2(1) (2018) 32 43. doi:10.21272/bel.2(1).32-43.2018. [19] V. Okulich-Kazarin, A. Artyukhov, L. Skowron, N. Artyukhova, O. Dluhopolskyi, W. Cwynar, Sustainability of higher education: study of student opinions about the possibility of replacing teachers with AI technologies. Sustainability 16 (2024) 55. doi:10.3390/su16010055. [20] H. Polianovskyi, T. Zatonatska, O. Dluhopolskyi, I. Liutyi, Digital and technological support of distance learning at universities under COVID-19 (case of Ukraine), Revista Romaneasca pentru Educatie Multidimensionala 13(4) (2021) 595-613. doi:10.18662/rrem/13.4/500 [21] C. Yeung, Economic and trade information: Hong Kong. Data and profiles market profiles, 2024. URL: https://research.hktdc.com/en/article/MzIwNjkzNTY5. [22] I. Georgiadou, A. Vlachou, P. Stavroussi, Quality of life and vocational education service quality in students with intellectual disability. International Journal of Developmental Disabilities 68(5) (2021) 681 691. doi:10.1080/20473869.2021.1887435. [23] S. Shkarlet, N. Kholiavko, M. Dubyna, Information economy: management of educational, innovation, and research determinants, Marketing and Management of Innovations, 3 (2019) 126 141. doi:10.21272/mmi.2019.3-10. [24] L. Matos, K. Kasztelnik, Transformational educational leadership and the innovative strategies engaging online faculty for the excellent teaching performance in the United States, Business Ethics and Leadership 5(1) (2021) 6 21. doi:10.21272/bel.5(1).6-21.2021. [25] F. M. Reimers, The sustainable development goals and education, achievements and opportunities. International Journal of Educational Development 104 (2024) 102965. doi:10.1016/j.ijedudev.2023.102965. [26] A. F. Pipa, K. Rasmussen, K. Pendrak, The state of the Sustainable Development Goals in the United States. Brookings Institution, 2022. URL: https://www.brookings.edu/articles/the-state- of-the-sustainable-development-goals-in-the-united-states. [27] M. Young, The Sustainable Development Goals look shaky but here are five ways to deliver. International Alert, 2019. URL: https://www.international-alert.org/blogs/sustainable- development-goals-look-shaky-here-are-five-ways-deliver. [28] S. J. Klees, Why SDG 4 and the other SDGs are failing and what needs to be done. International Journal of Educational Development 104 (2024) 102946. doi:10.1016/j.ijedudev.2023.102946. [29] J. Sachs, G. Lafortune, C. Kroll, G. Fuller, F. Woelm, From crisis to sustainable development: The SDGs as roadmap to 2030 and beyond. Sustainable Development Report 2022. Cambridge 250 University Press, Cambridge, UK 2022. URL: https://sdgtransformationcenter.org/reports/sustainable-development-report-2022. [30] of migration on forecasting budget expenditures on education: the sustainability context. Sustainability 15 (2023) 15473. doi:10.3390/su152115473. [31] O. Liuta, S. Lieonov, A. Artyukhov, M. Sushko-Bezdenezhnykh, O. Dluhopolskyi, Student survey as a tool for quality assurance in higher education: the case of Ukrainian university, Naukovyi Visnyk Natsionalnoho Hirnychoho Universytetu 4 (2021) 158 164. doi:10.33271/nvngu/2021- 4/158. [32] L. Olorogun, N. Yunusa, H. G. Audu, A.A. Mohammed, Management of educational innovations: of Innovations 2 (2018) 69 78. doi:10.21272/mmi.2018.2-06. [33] K. Upulee, J.M. Bieman, Testing scientific software: a systematic literature review, Information and software technology, 56(10) (2014) 1219-1232. doi:10.1016/j.infsof.2014.05.006. [34] A. Rauf, E.M. Reddy, Software test automation: an algorithm for solving system management automation problems, Procedia Computer Science, 46 (2015) 949 956. doi:10.1016/j.procs.2015.01.004. [35] A. Arcuri, Teaching software testing in an algorithms and data structures course, 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), 1 (2020) 419 424. doi:10.1109/ICSTW50294.2020.00075. [36] A. Bosworth, Metaโ€™s progress in augmented and virtual reality. Meta, 2022. URL: https://about.fb.com/news/2022/12/metas-progress-in-augmented-and-virtual-reality. [37] X. Muianga, The role of ICT in the shift towards student-centred learning in higher education: Eduardo Mondlane University, Mozambique: a case study. PhD thesis. Stockholm University, Stockholm, Sweden, 2019. [38] P. Breen, Developing educators for the digital age: a framework for capturing knowledge in action. London: University of Westminster Press, London, UK, 2018. doi:10.16997/book13. [39] Scopus, 2024. URL: https://www.scopus.com. [40] VOSviewer, 2024. URL: https://www.vosviewer.com. [41] QS World University Rankings, 2024. URL: https://www.topuniversities.com. [42] World University Rankings, 2024. URL: https://www.timeshighereducation.com/world- university-rankings. [43] Academic Ranking of World Universities, 2024. URL: https://www.shanghairanking.com/rankings/arwu/2024. 251