=Paper=
{{Paper
|id=Vol-3909/Paper_19.pdf
|storemode=property
|title=Evaluating the Quality of Education: Application of Black, Gray, and White Box Methods
|pdfUrl=https://ceur-ws.org/Vol-3909/Paper_19.pdf
|volume=Vol-3909
|authors=Artem Artyukhov,Iurii Volk,Oleksandr Dluhopolskyi,Nadiia Artyukhova,Olena Zelikovska
|dblpUrl=https://dblp.org/rec/conf/iti2/ArtyukhovVDAZ24
}}
==Evaluating the Quality of Education: Application of Black, Gray, and White Box Methods==
https://ceur-ws.org/Vol-3909/Paper_19.pdf
Evaluating the Quality of Education: Application of Black,
Gray, and White Box Methods
Artem Artyukhov1,2,3, , Iurii Volk3, , Oleksandr Dluhopolskyi2,4, , Nadiia Artyukhova1,3, ,
Olena Zelikovska5,
1
University of Economics in Bratislava, Dolnozemskรก str. 1, 85-235, Bratislava, Slovakia
2
WSEI University, Projektowa str. 4, 20-209, Lublin, Poland
3
Sumy State University, Kharkivska str. 116, 40-000, Sumy, Ukraine
4
West Ukrainian National University, Lvivska str. 11, 46-027, Ternopil, Ukraine
5
Taras Shevchenko National University of Kyiv, Volodymyrska str. 60, 01-033, Kyiv, Ukraine
Abstract
that includes initial data, system
parameters, control parameters, disturbing parameters, and output parameters. Utilizing analogies to
describe such a dynamic system enables a comprehensive understanding of the internal and external factors
that influence the system and allows for the prediction of its response to changes in power dynamics. To
accurately assess the quality of the education system, it is crucial to consider the relationship between its
socio-economic impact and several key criteria: the system s purposefulness, hierarchical structure,
interdependence with the external environment, level of autonomy and openness, reliability, and
dimensional characteristics. An algorithm was developed, informed by data from the black, gray, and white
box methods, to facilitate the transition between models aimed at enhancing the quality of education. The
description of the system advances sequentially from the black box model to the gray box model, and
ultimately to the white box model. Each model is capable of independently representing a specific set of
input and output parameters, but the level of determinism in the descriptive process increases as one
progresses from one model to the next. However, the degree of determinism in the descriptive process
increases with the transition from the previous model to the next one.
Keywords 1
box model, quality of education, black box method, gray box method, white box method
1. Introduction
In the field of engineering, physics, and other scientific
has been widely employed to characterize material systems. A technical system is defined as a system
that exhibits the ability to dynamically change over time in response to external & control parameters,
and other influencing factors. Such systems are purposefully designed tasked with executing a
predetermined set of functions. The functioning of such systems is described using a set of
mathematical formulas, known as a mathematical model that can be supplemented by experimental
findings in some instances.
A system can be deterministic when external influences, especially those that are random and lack
a discernible pattern, are absent. However, applying the concept of a technical system to intangible
entities presents certain challenges: the principles governing such entities can be subjective and
stochastic, and they are often subject to numerous external disturbances. Despite these challenges,
drawing parallels between dynamic systems and intangible entities enables a comprehensive
description of both internal and external influences on a dynamic system, facilitating the prediction
explaining the operation of dynamic education quality systems. Additionally, various methodologies
Information Technology and Implementation (IT&I-2024), November 20-21, 2024, Kyiv, Ukraine
Corresponding author.
These authors contributed equally.
a.artyukhov@pohnp.sumdu.edu.ua (A. Artyukhov); y.volk@mss.sumdu.edu.ua (I. Volk); dlugopolsky77@gmail.com (O.
Dluhopolskyi); n.artyukhova@biem.sumdu.edu.ua (N. Artyukhova); o.zelikovska@knu.ua (O. Zelikovska)
0000-0003-1112-6891 (A. Artyukhov); 0000-0002-0262-762X (I. Volk); 0000-0002-2040-8762 (O. Dluhopolskyi); 0000-
0002-2408-5737 (N. Artyukhova); 0000-0002-6559-9101 (O. Zelikovska)
2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
ceur-ws.org
ISSN 1613-0073
242
Proceedings
can be employed to explore causal relationships within the dynamic education system, each differing
in the openness of initial data, system parameters, control parameters, disturbances, and output
parameters. These methodologies are consistent with the theory of testing technical systems and
software products, commonly referred to as black, gray, and white box methods. When assessing the
quality of an education system, it is crucial to examine the interplay between its socio-economic
impact and several key criteria: the system s purposefulness, its hierarchical structure, the
interdependence between the system and its external environment, the degree of autonomy and
openness, the system s reliability, and its dimensional characteristics.
2. Literature review
The application of black, gray, and white box models for describing dynamic systems extends beyond
technical engineering or computer systems and has been effectively utilized in various fields,
including cybernetics, economics, and other areas of knowledge. Andersson & Johansson [1] model
the market interaction when each production and distribution unit have a positive effect on quality
improvement and corresponding pricing opportunities due to information about market variables at
individual locations. Boumans [2] & Kasianiuk [3] present two models of system identification
white box and black box as useful tools that help to understand the processes of self-
organization inside and outside organizations. Ji & Luo [4] analyzed the phenomenon of ecological
economics as a black box. In articles [5-11] authors provide technique of black box vs. white box
testing based on latest advancements in different domains.
Although the box method has been employed in the education sector, its use has primarily focused
on specific applications within the educational process, such as training, rather than on analyzing
management (control) processes. Shkarlet et al. [12] describe the development of the Quadruple Helix
model for solving the problems of the information economy. The studies [13-15] focus attention on
SDG (Sustainable Development Goals) in the context of environmental education. Papers [16-20] are
devoted to the analysis of the impact of modern technologies on the quality of higher education in
Ukraine.
within the educational sphere [21-30] and the increasing emphasis on achieving the SDG in education
[31-38].
Figure 1: Results of b
Source: composed by authors by VOSviewer in Scopus database
243
Figure 2: Results of b ,
Source: composed by authors by VOSviewer in Scopus database
A bibliometric analysis of literature sources obtained from the scientometric Scopus database
(Figures 1-2, using the VOSviewer bibliometric analysis tool) reveals a growing interest in applying
approaches that describe intangible systems by drawing analogies with technical systems.
[9-11]. It is important to clarify that in this context,
the term quality assurance refers to software testing, not education [19-22]. The framework of unit
testing quality control quality assurance can effectively establish a causal relationship between
-economic development. Simultaneously, it is crucial to
clearly define the prerequisites for maintaining the quality of the education system, particularly
concerning the input of initial data and the prediction of outcomes.
Thus, on the basis of bibliometric analysis and literature review, it is possible to formulate the
purpose of the article evaluation the quality of the education system by conceptualizing it through
the black, gray, and white box methods, drawing analogies between physical (engineering) systems
and intangible educational systems. In this context, it is imperative to assess how the design of the
education system impacts the socio-economic development of an organization, region, or country.
3. Methodology
Scopus scientometric database [39] was used for bibliometric analysis. Bibliometric analysis tool
VOSviewer [40]. The main method of the study was bibliometric analysis using the VOSviewer
software tool to analyze the relationships between different categories and build maps to visualize
the interconnectedness between them in publications indexed by the Scopus database. The
bibliographic analysis was carried out using the keywords: technical system, gray box, black box,
white box for 1991-2023.
An algorithm for developing the facilitation of the transition between various models is used to
assess the quality of the education system was constructed based on data obtained from the analysis
of the black, gray, and white box methods. The algorithm is presented in the diagram in Figure 3.
The diagram (Figure 3) is accompanied by the following detailed description of each level within
the three-level box model framework, specifically in relation to education system quality: 1) Black
box: only inputs (๐) and outputs (๐); 2) Gray box: inputs (๐), control parameters (๐), and outputs (๐);
244
3) White box: inputs (๐), control parameters (๐), system parameters (๐พ), disturbing parameters (๐),
and outputs (๐).
Figure 3: Box model framework of quality of education system: X input parameters; Y output
parameters; Z disturbing parameters; U control parameters; system parameters
Source: original research
The system is characterized by a sequential progression from the black box model to the gray box
model, and ultimately to the white box model. Each of these models can operate autonomously and
effectively representing a specific set of input and output parameters. However, as the transition is
made from the simpler to the more complex models, there is a corresponding increase in the
determinism of the description process. Such layered approach enables progressively deeper insights
into the internal structure and influencing factors of the system.
4. Results
s economic
growth and the surrounding region.
Output parameters (Y) include: the ranking of the university s educational programs among
employers; the average salary of graduates from the educational programs; the career advancement
of graduates of the educational programs.
Input parameters (X) consist of: the range of educational programs offered; the availability of state-
funded training opportunities for students; the cost of tuition; the presence of competing educational
institutions within the region; the availability of practical training and internships opportunities; the
list of potential employers.
245
Disturbing parameters (Z) refer to external influences, including: changes in the policies of the
Ministry of Education and Science of Ukraine, particularly those related to the financing of the
activities; the impact of military activities on the national economy, particularly in war-affected
regions.
The parameters of the K system can be represented as characteristics of the educational program
and its surrounding environment, especially when evaluated against similar educational programs
through benchmarking (Figure 4) or by using comparative data from rating agencies (Figure 5) such
as QS World University Rankings [41], World University Rankings [42], Academic Ranking of World
Universities [43] etc. To evidence the functionality of the proposed model works, we will apply
assumptions, the results of which can be obtained through a survey.
100 100 100 100 100 100 100 100 100 100
90 90
80
71 70
57 60
50
44
35
Fact Target
Figure 4: Outline of the educational program and its environment (benchmarking of educational
programs, visualization, random data)
Source: original research
Key observation from Figure 4:
1. Learning through research: the target reached 90% (good result).
2. Transparency and publicity: the target reached the lower level of 57%.
3. Internal quality assurance of the educational program: the target reached 71%, indicating strong
internal quality assurance overall, though one perspective sees room for improvement.
4. Educational environment and material resources: the target reached a lower level of 35%,
suggesting significant concerns about the educational environment and resources.
5. Human resources: the target reached 44%, indicating a perceived difference in the adequacy of
human resources.
6. Control measures, evaluation of students, and academic integrity: the target reached 50%,
showing a moderate gap.
246
7. Teaching and learning in the educational program: the target reached 60%, pointing to a
discrepancy in perceived effectiveness in teaching and learning practices.
8. Access to the educational program and recognition of learning outcomes: the target reached
90% (good result).
9. The structure and content of the educational program: the target reached 80%, showing that the
structure and content are viewed positively overall, though one perspective notes some areas for
enhancement.
10. Design and objectives of the educational program: the target reached 70%.
Considering the above, some aspects such as transparency, human resources, and the educational
environment being perceived as needing improvement.
4 4 4 4 4 4 4 4 4 4
4 4 4 4
3 3 3 3
2 2
Fact Target
Figure 5: Outline of the educational program and its environment (0 the parameter is not applied;
1 parameter mismatch; 2 critical remarks that can be eliminated; 3 compliance of the parameter;
4 compliance of the innovative parameter)
Source: original research
Key observations from the Figure 5:
1. Educational environment and material resources: high or maximum rating (4 from 4).
2. Design and objectives of the educational program: high or maximum rating (4 from 4).
3. Transparency and publicity: high or maximum rating (4 from 4).
4. Control measures, evaluation of students and academic integrity: high or maximum rating (4
from 4).
5. Internal quality assurance of the educational program: 3 of 4 points (good level).
6. Teaching and learning in the educational program: 3 of 4 points (good level).
247
7. Access to the educational program and recognition of learning outcomes: 3 of 4 points (good
level).
8. The structure and content of the educational program: 3 of 4 points (good level).
9. Human resources: 2 of 4 points (average level).
10. Learning through research: 2 of 4 points (average level).
The evaluation of various aspects of the educational program reveals that several areas, including
the educational environment and material resources, design and objectives of the educational
program, transparency and publicity, and control measures and academic integrity, received the
highest possible rating. Other areas, such as internal quality assurance, teaching and learning, access
to the program and recognition of learning outcomes, and the structure and content of the program,
were rated at a good level. However, human resources and learning through research were rated at
an average level.
Parameters of control U can be displayed on a radial diagram with an assessment of the degree of
the impact on the output parameter on a scale from 1 to 10 points (Figure 6).
government funding
10
8
quality of infrastructure grant funding
6
4
2
0
funding from the creation of
popularity of the educational
developments and the
program
provision of services
ranking position international partnership
Figure 6: Evaluation of the impact level of the control parameter on the output parameter
(visualization, random data)
Source: original research
Key observation form Figure 6:
1. Government funding: the score for government funding is at 5, indicating a moderate level of
government support.
2. Grant funding: this parameter is also rated at 5, suggesting a similar level of support from grants
as from government funding.
3. Funding from the creation of developments and provision of services: this parameter has a 7
score.
4. International partnership: this parameter is rated very low 4, suggesting that international
partnerships are limited.
6. Ranking position: the ranking position is rated around 6.
7. Popularity of the educational program: this parameter has an 8 score (the higher result).
8. Quality of infrastructure: this parameter has a higher score of 5.
Overall, while the program shows strengths in areas such as service-based funding and popularity,
it has moderate government and grant support, and there is a clear need for improvement in
international partnerships.
5. Conclusions
The proposed algorithm for the sequential description of the quality of education system and its
socio-economic impact allows for several key outcomes: 1) establishing a clear set of indicators that
248
define the influence of input, control, and system parameters on output parameters, thereby enabling
the prediction of their changes; 2) developing mechanisms to enhance the system by adjusting control
parameters to increase the value of output parameters; 3) evaluating the current state of system
parameters and identifying potential areas for improvement; 4) predicting the system's behavior
under the influence of external disturbances; 5) creating a roadmap to achieve desired output
parameters at operational (situational), tactical, and strategic levels.
The novelty of this study lies in the application of the technical system approach typically used
in engineering and physical sciences to the assessment of intangible systems, specifically the quality
of education. Through the deployment of the black, gray, and white box models, the study
innovatively conceptualizes the education system as a dynamic entity with inputs, control
parameters, and external disturbances, allowing for a systematic evaluation of its impact on socio-
economic development.
Aknowledgments
-03-V01-
-03-V04-00522/2024/VA) and a grant from the
Ministry of Education and Science of Ukraine under project Modeling and forecasting of socio-
economic consequences of higher education and science reforms in wartime (No. 0124U000545).
Declaration on Generative AI
The authors have not employed any Generative AI tools.
References
[1] A. E. Andersson, B. Johansson, Inside and outside the black box: organization of
interdependencies, The Annals of Regional Science 61 (2018) 501 516. doi:10.1007/s00168-018-
0886-1.
[2] M. J. Boumans, Understanding in economics: gray-box models, in: H. W. de Regt,S. Leonelli, K.
Eigner (Eds.), Scientific Understanding: Philosophical Perspectives, University of Pittsburgh
Press, 2009, pp. 210 229. doi:10.2307/j.ctt9qh59s.14.
[3] K. Kasianiuk, White box, black box and self-organization. A system-to-environment approach to
leadership, Kybernetes 45(1) (2016) 1 16. doi:10.1108/K-02-2015-0057.
[4] X. Ji, Z. Luo, Opening the black box of economic processes: ecological economics from its
biophysical foundation to a sustainable economic institution, The Anthropocene Review 7(3)
(2020) 231 247. doi:10.1177/2053019620940753.
[5] R. Bierig. S. Brown, E. Galvรกn, J. Timoney, Black-Box and White-Box Testing, in: Essentials of
Software Testing, Cambridge University Press, Cambridge, UK, 2021, pp. 141 164.
doi:10.1017/9781108974073.
[6] I. Komargodski, M. Naor, E. Eylon Yogev, White-box vs. black-box complexity of search
problems: Ramsey and graph property testing, Journal of the ACM 34 (2019).
doi:10.1145/3341106.
[7] D. Wintana, D. Pribadi, M. Y. Nurhadi, Analisis Perbandingan Efektifitas White-Box Testing dan
Black-Box Testing. Jurnal Larik Ladang Artikel Ilmu Komputer 2(1) (2022) 8 16.
doi:10.31294/larik.v2i1.1382.
[8] M. M. Syaikhuddin, C. Anam, A. R. Rinaldi, M. E. Conoras, Conventional software testing using
white box method, Kinetik 3(1) (2018) 65 72. doi:10.22219/kinetik.v3i1.231.
[9] M. Trucano, Open data, closed algorithms, and the black box of education, 2016. URL:
https://blogs.worldbank.org/en/education/open-data-closed-algorithms-and-black-box-
education.
[10] X. D. Yu, M. Y. Zhang, M. Q. Zhu, K. H. Xu, Q. C. Xiang, Research on gray-box testing methods
for software fault injection, Applied Mechanics and Materials (2014) 543 547.
doi:10.4028/www.scientific.net/AMM.543-547.3360.
249
[11] V. Hassija, V. Chamola, A. Mahapatra, A. Singal, D. Goel, K. Huang, S. Scardapane, I. Spinelli, M.
Mahmud, A. Hussain, Interpreting black-box models: a review on explainable artificial
intelligence. Cognitive Computation 16(1) (2024) 45 74. doi:10.1007/s12559-023-10179-8.
[12] S. Shkarlet, N. Kholiavko, M. Dubyna, O. Zhuk, Innovation, education, research components of
the evaluation of information economy development (as exemplified by Eastern partnership
countries), Marketing and Management of Innovations 1 (2019) 70 83. doi:10.21272/mmi.2019.1-
06.
[13] W. T. Fang, A. Hassan, B. LePage, The living environmental education: Sound science toward a
cleaner, safer, and healthier future. Springer, 2023. doi:10.1007/978-981-19-4234-1.
[14] M. Mazurkiewicz, O. Liuta, K. Kyrychenko, Internal quality assurance system for the higher
education: experience of Ukraine and Poland, Business Ethics and Leadership 1(4) (2017) 74 83.
doi:10.21272/bel.1(4).74-83.2017.
[15] A. Artyukhov, I. Volk, A. Surowiec, S. Skrzypek-Ahmed, K. Bliumska-Danko, O. Dluhopolskyi,
V. Shablystyi, Quality of education and science in the context of Sustainable Development Goals
from Millennium Goals to Agenda 2030: factors of innovation activity and socio-economic
impact, Sustainability 14 (2022) 11468. doi:10.3390/su141811468.
[16] A. Artyukhov, I. Volk, O. Dluhopolskyi, Immersive university
model: a tool to increase higher education competitiveness, Sustainability 15 (2023) 7771.
doi:10.3390/su15107771.
[17] I. Skliar, Towards the assurance of transparency and quality of higher education in Ukraine:
National Qualification Framework, Business Ethics and Leadership 2(1) (2018) 96 105.
doi:10.21272/bel.2(1).96-105.2018.
[18] L. Savga, O. Krykliy, K. Kyrychenko, The role of internal and external stakeholders in higher
education system in Ukraine, Business Ethics and Leadership 2(1) (2018) 32 43.
doi:10.21272/bel.2(1).32-43.2018.
[19] V. Okulich-Kazarin, A. Artyukhov, L. Skowron, N. Artyukhova, O. Dluhopolskyi, W. Cwynar,
Sustainability of higher education: study of student opinions about the possibility of replacing
teachers with AI technologies. Sustainability 16 (2024) 55. doi:10.3390/su16010055.
[20] H. Polianovskyi, T. Zatonatska, O. Dluhopolskyi, I. Liutyi, Digital and technological support of
distance learning at universities under COVID-19 (case of Ukraine), Revista Romaneasca pentru
Educatie Multidimensionala 13(4) (2021) 595-613. doi:10.18662/rrem/13.4/500
[21] C. Yeung, Economic and trade information: Hong Kong. Data and profiles market profiles,
2024. URL: https://research.hktdc.com/en/article/MzIwNjkzNTY5.
[22] I. Georgiadou, A. Vlachou, P. Stavroussi, Quality of life and vocational education service quality
in students with intellectual disability. International Journal of Developmental Disabilities 68(5)
(2021) 681 691. doi:10.1080/20473869.2021.1887435.
[23] S. Shkarlet, N. Kholiavko, M. Dubyna, Information economy: management of educational,
innovation, and research determinants, Marketing and Management of Innovations, 3 (2019)
126 141. doi:10.21272/mmi.2019.3-10.
[24] L. Matos, K. Kasztelnik, Transformational educational leadership and the innovative strategies
engaging online faculty for the excellent teaching performance in the United States, Business
Ethics and Leadership 5(1) (2021) 6 21. doi:10.21272/bel.5(1).6-21.2021.
[25] F. M. Reimers, The sustainable development goals and education, achievements and
opportunities. International Journal of Educational Development 104 (2024) 102965.
doi:10.1016/j.ijedudev.2023.102965.
[26] A. F. Pipa, K. Rasmussen, K. Pendrak, The state of the Sustainable Development Goals in the
United States. Brookings Institution, 2022. URL: https://www.brookings.edu/articles/the-state-
of-the-sustainable-development-goals-in-the-united-states.
[27] M. Young, The Sustainable Development Goals look shaky but here are five ways to deliver.
International Alert, 2019. URL: https://www.international-alert.org/blogs/sustainable-
development-goals-look-shaky-here-are-five-ways-deliver.
[28] S. J. Klees, Why SDG 4 and the other SDGs are failing and what needs to be done. International
Journal of Educational Development 104 (2024) 102946. doi:10.1016/j.ijedudev.2023.102946.
[29] J. Sachs, G. Lafortune, C. Kroll, G. Fuller, F. Woelm, From crisis to sustainable development: The
SDGs as roadmap to 2030 and beyond. Sustainable Development Report 2022. Cambridge
250
University Press, Cambridge, UK 2022. URL:
https://sdgtransformationcenter.org/reports/sustainable-development-report-2022.
[30]
of migration on forecasting budget expenditures on education: the sustainability context.
Sustainability 15 (2023) 15473. doi:10.3390/su152115473.
[31] O. Liuta, S. Lieonov, A. Artyukhov, M. Sushko-Bezdenezhnykh, O. Dluhopolskyi, Student survey
as a tool for quality assurance in higher education: the case of Ukrainian university, Naukovyi
Visnyk Natsionalnoho Hirnychoho Universytetu 4 (2021) 158 164. doi:10.33271/nvngu/2021-
4/158.
[32] L. Olorogun, N. Yunusa, H. G. Audu, A.A. Mohammed, Management of educational innovations:
of Innovations 2 (2018) 69 78. doi:10.21272/mmi.2018.2-06.
[33] K. Upulee, J.M. Bieman, Testing scientific software: a systematic literature review, Information
and software technology, 56(10) (2014) 1219-1232. doi:10.1016/j.infsof.2014.05.006.
[34] A. Rauf, E.M. Reddy, Software test automation: an algorithm for solving system management
automation problems, Procedia Computer Science, 46 (2015) 949 956.
doi:10.1016/j.procs.2015.01.004.
[35] A. Arcuri, Teaching software testing in an algorithms and data structures course, 2020 IEEE
International Conference on Software Testing, Verification and Validation Workshops (ICSTW),
1 (2020) 419 424. doi:10.1109/ICSTW50294.2020.00075.
[36] A. Bosworth, Metaโs progress in augmented and virtual reality. Meta, 2022. URL:
https://about.fb.com/news/2022/12/metas-progress-in-augmented-and-virtual-reality.
[37] X. Muianga, The role of ICT in the shift towards student-centred learning in higher education:
Eduardo Mondlane University, Mozambique: a case study. PhD thesis. Stockholm University,
Stockholm, Sweden, 2019.
[38] P. Breen, Developing educators for the digital age: a framework for capturing knowledge in
action. London: University of Westminster Press, London, UK, 2018. doi:10.16997/book13.
[39] Scopus, 2024. URL: https://www.scopus.com.
[40] VOSviewer, 2024. URL: https://www.vosviewer.com.
[41] QS World University Rankings, 2024. URL: https://www.topuniversities.com.
[42] World University Rankings, 2024. URL: https://www.timeshighereducation.com/world-
university-rankings.
[43] Academic Ranking of World Universities, 2024. URL:
https://www.shanghairanking.com/rankings/arwu/2024.
251