An Information System to Support Internal Quality Assurance of Educational Programs Viktoriia Dyomina1, Tetiana Bilova 2 and Iryna Pobizhenko3 1 V. Dokuchaev Kharkiv National Agrarian University, post office «Dokuchaevske - 2», Kharkiv region, Kharkiv district, 62483, Ukraine 2 Kharkiv National University of Radio Electronics, Pr. Nauki, 14 Kharkiv, 61000, Ukraine 3 Kharkiv National University of Radio Electronics, Pr. Nauki, 14 Kharkiv, 61000, Ukraine Abstract The article examines the issues of internal evaluation of the quality of educational programs in higher education institutions. The estimation is based on the involvement of stakeholder representatives. The evaluation criteria of the educational program are presented in the form of a rooted tree, where the root is the final grade. Multiple-valued logic is used to analyze the estimates made by stakeholders. It is proposed to translate the quantitative component of the assessment into a qualitative one. An algorithm for processing the set estimates has been developed. Differences in the assessments of different groups of stakeholders confirm the need to improve the elements of the educational program. A model of a database designed for the storage and processing of information related to the internal evaluation of the quality of educational programs has been developed. The proposed information system to support the process of internal quality assurance of educational programs allows to process quality indicators from different target groups and derive an overall assessment. The obtained data can be used for further improvement of educational programs. Keywords 1 information system, higher educational program, internal quality assurance, stakeholders, multiple-valued logic 1. Introduction The level of socio-economic and cultural development of society depends on the quality of higher education. The quality assurance process is a cyclical activity aimed at continuous improvement. Continuous quality assurance of the educational process will allow higher education institutions to respond in a timely manner to moving changes in society and enable each student to form an individual educational trajectory. An educational program (EP) generally refers to a system of educational components within a specialty, as well as the expected learning outcomes (competencies) that must be mastered by the student. Assessing the quality of EP is a complex multi-iterative process that requires time and the involvement of a large number of human resources. The assessment task is uncertain and multi-objective. In solving it, the interests of many parties should be taken into account. According to European standards [1], the process of quality assurance in education has three parts: internal quality assurance, external quality assurance, and quality assurance agencies. With the help of internal assessment, the institution of higher education analyzes the current state of the quality of the EP, identifies the main deficiencies in order to take appropriate measures to eliminate them. Internal assessment should answer the question of whether the EP meets the requirements of the current legislation and the modern needs of society. The internal assessment policy should have a formal status and be public. In addition to internal stakeholders - students and representatives of the institution, representatives of other interested parties are involved in the ICTERI-2021, Vol I: Main Conference, PhD Symposium, Posters and Demonstrations, September 28 – October 2, 2021, Kherson, Ukraine EMAIL: vvdemina17@gmail.com (A. 1); tetiana.bilova@nure.ua (A. 2); irina_pob@ukr.net (A. 3) ORCID: 0000-0001-6467-5021 (A. 1); 0000-0002-1085-7361 (A. 2); 0000-0002-0723-1878 (A. 3) ©️ 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) assessment. Employers, graduates, government representatives, public organizations, and others can act as external stakeholders. They have conflicting goals, this complicates the process of improving EPs and requires feedback from various groups of stakeholders. 2. Literature review The quality of teaching is the subject of much research. After analyzing publications on this topic, several directions can be distinguished. The first direction is the definition of quality assessment criteria. Thus, the basic standards and guidelines for quality assurance in higher education for internal, external quality assurance and quality assurance agencies were defined in 2015 [1]. In [2] showed trends 2018 learning and teaching in the European higher education area. The formulated quality standards formed the basis for the development of an internal quality assurance system. The work [2] provides examples of the formation of a strategic goal of learning and teaching, specific indicators and accreditation of EPs. To improve the quality of higher education in Europe [1], it is necessary to have in-ternal and external feedback [3]. The objectives of the EP are adjusted [4]. The results of various surveys of internal and external stakeholders who are informally involved in the process of developing the curriculum at the university, the content of EPs and curricula are analyzed. Surveys led to conclusions about a low percentage of stakeholder involvement in the design, monitoring and revision of programs. The influence of the opinion of key holders on the quality of particular criteria on a higher level of evaluation of the EP is not investigated. In surveys, stakeholders put high-quality ratings in the proposed scales. For the analysis and final assessment of the EP, it is required to apply a logical approach based on the logic of experts, the methodology of which is described in [5]. In 2012 study [6] defined criteria and grading scales ranging from 1 to 5 for raising awareness of quality teaching. This article does not propose to apply a computer approach to the processing of assessment results. The second direction is the study of the role of stakeholders in education, determination of their preferences and methods of managing them. The 2010 article [7] shows that higher education institutions need to forge beneficial relationships with various stakeholders and incorporate their respective visions and goals into their own governance practices. In a 2013 article [8] showed the empowerment of under-standing the relationship between organizations and their environment in the context of stakeholder identification and analysis, identified critical parameters for stake-holder analysis and its implementation in the higher education sector, and analyzed the impact of stakeholders on improving modern HEI governance. Indeed, this concept of stakeholders is critically important, but not adapted for building an information system for assessing the quality of education. The development of integrated approaches to quality assessment, including information support and information systems, is the third area. In the article [9] of 2020 showed a general scheme of a functional model for determining integral quality indicators for educational activity and the quality of higher education in universities. This model can serve as a basis for the development of an information system for assessing the quality of education. In [10] showed in their research in 2020 the development of an approach to building a management system for a higher educational institution based on a single hardware platform that allows to combine all areas of administrative and education-al activities within the framework of a unified system, as well as the organization of management of the educational process and monitoring the quality of educational services. The development of an automated subsystem for supporting quality assessment has not been investigated. In 2009 [11] published “the concept of „quality teaching‟ is complex and open to a range of definitions and interpretations.” One of the promising educational areas is mass personalized learning (student-centered), in the article [12] considered “The model of teaching quality by applying a resource-based approach and Scheme of the student-centered educational process”. “The exigencies of student participation in university governance” (examples, Ethiopian universities) are given in the article [13]. This way of understanding is reflected in article [14]. Based on the classification of higher education institutions (HEIs) as non-profit organizations and a review of student perception and self-esteem, various traditional concepts that explain student-university relationships are explored, and a new concept is proposed that is adopted from the business world, the concept of an active partner. This brings us to the idea of generalizing the application areas of the EP quality assessment information system. 3. Formal problem statement Assessment is the main tool for identifying shortcomings of EP and its elimination. The internal process of evaluating the EP is cyclical and includes: • analysis of the regulatory framework; • determination of the criteria for quality assessment; • determination of the timing of the assessment; • identification and involvement of stakeholders for the assessment; • ensuring that the assessment process is carried out; • analysis of the assessment results; development of recommendations to eliminate the identified deficiencies; • elimination of deficiencies. The quality of the internal assessment significantly depends on the stakeholder selection procedure. To do this, their area of interest has to be identified and all the documents necessary for quality assessment have to be provided. The procedure for involving stakeholders in the assessment in order to obtain a reliable result is uncertain. For example, employers give preference to disciplines that develop precisely practical skills, and the management of an institution - general subjects. The presence of contradictions allows us to identify the most problematic areas, improve the quality of educational services, change the paradigm in the direction of student-centered learning and teaching (all these contributing to a paradigm shift towards student-centered learning and teaching), prepare competitive professionals. Normative documents distinguish the following participants in the educational process: higher education institutions (HEI), employers and students. The goals of the participants in each of the groups may conflict, in which case the assessment of both a certain criterion and the educational process as a whole may differ significantly. If we focus specifically on the contradictions between management and stakeholders, then more accurately the diversity of vectors of hopes is reflected by five groups of stakeholders: higher education management, employers, graduates, educators, students. Students cannot yet assess the final learning outcome, but they are directly in-volved in the educational process. They are having information on what opportunities are provided for education, how democratically a higher education institution implements the principles defined by European standards. To assess the quality of the EP in general should be, graduates should be involved. It is they who have a broad view of the learning outcomes, how useful they were in their professional activities. Employers are also traditionally referred to as external stakeholders. In the process of developing an EP this group is constantly interacted with, their interests and wishes are taken into account in the formation of competencies and a block of free choice disciplines of students. The management of the university makes conflicting demands. On the one hand, there is an interest in licensing PRs and improving their quality, on the other hand, their assessment is influenced by many subjective factors, and the main criterion may not be quality, but the possibility of attracting additional funds for various grants and research. Scientists and teachers have their own interests. These include the possibility of creative implementation, financial security, satisfaction with the results of their activities. To meet the conflicting requirements of many stakeholders, an information support system for EP assessment should be created, which will reliably and effectively reflect the quality of the EP from the point of view of all participants in the educational process with the possibility of further improvement. The system of quality criteria has its own sub-criteria and their local detail. These components are determined on the basis of valid regulations [15]. In fig.1 an example of decomposition of criterion 3 is given. Thus, four levels of detail were obtained: an EP as a whole, criteria determined by law, sub- criteria and their detailing. This diagram can be represented as a tree of four possible mapping levels. The root of the tree is the EP and accordingly - its final assessment, on the opposite side of the tree there are leaves - details of sub-criteria. The quality of teaching is the subject of much research. After analyzing publications on this topic, several directions can be distinguished. Figure 1: Decomposition of the criterion “Access to the EP and recognition of learning outcomes” As a result of the analysis, five main groups of stakeholders were identified, which can be represented as a set: Ξ = {𝜉1 , 𝜉1 , … , 𝜉5 }, where 𝜉1 – leadership, 𝜉2 – employers, 𝜉3 – graduates, 𝜉4 – teachers, 𝜉5 – students. Representatives of all five stakeholders 𝜉𝑖 ∈ Ξ, (𝑖 = 1, 2, … , 5), should do so at this fourth level. The following quality scale is offered for evaluation Г = {𝐼, 𝐹, 𝐸, 𝐵, 𝐴}: «A» – full compliance with a certain criterion/sub-classification, including an innovative/exemplary character. «B» – general compliance with minor deficiencies. «E» – criterion as a whole is not met, but the shortcomings identified deficiencies can be eliminated within the specified period (one hour); «F» – certain criterion as a whole is not met, and the identified deficiencies are of a fundamental nature and/or cannot be eliminated within the specified term (one hour); «I» – evaluation is difficult to assess; I denotes the indifferent statement, 1 2 3 1 2 3 Ф = {0, , , , 1 } = {0, , , , 1 }. 𝑘−1 𝑘−1 𝑘−1 4 4 4 The selected estimate is stored as a specified variable. Thus, we obtain a vector of five elements, which contains an estimate of each stakeholder Ξ. It is necessary to create a mathematical model of exhibiting qualitative assessments at all levels of the tree criteria EP. 4. Development of a mathematical model Since all the requirements for the quality of the EP are presented in the form of a tree. The main nodes of the tree are indicated by the following variables: 𝜊1 is assessment of the first root level, the final assessment; 𝜊2𝑖 is assessment of the second level, 𝑖 = 1, 2, … , 𝑛, where n is the number of 𝑗 criteria for the assessment of the EP; 𝜊3,𝑖 is assessment of the third level, 𝑗 = 1, 2, … , 𝑘𝑖 , where 𝑘𝑖 is 𝑟𝑖,𝑘 , the number of sub-criteria of criterion i; 𝜊4,𝑖,𝑘𝑖 is assessment of the fourth level when detailing the 𝑖 sub-criteria – initial assessment by experts, where 𝑟𝑖,𝑘𝑖 = 1, 2, … , 𝑚𝑖,𝑘𝑖 , 𝑚𝑖,𝑘𝑖 is the number of details of the sub-criteria 𝑘𝑖 of criterion i. The expert group carries out evaluation at the fourth level. Their estimate must belong to the five element set Г = {𝐼, 𝐹, 𝐸, 𝐵, 𝐴}. Let us denote the expert's estimate of some sub-criterion (this is the 𝑣 fourth level of this graph) as 𝑒4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘 𝜖 Г, where v is the number in the ranked list of experts. This, 𝑖 𝑟𝑖,𝑘𝑖 1 2 3 4 5 𝜊4,𝑖,𝑘 = 𝜛 (𝑓 (𝑒4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘𝑖 , 𝑒4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘𝑖 , 𝑒4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘 , 𝑒4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘𝑖 , 𝑒4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘 )). 𝑖 𝑖 𝑖 We apply the function f to obtain a fourth-level estimate that converts a five-element vector of expert estimates into an intermediate estimate from a set of Г. The interim estimate will be obtained as the arithmetic average from the quantitative component 𝑣 𝜂4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘 𝜖 Ф (𝑣 = 1, 2, … , 5) of qualitative expert assessments. 𝑖 The quantitative evaluation thus obtained will be converted to qualitative by means of the function 𝜛: Ф → Г of the following threshold correspondence 𝐼 𝜖𝜑𝐼 = [0; 0,1), 𝐹 𝜖𝜑𝐹 = [0,1; 0,3), 𝐸 𝜖𝜑𝐸 = [0,3; 0,6), 𝐵 𝜖𝜑𝐵 = [0,6; 0,9), (1) 𝐴 𝜖𝜑𝐴 = [0,9; 1], 1 5 𝑟𝑖,𝑘𝑖 𝜂4,𝑖,𝑘 ,𝑟 + …+𝜂4,𝑖,𝑘 ,𝑟 1 5 𝑖 𝑖,𝑘𝑖 𝑖 𝑖,𝑘𝑖 𝜊4,𝑖,𝑘 = 𝜛 (𝑓 (𝜂4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘𝑖 , … , 𝜂4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘 )) = 𝜛 ( 5 ). 𝑖 𝑖 Let's move on to the formation of the assessment of the third level, the level of sub-criteria. For a more accurate analysis of the quality of the EP, it is proposed at this level to form 2 estimates for each 𝑗 vertex of the graph. First score 𝜊3,𝑖 (𝑗 = 1, 2, … , 𝑘𝑖 ). The estimate with the function g is also calculated as the arithmetic-scope mean of the quantitative analogues Ф of the letter vector Г. This, 1 2 𝑚𝑖,𝑘 (2) 𝑗 𝑚𝑖,𝑘𝑖 𝜂4,𝑖,𝑗 + 𝜂4,𝑖,𝑗 + ⋯ + 𝜂4,𝑖,𝑗 𝑖 1 2 𝜊3,𝑖 = 𝜛 (𝑔 (𝜊4,𝑖,𝑗 , 𝜊4,𝑖,𝑗 , … , 𝜊4,𝑖,𝑗 )) = 𝜛 ( ), 𝑚𝑖,𝑘𝑖 where 𝑖 = 1, 2, … , 𝑛, 𝑗 = 1, 2, … , 𝑘𝑖 , 𝑘𝑖 is the number of sub-criteria of criterion i, 𝑚𝑖,𝑘𝑖 is the number of details of sub-criteria 𝑘𝑖 of criteria i, 𝜛 is a function that converts the obtained quantitative score into a qualitative analogue Г = {𝐼, 𝐹, 𝐸, 𝐵, 𝐴} a vector according to the rule (1). We introduce the second assessment to obtain the real attitude to the EP of all stakeholders from the set Ξ. This second assessment forms a vector of intermediate assessments set by experts, while maintaining the structure. This will help to showing nodes on the EP graph that need special attention. It is possible to save the assessment of all details of the sub-criteria for each of the five expert groups at the fourth level: 𝑣 𝑣 𝑣 𝑣 𝑣 𝜊̅4,𝑖,𝑗 = 𝜛 (𝑔(𝑒4,𝑖,𝑗 , 𝑒4,𝑖,𝑗 , 𝑒4,𝑖,𝑗 , … , 𝑒4,𝑖,𝑗 )). When moving to the third level, we use the resulting vector for evaluating each of the detailing sub-criteria: 𝑗 1 2 3 4 5 (3) 𝜊 ∗ 3,𝑖 = 𝜛[𝑓(𝜊̅4,𝑖,𝑗 , 𝜊̅4,𝑖,𝑗 , 𝜊̅4,𝑖,𝑗 , 𝜊̅4,𝑖,𝑗 , 𝜊̅4,𝑖,𝑗 )], 𝑖 = 1, 2, … , 𝑛, where n is the number of criteria, 𝑗 = 1, 2, … , 𝑘𝑖 , where 𝑘𝑖 is the number of sub-criteria of criterion i. We form the vector of estimates of the expert group by sub-criteria: 𝑣 𝑣 𝑣 𝑣 𝑣 𝜊̅3,𝑖 = 𝜛 (𝑔(𝜊̅4,𝑖,1 , 𝜊̅4,𝑖,2 , 𝜊̅4,𝑖,3 , … , 𝜊̅4,𝑖,𝑘 𝑖 )). It is quite possible that with a wide scatter of expert assessments, when some put "A" and others - "E" (in fact, this speaks of "confusion" and misunderstanding by some experts of the assessment 𝑗 𝑗 area), evaluation 𝜊3,𝑖 and 𝜊 ∗ 3,𝑖 . This is an indicator of the urgent need to eliminate the shortcomings of the EP and stimulates the improvement of the quality of the evaluated program. Let's move on to the second level assessment: 1 1 𝑘 (4) 1 2 𝑘 𝜂3,𝑖 + 𝜂3,𝑖 + ⋯ + 𝜂3,𝑖𝑖 𝜊2𝑖 = 𝜛 (𝑔(𝜊3,𝑖 , 𝜊3,𝑖 , … , 𝜊3,𝑖𝑖 )) = 𝜛 ( ). 𝑘𝑖 As a result, we will obtain n assessments of all criteria and a mandatory second assessment, revealing the opinion of expert groups Ξ: 1 1 5 𝜂3,𝑖 + 𝜂3,𝑖 + ⋯ + 𝜂3,𝑖 𝜊 ∗ 𝑖2 = 𝜛 (𝑓(𝜊̅3,𝑖 1 2 , 𝜊̅3,𝑖 5 , … , 𝜊̅3,𝑖 )) = 𝜛 ( ). 5 We form a vector of intermediate estimates of all expert groups (𝑣 = 1, 2, … , 5) by criteria at the second level similar to the third level (3): 𝜊̅ 𝑣 = 𝜛 (𝑔(𝜊̅ 𝑣 , 𝜊̅ 𝑣 , 𝜊̅ 𝑣 , … , 𝜊̅ 𝑣 )). (5) 2 3,1 3,2 3,3 3,𝑘𝑖 𝑖 ∗𝑖 We compare the estimates 𝜊2 and 𝜊 2 , we identify to the "problem zones" of the EP. 𝛾 𝑛 We obtain a vector {𝑎2,𝑖 } , (𝛾𝜖Г = {𝐼, 𝐹, 𝐸, 𝐵, 𝐴}, 𝑖 = 1, 2, … , 𝑛), n is the number of criteria, 𝑖=1 𝐵 when evaluating each criterion separately. The designation 𝑎2,1 , indicates that the first criterion of the EP was rated "B" (index 2 indicates the second level of assessment). The final estimate at the top level will be based on this vector. Consider an indicator of the level of the EP, which is an increasing discrete function, the values of which correspond to the estimates obtained according to the criteria. 𝛾 𝛾 Consider the segment corresponding to the positive score 𝑎2,𝑖 : 𝑎2,𝑖 𝜖[0,1], (𝑖 = 1, 2, 3, … , 𝑛) on the five-place scale 𝛾 ∈ Г. The point τ divides the evaluation segment into two areas: the area 𝐼𝑛𝛾 , lying to the left of the τ, where the EP criteria are lower than required by a given quality, and the area 𝐼𝑑𝛾 lying to the right of the τ, the level of the criteria of which exceeds the required boundary of a sufficient assessment. The criteria in area 𝐼𝑛𝛾 (𝛾 ∈ Г) are necessary for obtaining the final o rating and the criteria in area 𝐼𝑑 𝛾 are sufficient. 𝛾 𝑎 𝛾 We will enter a logical variable 𝐵𝑖 2,𝑖 (where 𝑎2,𝑖 is the ball corresponding to the quality level for this criterion, i is the criterion number 𝑖 = 1, 2, 3, … , 𝑛, n is the number of criteria, 𝛾 ∈ Г) as an 𝛾 𝜊 indicator of the necessary and sufficient condition for rating. 𝐵𝑖 2,𝑖 = 1, if the quality of the EP according to the criterion i is necessary and mandatory when evaluating by the  -th point and belong 𝛾 𝛾 to the area 𝐼𝑛𝛾 : 𝑎2,𝑖 𝜖𝐼𝑛𝛾 , Similarly, if 𝑎2,𝑝 𝜖𝐼𝑑𝛾 , and the scores for the criterion 𝑘𝑝 are sufficient when 𝛾 𝜊 judging by the γ-th score, then 𝐵𝑝 2,𝑝 = 0. We obtained a final assessment of the EP level, taking into account the necessary and sufficient condition for issuing the assessment based on the assessments 𝛾 𝑛 according to the criteria {𝑎2,𝑖 } , n is the number of criteria: 𝑖=1 ̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅ (6) 𝛾 𝑛 𝑛 𝜃 𝑛 𝑛 𝛾 𝑘 𝑘 𝑑 𝑑 𝜊 =⋀ ⋀ 𝑎2,𝑗 ∨ ⋀ 𝑎2,𝑖 ⋀ ⋀ 𝑎2,𝑣 ∨ ⋁ 𝑎2,𝜎 , 𝑘=𝛼 𝑗=1 𝑖=1 𝑑=𝛾+1 𝑣=1 𝜎=1 𝑎𝑘 𝑎𝑘 𝑎𝑑 𝑎𝑑 2,𝑗 𝐵𝑗 =1 ( 𝐵𝑖 2,𝑖 =0 )] [ ( 𝐵𝑣 2,𝑣 =1 )] 𝐵𝜎 2,𝜎 =0 [ where 𝛾𝜖Г is the scale score {𝐼, 𝐹, 𝐸, 𝐵, 𝐴}; 𝛼 is the initial value of the scale for which the positive score is set, 𝛼 ≤ 𝜏 (𝜏 = "B"); 𝜃 is the final value of the entered scale 𝜃 ≥ 𝑖. It is understood that 𝜏 = "A". The criteria may be that are not calculated for the highest score. If the knowledge of the test subject does not correspond to the required level and 𝜊 𝛼 = 0, according to formula (6), the score is set as a point below the established lower rating limit 𝛼: 𝜊 𝛼−1 = 1. 5. System for supporting the process of internal evaluation of the quality of educational programs On the basis of the developed assessment model, an information system is proposed for conducting an internal assessment of EPs. System contains: • information on stakeholder experts involved in quality assessment procedures; • a system of criteria for assessing the quality of an EP with the possibility of editing and adjusting them to the characteristics of a particular EP; • means for providing stakeholders with access to the documents necessary for the assessment structured according to the evaluation tree; • support for the logic of assessment inference procedures; • identification of bottlenecks in the EP and contradictions between the assessments of various groups of stakeholders; • storage of information and statistical processing of the results of examinations. The information system consists of the following components: 1. A database containing stakeholder information, assessment criteria, and the results of detailed sub-criteria assessments. 2. Document database containing documents structured according to detailed sub-criteria. Stakeholders are given access to these documents. 3. The assessment planning unit is designed to prepare an assessment, determine the documents required for each level of detail, form a set of stakeholders for the assessment and the main assessment criteria. 4. Synchronization unit for determining if there is already complete information for the assessment. In case of lack of data, the system will generate a preliminary assessment. 5. The unit for forming an assessment by the formulas (2) - (6) performs the formation of quantitative indicators at the second, third, and fourth levels of assessment and translates them into a qualitative scale. 6. The analysis unit is designed for statistical analysis of the history of the assessment, identifying the most critical areas of the EP that is being assessed. 7. Output unit - generates reports and screen forms at the request of users. The assessment process is preceded by the preparation of the documents necessary for the assessment of each level of detail. Input information for the operation of the system - estimates of stakeholders at tree levels, defined as leaves. As a result of the assessment, the system can provide the following information: • the final assessment of the EP and its vector of assessments concerning all stake-holders; • assessment of each of the criteria and sub-criteria and their details; • tables for the formation of the final assessment, individual criteria, and sub-criteria; • vector of interim assessments of stakeholders at each level of assessment; • a list of conflicting criteria and sub-criteria on which the assessments of various stakeholder groups had the greatest differences; • a list of criteria and sub-criteria that are most critical and have had a negative impact on the overall assessment. The scheme of the database is the system presented in Figure 2. Figure 2: Scheme of the database of the support system for internal evaluation of EPs Table 1 shows the result of the program in the evaluation of sub-criterion 3.3 (Fig. 1). Thus, in as a result of the operation of the system using the formulas (2) we obtain a third level estimate for the third sub-criterion of the third criterion "E" with a quantitative analogue of 0.5 points. Applying this technique, we obtain intermediate evaluations at the third level, then - at the second (4). The final grade is set taking into account the necessary and sufficient conditions for making a grade (6) on the basis of grades by criteria. This is how the vector of second level evaluations is obtained 𝑜2 = {𝐴, 𝐵, 𝐸, 𝐵, 𝐵, 𝐸, 𝐴, 𝐵, 𝐵, 𝐵, 𝐸}, final assessment 𝑜1 = "𝐵". Table 1 The result of the evaluation of sub-criterion 3.3 "Rules for recognition of learning outcomes obtained in other educational institutions" Sub- Evaluation Detailing Summary criteria 3.3 leadership employers graduates teachers students 0.5 (E) 1 0,4 (E) E (0,5) F (0.25) E (0,5) E (0,5) F (0.25) 2 0,65 (B) B (0,75) E (0,5) E (0,5) B (0,5) B (0,75) 3 0.4 (E) E (0,5) E (0,5) F (0.25) E (0,5) F (0.25) 4 0.5 (E) B (0,75) E (0,5) E (0,5) B (0,75) I (0) Summary 0,67 (B) 0,5 (E) 0,42 (E) 0,67 (B) 0,33 (E) The resulting vector of assessments made it possible to determine the problem areas of the EP. For the example considered, attention should be paid to expanding the opportunities for access to EP and recognition of learning outcomes, advanced training of human resources, a clear formulation of the prospects for the development of EP. Similarly, the assessments of each of the sub-criteria can be analyzes and the corresponding component of the EP can be adjusted. Next, the system for supporting the process of internal evaluation of the quality of EPs issues a ∗ final score for five stakeholders (5), 𝑜1 = {𝐸, 𝐴, 𝐴, 𝐵, 𝐵}. Thus, as a result of this feedback, it is possible to adapt the EP to the various goals of the stakeholders. 6. Conclusion The analysis of the stakeholders of the educational institution made it possible to identify the main stakeholders who could potentially participate in the assessment of the quality of EP. The proposed information system is designed to support the internal assessment of the quality of EPs by higher education institutions. The system allows combining information about the stakeholders of an educational institution with an assessment of their satisfaction with a particular EP. The main purpose of the system's usage is an integrated approach to the internal assessment of the EP quality and reducing the subjectivity of the final assessment. The system allows storing information about the assessment according to the system of criteria defined by regulatory documents. The advantages of the system are centralized storage and the ability to analyze information obtained during the assessment, if necessary, to carry out a partial assessment of any element. The adaptability of the system to changes in regulatory legislation is determined by the ease of changing any component of the evaluation tree and the ability to expand the tree by adding new nodes. The system is also adaptable to changes in the depth of the evaluation tree. The information obtained as a result of the analysis can become the basis for identifying problem areas and improving educational activities for this program. The system can be used to monitor the satisfaction of the educational process by all stakeholders. The number of stakeholder groups can also be expanded as needed. Needs further development of the issue of improving the methods of static analysis of the obtained estimates and the use of expert logic in the formation of intermediate assessments at different levels of quality assessment of EP. 7. References [1] Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG), 2015. URL: https://www.enqa.eu/wp-content/uploads/2015/11/ESG_2015.pdf. [2] M. Gaebel, Th. Zhang, Trends 2018. Learning and teaching in the European Higher Education Area, 2018. URL: https://eua.eu/downloads/publications/trends-2018-learning- and-teaching-in-the-european-higher-education-area.pdf. [3] C. Tomas, M. Kelo, ESG 2015–2018 ENQA agency reports: Thematic analysis. ENQA Occasional Paper 28, Brussles, Belgium, 2020. URL: http://www.enqa.eu/index.php/home/esg/. [4] O. Krykliy, K. Kyrychenko, O. Liuta, M. Mazurkiewicz, V. Yuskaiev, The State of the Development, Structure and Functionality of Internal Quality Assurance Systems in Ukrainian Higher Education Institutions. Quality Assurance in Higher Education Institutions in Ukraine 2016 through the prism of European Guidelines and Standards ESG 2015. EXANTE. Wrocław, 2016. URL: https://projects.lnu.edu.ua/quaere/wp- content/uploads/sites/6/2018/03/QA-in-Ukraine-reports.pdf. [5] D. E. Sitnikov, V. M. Demina, Logical approach to the assessment of knowledge according to the R-point system [in Russian], Bulletin of the National Technical University : Kharkov Polytechnic Institute Systems analysis, management and information technology, 125, Kharkiv, KPI, 2000, pp. 41–45. [6] F. Henard, D. Roseveare, Fostering Quality Teaching in Higher Education: Policies and Practices, 2012. URL: https://www.oecd.org/education/imhe/QT%20policies%20 and%20practices.pdf. [7] E. W. Mainardes, H. Alves, M. Raposo, An Exploratory Research on the Stakeholders of a University, Journal of Management and Strategy, 1, volume 1, 2010, pp. 76–88. URL: http://www.sciedu.ca/journal/index.php/jms/article/view/15. [8] I. Marić, Stakeholder analysis of higher education institutions. In Interdisciplinary Description of Complex Systems, 2, volume 11, 2013, pp. 217–226. URL: https://hrcak.srce.hr/100637. [9] O. Baturin, , V. Ivanov, V. Lyfar, S. Mytrokhin, L. Lyhina, Functional model of definition of integral indicators of quality of educational activity and quality of higher education by institutions of higher education, Scientific news of Dalev University, volume 18, 2020. URL: http://nvdu.snu.edu.ua/wp-content/uploads/2020/02/2020-18-3.pdf. [10] A.I. Baturin, V.O. Lyfar, O.I. Zakhozhay, V.H. Ivanov, «A Uniform Interaction Model Of Educational Process Agents In The Unified Management System Of A Higher Education Institution», Information Technologies and Learning Tools, 4, volume 78, 2020, pp. 266–277. URL: https://journal.iitta.gov.ua/index.php/itlt/article/download/3178/1704. [11] F. Hénard, Learning our Lesson: Review of Quality Teaching in Higher Education, OECD Publishing, Paris, 2010. URL: https://www.oecd.org/education/imhe/43961761.pdf. [12] N. Komleva, V. Liubchenko, S. Zinovatna, Improvement of Teaching Quality in the View of a Resource-Based Approach, 16th International Conference on ICT in Education, Research, and Industrial Applications, Kharkiv, Ukraine, 2020, pp. 262–277. URL: http://ceur-ws.org/Vol-2740/20200262.pdf. [13] W. Tamrat, The exigencies of student participation in university governance: Lip services and bottlenecks, School of Graduate Studies, St. Mary's University, Addis Ababa, Ethiopia, Higher Education Quarterly, 1, volume 74, 2020, pp. 35–47. URL: https://www.researchgate.net/publication/336695816_The_exigencies_of_student_participati on_in_university_governance_Lip_services_and_bottlenecks. [14] A. Sperlich, K. Spraul, Students As Active Partners: Higher Education Management in Germany Political Science, volume 12, issue 3, article 11, 2007, pp. 1–19 URL: https://www.researchgate.net/publication/237517180_Students_As_Active_Partners_Higher_ Education_Managem–ent_in_Germany. [15] Verkhovna Rada of Ukraine: On approval of the Regulation on accreditation of educational programs for which higher education applicants are trained: The Law “On Education”, 880/33851, 2019. URL: https://zakon.rada.gov.ua/laws/show/z0880-19#Text.