=Paper=
{{Paper
|id=Vol-3013/20210173
|storemode=property
|title=An Information System to Support Internal Quality Assurance of Educational Programs
|pdfUrl=https://ceur-ws.org/Vol-3013/20210173.pdf
|volume=Vol-3013
|authors=Viktoriia Dyomina,Tetiana Bilova,Iryna Pobizhenko
|dblpUrl=https://dblp.org/rec/conf/icteri/DyominaBP21
}}
==An Information System to Support Internal Quality Assurance of Educational Programs==
https://ceur-ws.org/Vol-3013/20210173.pdf
An Information System to Support Internal Quality Assurance
of Educational Programs
Viktoriia Dyomina1, Tetiana Bilova 2 and Iryna Pobizhenko3
1
V. Dokuchaev Kharkiv National Agrarian University, post office «Dokuchaevske - 2», Kharkiv region, Kharkiv
district, 62483, Ukraine
2
Kharkiv National University of Radio Electronics, Pr. Nauki, 14 Kharkiv, 61000, Ukraine
3
Kharkiv National University of Radio Electronics, Pr. Nauki, 14 Kharkiv, 61000, Ukraine
Abstract
The article examines the issues of internal evaluation of the quality of educational programs
in higher education institutions. The estimation is based on the involvement of stakeholder
representatives. The evaluation criteria of the educational program are presented in the form
of a rooted tree, where the root is the final grade. Multiple-valued logic is used to analyze the
estimates made by stakeholders. It is proposed to translate the quantitative component of the
assessment into a qualitative one. An algorithm for processing the set estimates has been
developed. Differences in the assessments of different groups of stakeholders confirm the
need to improve the elements of the educational program. A model of a database designed for
the storage and processing of information related to the internal evaluation of the quality of
educational programs has been developed. The proposed information system to support the
process of internal quality assurance of educational programs allows to process quality
indicators from different target groups and derive an overall assessment. The obtained data
can be used for further improvement of educational programs.
Keywords 1
information system, higher educational program, internal quality assurance, stakeholders,
multiple-valued logic
1. Introduction
The level of socio-economic and cultural development of society depends on the quality of higher
education. The quality assurance process is a cyclical activity aimed at continuous improvement.
Continuous quality assurance of the educational process will allow higher education institutions to
respond in a timely manner to moving changes in society and enable each student to form an
individual educational trajectory. An educational program (EP) generally refers to a system of
educational components within a specialty, as well as the expected learning outcomes (competencies)
that must be mastered by the student. Assessing the quality of EP is a complex multi-iterative process
that requires time and the involvement of a large number of human resources. The assessment task is
uncertain and multi-objective. In solving it, the interests of many parties should be taken into account.
According to European standards [1], the process of quality assurance in education has three parts:
internal quality assurance, external quality assurance, and quality assurance agencies.
With the help of internal assessment, the institution of higher education analyzes the current state
of the quality of the EP, identifies the main deficiencies in order to take appropriate measures to
eliminate them. Internal assessment should answer the question of whether the EP meets the
requirements of the current legislation and the modern needs of society. The internal assessment
policy should have a formal status and be public. In addition to internal stakeholders - students and
representatives of the institution, representatives of other interested parties are involved in the
ICTERI-2021, Vol I: Main Conference, PhD Symposium, Posters and Demonstrations, September 28 – October 2, 2021, Kherson, Ukraine
EMAIL: vvdemina17@gmail.com (A. 1); tetiana.bilova@nure.ua (A. 2); irina_pob@ukr.net (A. 3)
ORCID: 0000-0001-6467-5021 (A. 1); 0000-0002-1085-7361 (A. 2); 0000-0002-0723-1878 (A. 3)
©️ 2020 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
assessment. Employers, graduates, government representatives, public organizations, and others can
act as external stakeholders. They have conflicting goals, this complicates the process of improving
EPs and requires feedback from various groups of stakeholders.
2. Literature review
The quality of teaching is the subject of much research. After analyzing publications on this topic,
several directions can be distinguished. The first direction is the definition of quality assessment
criteria. Thus, the basic standards and guidelines for quality assurance in higher education for internal,
external quality assurance and quality assurance agencies were defined in 2015 [1]. In [2] showed
trends 2018 learning and teaching in the European higher education area. The formulated quality
standards formed the basis for the development of an internal quality assurance system. The work [2]
provides examples of the formation of a strategic goal of learning and teaching, specific indicators
and accreditation of EPs.
To improve the quality of higher education in Europe [1], it is necessary to have in-ternal and
external feedback [3]. The objectives of the EP are adjusted [4]. The results of various surveys of
internal and external stakeholders who are informally involved in the process of developing the
curriculum at the university, the content of EPs and curricula are analyzed. Surveys led to conclusions
about a low percentage of stakeholder involvement in the design, monitoring and revision of
programs. The influence of the opinion of key holders on the quality of particular criteria on a higher
level of evaluation of the EP is not investigated. In surveys, stakeholders put high-quality ratings in
the proposed scales. For the analysis and final assessment of the EP, it is required to apply a logical
approach based on the logic of experts, the methodology of which is described in [5]. In 2012 study
[6] defined criteria and grading scales ranging from 1 to 5 for raising awareness of quality teaching.
This article does not propose to apply a computer approach to the processing of assessment results.
The second direction is the study of the role of stakeholders in education, determination of their
preferences and methods of managing them. The 2010 article [7] shows that higher education
institutions need to forge beneficial relationships with various stakeholders and incorporate their
respective visions and goals into their own governance practices. In a 2013 article [8] showed the
empowerment of under-standing the relationship between organizations and their environment in the
context of stakeholder identification and analysis, identified critical parameters for stake-holder
analysis and its implementation in the higher education sector, and analyzed the impact of
stakeholders on improving modern HEI governance. Indeed, this concept of stakeholders is critically
important, but not adapted for building an information system for assessing the quality of education.
The development of integrated approaches to quality assessment, including information support and
information systems, is the third area. In the article [9] of 2020 showed a general scheme of a
functional model for determining integral quality indicators for educational activity and the quality of
higher education in universities. This model can serve as a basis for the development of an
information system for assessing the quality of education.
In [10] showed in their research in 2020 the development of an approach to building a
management system for a higher educational institution based on a single hardware platform that
allows to combine all areas of administrative and education-al activities within the framework of a
unified system, as well as the organization of management of the educational process and monitoring
the quality of educational services. The development of an automated subsystem for supporting
quality assessment has not been investigated.
In 2009 [11] published “the concept of „quality teaching‟ is complex and open to a range of
definitions and interpretations.”
One of the promising educational areas is mass personalized learning (student-centered), in the
article [12] considered “The model of teaching quality by applying a resource-based approach and
Scheme of the student-centered educational process”. “The exigencies of student participation in
university governance” (examples, Ethiopian universities) are given in the article [13]. This way of
understanding is reflected in article [14]. Based on the classification of higher education institutions
(HEIs) as non-profit organizations and a review of student perception and self-esteem, various
traditional concepts that explain student-university relationships are explored, and a new concept is
proposed that is adopted from the business world, the concept of an active partner. This brings us to
the idea of generalizing the application areas of the EP quality assessment information system.
3. Formal problem statement
Assessment is the main tool for identifying shortcomings of EP and its elimination. The internal
process of evaluating the EP is cyclical and includes:
• analysis of the regulatory framework;
• determination of the criteria for quality assessment;
• determination of the timing of the assessment;
• identification and involvement of stakeholders for the assessment;
• ensuring that the assessment process is carried out;
• analysis of the assessment results; development of recommendations to eliminate the
identified deficiencies;
• elimination of deficiencies.
The quality of the internal assessment significantly depends on the stakeholder selection
procedure. To do this, their area of interest has to be identified and all the documents necessary for
quality assessment have to be provided. The procedure for involving stakeholders in the assessment in
order to obtain a reliable result is uncertain. For example, employers give preference to disciplines
that develop precisely practical skills, and the management of an institution - general subjects. The
presence of contradictions allows us to identify the most problematic areas, improve the quality of
educational services, change the paradigm in the direction of student-centered learning and teaching
(all these contributing to a paradigm shift towards student-centered learning and teaching), prepare
competitive professionals.
Normative documents distinguish the following participants in the educational process: higher
education institutions (HEI), employers and students. The goals of the participants in each of the
groups may conflict, in which case the assessment of both a certain criterion and the educational
process as a whole may differ significantly. If we focus specifically on the contradictions between
management and stakeholders, then more accurately the diversity of vectors of hopes is reflected by
five groups of stakeholders: higher education management, employers, graduates, educators, students.
Students cannot yet assess the final learning outcome, but they are directly in-volved in the
educational process. They are having information on what opportunities are provided for education,
how democratically a higher education institution implements the principles defined by European
standards.
To assess the quality of the EP in general should be, graduates should be involved. It is they who
have a broad view of the learning outcomes, how useful they were in their professional activities.
Employers are also traditionally referred to as external stakeholders. In the process of developing an
EP this group is constantly interacted with, their interests and wishes are taken into account in the
formation of competencies and a block of free choice disciplines of students.
The management of the university makes conflicting demands. On the one hand, there is an
interest in licensing PRs and improving their quality, on the other hand, their assessment is influenced
by many subjective factors, and the main criterion may not be quality, but the possibility of attracting
additional funds for various grants and research. Scientists and teachers have their own interests.
These include the possibility of creative implementation, financial security, satisfaction with the
results of their activities. To meet the conflicting requirements of many stakeholders, an information
support system for EP assessment should be created, which will reliably and effectively reflect the
quality of the EP from the point of view of all participants in the educational process with the
possibility of further improvement.
The system of quality criteria has its own sub-criteria and their local detail. These components are
determined on the basis of valid regulations [15]. In fig.1 an example of decomposition of criterion 3
is given. Thus, four levels of detail were obtained: an EP as a whole, criteria determined by law, sub-
criteria and their detailing. This diagram can be represented as a tree of four possible mapping levels.
The root of the tree is the EP and accordingly - its final assessment, on the opposite side of the tree
there are leaves - details of sub-criteria. The quality of teaching is the subject of much research. After
analyzing publications on this topic, several directions can be distinguished.
Figure 1: Decomposition of the criterion “Access to the EP and recognition of learning outcomes”
As a result of the analysis, five main groups of stakeholders were identified, which can be
represented as a set: Ξ = {𝜉1 , 𝜉1 , … , 𝜉5 }, where 𝜉1 – leadership, 𝜉2 – employers, 𝜉3 – graduates, 𝜉4 –
teachers, 𝜉5 – students. Representatives of all five stakeholders 𝜉𝑖 ∈ Ξ, (𝑖 = 1, 2, … , 5), should do so
at this fourth level. The following quality scale is offered for evaluation Г = {𝐼, 𝐹, 𝐸, 𝐵, 𝐴}:
«A» – full compliance with a certain criterion/sub-classification, including an
innovative/exemplary character.
«B» – general compliance with minor deficiencies.
«E» – criterion as a whole is not met, but the shortcomings identified deficiencies can be
eliminated within the specified period (one hour);
«F» – certain criterion as a whole is not met, and the identified deficiencies are of a fundamental
nature and/or cannot be eliminated within the specified term (one hour);
«I» – evaluation is difficult to assess; I denotes the indifferent statement,
1 2 3 1 2 3
Ф = {0, , , , 1 } = {0, , , , 1 }.
𝑘−1 𝑘−1 𝑘−1 4 4 4
The selected estimate is stored as a specified variable. Thus, we obtain a vector of five elements,
which contains an estimate of each stakeholder Ξ. It is necessary to create a mathematical model of
exhibiting qualitative assessments at all levels of the tree criteria EP.
4. Development of a mathematical model
Since all the requirements for the quality of the EP are presented in the form of a tree. The main
nodes of the tree are indicated by the following variables: 𝜊1 is assessment of the first root level, the
final assessment; 𝜊2𝑖 is assessment of the second level, 𝑖 = 1, 2, … , 𝑛, where n is the number of
𝑗
criteria for the assessment of the EP; 𝜊3,𝑖 is assessment of the third level, 𝑗 = 1, 2, … , 𝑘𝑖 , where 𝑘𝑖 is
𝑟𝑖,𝑘 ,
the number of sub-criteria of criterion i; 𝜊4,𝑖,𝑘𝑖 is assessment of the fourth level when detailing the
𝑖
sub-criteria – initial assessment by experts, where 𝑟𝑖,𝑘𝑖 = 1, 2, … , 𝑚𝑖,𝑘𝑖 , 𝑚𝑖,𝑘𝑖 is the number of details
of the sub-criteria 𝑘𝑖 of criterion i.
The expert group carries out evaluation at the fourth level. Their estimate must belong to the five
element set Г = {𝐼, 𝐹, 𝐸, 𝐵, 𝐴}. Let us denote the expert's estimate of some sub-criterion (this is the
𝑣
fourth level of this graph) as 𝑒4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘
𝜖 Г, where v is the number in the ranked list of experts. This,
𝑖
𝑟𝑖,𝑘𝑖 1 2 3 4 5
𝜊4,𝑖,𝑘 = 𝜛 (𝑓 (𝑒4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘𝑖
, 𝑒4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘𝑖
, 𝑒4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘
, 𝑒4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘𝑖
, 𝑒4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘
)).
𝑖 𝑖 𝑖
We apply the function f to obtain a fourth-level estimate that converts a five-element vector of
expert estimates into an intermediate estimate from a set of Г.
The interim estimate will be obtained as the arithmetic average from the quantitative component
𝑣
𝜂4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘
𝜖 Ф (𝑣 = 1, 2, … , 5) of qualitative expert assessments.
𝑖
The quantitative evaluation thus obtained will be converted to qualitative by means of the function
𝜛: Ф → Г of the following threshold correspondence
𝐼 𝜖𝜑𝐼 = [0; 0,1), 𝐹 𝜖𝜑𝐹 = [0,1; 0,3), 𝐸 𝜖𝜑𝐸 = [0,3; 0,6), 𝐵 𝜖𝜑𝐵 = [0,6; 0,9), (1)
𝐴 𝜖𝜑𝐴 = [0,9; 1],
1 5
𝑟𝑖,𝑘𝑖 𝜂4,𝑖,𝑘 ,𝑟 + …+𝜂4,𝑖,𝑘 ,𝑟
1 5 𝑖 𝑖,𝑘𝑖 𝑖 𝑖,𝑘𝑖
𝜊4,𝑖,𝑘 = 𝜛 (𝑓 (𝜂4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘𝑖
, … , 𝜂4,𝑖,𝑘 𝑖 ,𝑟𝑖,𝑘
)) = 𝜛 ( 5
).
𝑖 𝑖
Let's move on to the formation of the assessment of the third level, the level of sub-criteria. For a
more accurate analysis of the quality of the EP, it is proposed at this level to form 2 estimates for each
𝑗
vertex of the graph. First score 𝜊3,𝑖 (𝑗 = 1, 2, … , 𝑘𝑖 ). The estimate with the function g is also calculated
as the arithmetic-scope mean of the quantitative analogues Ф of the letter vector Г. This,
1 2 𝑚𝑖,𝑘 (2)
𝑗 𝑚𝑖,𝑘𝑖 𝜂4,𝑖,𝑗 + 𝜂4,𝑖,𝑗 + ⋯ + 𝜂4,𝑖,𝑗 𝑖
1 2
𝜊3,𝑖 = 𝜛 (𝑔 (𝜊4,𝑖,𝑗 , 𝜊4,𝑖,𝑗 , … , 𝜊4,𝑖,𝑗 )) = 𝜛 ( ),
𝑚𝑖,𝑘𝑖
where 𝑖 = 1, 2, … , 𝑛, 𝑗 = 1, 2, … , 𝑘𝑖 , 𝑘𝑖 is the number of sub-criteria of criterion i, 𝑚𝑖,𝑘𝑖 is the
number of details of sub-criteria 𝑘𝑖 of criteria i, 𝜛 is a function that converts the obtained quantitative
score into a qualitative analogue Г = {𝐼, 𝐹, 𝐸, 𝐵, 𝐴} a vector according to the rule (1).
We introduce the second assessment to obtain the real attitude to the EP of all stakeholders from
the set Ξ. This second assessment forms a vector of intermediate assessments set by experts, while
maintaining the structure. This will help to showing nodes on the EP graph that need special attention.
It is possible to save the assessment of all details of the sub-criteria for each of the five expert groups
at the fourth level:
𝑣 𝑣 𝑣 𝑣 𝑣
𝜊̅4,𝑖,𝑗 = 𝜛 (𝑔(𝑒4,𝑖,𝑗 , 𝑒4,𝑖,𝑗 , 𝑒4,𝑖,𝑗 , … , 𝑒4,𝑖,𝑗 )).
When moving to the third level, we use the resulting vector for evaluating each of the detailing
sub-criteria:
𝑗 1 2 3 4 5 (3)
𝜊 ∗ 3,𝑖 = 𝜛[𝑓(𝜊̅4,𝑖,𝑗 , 𝜊̅4,𝑖,𝑗 , 𝜊̅4,𝑖,𝑗 , 𝜊̅4,𝑖,𝑗 , 𝜊̅4,𝑖,𝑗 )],
𝑖 = 1, 2, … , 𝑛, where n is the number of criteria, 𝑗 = 1, 2, … , 𝑘𝑖 , where 𝑘𝑖 is the number of sub-criteria
of criterion i.
We form the vector of estimates of the expert group by sub-criteria:
𝑣 𝑣 𝑣 𝑣 𝑣
𝜊̅3,𝑖 = 𝜛 (𝑔(𝜊̅4,𝑖,1 , 𝜊̅4,𝑖,2 , 𝜊̅4,𝑖,3 , … , 𝜊̅4,𝑖,𝑘 𝑖
)).
It is quite possible that with a wide scatter of expert assessments, when some put "A" and others -
"E" (in fact, this speaks of "confusion" and misunderstanding by some experts of the assessment
𝑗 𝑗
area), evaluation 𝜊3,𝑖 and 𝜊 ∗ 3,𝑖 . This is an indicator of the urgent need to eliminate the shortcomings of
the EP and stimulates the improvement of the quality of the evaluated program.
Let's move on to the second level assessment:
1 1 𝑘 (4)
1 2 𝑘 𝜂3,𝑖 + 𝜂3,𝑖 + ⋯ + 𝜂3,𝑖𝑖
𝜊2𝑖 = 𝜛 (𝑔(𝜊3,𝑖 , 𝜊3,𝑖 , … , 𝜊3,𝑖𝑖 )) = 𝜛 ( ).
𝑘𝑖
As a result, we will obtain n assessments of all criteria and a mandatory second assessment,
revealing the opinion of expert groups Ξ:
1 1 5
𝜂3,𝑖 + 𝜂3,𝑖 + ⋯ + 𝜂3,𝑖
𝜊 ∗ 𝑖2 = 𝜛 (𝑓(𝜊̅3,𝑖 1 2
, 𝜊̅3,𝑖 5
, … , 𝜊̅3,𝑖 )) = 𝜛 ( ).
5
We form a vector of intermediate estimates of all expert groups (𝑣 = 1, 2, … , 5) by criteria at the
second level similar to the third level (3):
𝜊̅ 𝑣 = 𝜛 (𝑔(𝜊̅ 𝑣 , 𝜊̅ 𝑣 , 𝜊̅ 𝑣 , … , 𝜊̅ 𝑣 )). (5)
2 3,1 3,2 3,3 3,𝑘𝑖
𝑖 ∗𝑖
We compare the estimates 𝜊2 and 𝜊 2 , we identify to the "problem zones" of the EP.
𝛾 𝑛
We obtain a vector {𝑎2,𝑖 } , (𝛾𝜖Г = {𝐼, 𝐹, 𝐸, 𝐵, 𝐴}, 𝑖 = 1, 2, … , 𝑛), n is the number of criteria,
𝑖=1
𝐵
when evaluating each criterion separately. The designation 𝑎2,1 , indicates that the first criterion of the
EP was rated "B" (index 2 indicates the second level of assessment). The final estimate at the top level
will be based on this vector. Consider an indicator of the level of the EP, which is an increasing
discrete function, the values of which correspond to the estimates obtained according to the criteria.
𝛾 𝛾
Consider the segment corresponding to the positive score 𝑎2,𝑖 : 𝑎2,𝑖 𝜖[0,1], (𝑖 = 1, 2, 3, … , 𝑛) on the
five-place scale 𝛾 ∈ Г. The point τ divides the evaluation segment into two areas: the area 𝐼𝑛𝛾 , lying
to the left of the τ, where the EP criteria are lower than required by a given quality, and the area 𝐼𝑑𝛾
lying to the right of the τ, the level of the criteria of which exceeds the required boundary of a
sufficient assessment. The criteria in area 𝐼𝑛𝛾 (𝛾 ∈ Г) are necessary for obtaining the final o rating
and the criteria in area 𝐼𝑑 𝛾 are sufficient.
𝛾
𝑎 𝛾
We will enter a logical variable 𝐵𝑖 2,𝑖 (where 𝑎2,𝑖 is the ball corresponding to the quality level for
this criterion, i is the criterion number 𝑖 = 1, 2, 3, … , 𝑛, n is the number of criteria, 𝛾 ∈ Г) as an
𝛾
𝜊
indicator of the necessary and sufficient condition for rating. 𝐵𝑖 2,𝑖 = 1, if the quality of the EP
according to the criterion i is necessary and mandatory when evaluating by the -th point and belong
𝛾 𝛾
to the area 𝐼𝑛𝛾 : 𝑎2,𝑖 𝜖𝐼𝑛𝛾 , Similarly, if 𝑎2,𝑝 𝜖𝐼𝑑𝛾 , and the scores for the criterion 𝑘𝑝 are sufficient when
𝛾
𝜊
judging by the γ-th score, then 𝐵𝑝 2,𝑝 = 0. We obtained a final assessment of the EP level, taking into
account the necessary and sufficient condition for issuing the assessment based on the assessments
𝛾 𝑛
according to the criteria {𝑎2,𝑖 } , n is the number of criteria:
𝑖=1
̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅ (6)
𝛾 𝑛 𝑛 𝜃 𝑛 𝑛
𝛾 𝑘 𝑘 𝑑 𝑑
𝜊 =⋀ ⋀ 𝑎2,𝑗 ∨ ⋀ 𝑎2,𝑖 ⋀ ⋀ 𝑎2,𝑣 ∨ ⋁ 𝑎2,𝜎 ,
𝑘=𝛼 𝑗=1 𝑖=1 𝑑=𝛾+1 𝑣=1 𝜎=1
𝑎𝑘 𝑎𝑘 𝑎𝑑 𝑎𝑑
2,𝑗
𝐵𝑗 =1 ( 𝐵𝑖 2,𝑖 =0
)] [ ( 𝐵𝑣 2,𝑣 =1 )] 𝐵𝜎 2,𝜎 =0
[
where 𝛾𝜖Г is the scale score {𝐼, 𝐹, 𝐸, 𝐵, 𝐴}; 𝛼 is the initial value of the scale for which the positive
score is set, 𝛼 ≤ 𝜏 (𝜏 = "B"); 𝜃 is the final value of the entered scale 𝜃 ≥ 𝑖. It is understood that 𝜏 =
"A". The criteria may be that are not calculated for the highest score. If the knowledge of the test
subject does not correspond to the required level and 𝜊 𝛼 = 0, according to formula (6), the score is set
as a point below the established lower rating limit 𝛼: 𝜊 𝛼−1 = 1.
5. System for supporting the process of internal evaluation of the quality of
educational programs
On the basis of the developed assessment model, an information system is proposed for conducting
an internal assessment of EPs. System contains:
• information on stakeholder experts involved in quality assessment procedures;
• a system of criteria for assessing the quality of an EP with the possibility of editing and
adjusting them to the characteristics of a particular EP;
• means for providing stakeholders with access to the documents necessary for the
assessment structured according to the evaluation tree;
• support for the logic of assessment inference procedures;
• identification of bottlenecks in the EP and contradictions between the assessments of
various groups of stakeholders;
• storage of information and statistical processing of the results of examinations.
The information system consists of the following components:
1. A database containing stakeholder information, assessment criteria, and the results of detailed
sub-criteria assessments.
2. Document database containing documents structured according to detailed sub-criteria.
Stakeholders are given access to these documents.
3. The assessment planning unit is designed to prepare an assessment, determine the documents
required for each level of detail, form a set of stakeholders for the assessment and the main
assessment criteria.
4. Synchronization unit for determining if there is already complete information for the
assessment. In case of lack of data, the system will generate a preliminary assessment.
5. The unit for forming an assessment by the formulas (2) - (6) performs the formation of
quantitative indicators at the second, third, and fourth levels of assessment and translates them into
a qualitative scale.
6. The analysis unit is designed for statistical analysis of the history of the assessment,
identifying the most critical areas of the EP that is being assessed.
7. Output unit - generates reports and screen forms at the request of users.
The assessment process is preceded by the preparation of the documents necessary for the
assessment of each level of detail. Input information for the operation of the system - estimates of
stakeholders at tree levels, defined as leaves.
As a result of the assessment, the system can provide the following information:
• the final assessment of the EP and its vector of assessments concerning all stake-holders;
• assessment of each of the criteria and sub-criteria and their details;
• tables for the formation of the final assessment, individual criteria, and sub-criteria;
• vector of interim assessments of stakeholders at each level of assessment;
• a list of conflicting criteria and sub-criteria on which the assessments of various
stakeholder groups had the greatest differences;
• a list of criteria and sub-criteria that are most critical and have had a negative impact on
the overall assessment.
The scheme of the database is the system presented in Figure 2.
Figure 2: Scheme of the database of the support system for internal evaluation of EPs
Table 1 shows the result of the program in the evaluation of sub-criterion 3.3 (Fig. 1). Thus, in as a
result of the operation of the system using the formulas (2) we obtain a third level estimate for the
third sub-criterion of the third criterion "E" with a quantitative analogue of 0.5 points. Applying this
technique, we obtain intermediate evaluations at the third level, then - at the second (4). The final
grade is set taking into account the necessary and sufficient conditions for making a grade (6) on the
basis of grades by criteria. This is how the vector of second level evaluations is obtained
𝑜2 = {𝐴, 𝐵, 𝐸, 𝐵, 𝐵, 𝐸, 𝐴, 𝐵, 𝐵, 𝐵, 𝐸}, final assessment 𝑜1 = "𝐵".
Table 1
The result of the evaluation of sub-criterion 3.3 "Rules for recognition of learning outcomes obtained
in other educational institutions"
Sub- Evaluation
Detailing Summary
criteria 3.3 leadership employers graduates teachers students
0.5 (E) 1 0,4 (E) E (0,5) F (0.25) E (0,5) E (0,5) F (0.25)
2 0,65 (B) B (0,75) E (0,5) E (0,5) B (0,5) B (0,75)
3 0.4 (E) E (0,5) E (0,5) F (0.25) E (0,5) F (0.25)
4 0.5 (E) B (0,75) E (0,5) E (0,5) B (0,75) I (0)
Summary 0,67 (B) 0,5 (E) 0,42 (E) 0,67 (B) 0,33 (E)
The resulting vector of assessments made it possible to determine the problem areas of the EP. For
the example considered, attention should be paid to expanding the opportunities for access to EP and
recognition of learning outcomes, advanced training of human resources, a clear formulation of the
prospects for the development of EP. Similarly, the assessments of each of the sub-criteria can be
analyzes and the corresponding component of the EP can be adjusted.
Next, the system for supporting the process of internal evaluation of the quality of EPs issues a
∗
final score for five stakeholders (5), 𝑜1 = {𝐸, 𝐴, 𝐴, 𝐵, 𝐵}. Thus, as a result of this feedback, it is
possible to adapt the EP to the various goals of the stakeholders.
6. Conclusion
The analysis of the stakeholders of the educational institution made it possible to identify the main
stakeholders who could potentially participate in the assessment of the quality of EP. The proposed
information system is designed to support the internal assessment of the quality of EPs by higher
education institutions. The system allows combining information about the stakeholders of an
educational institution with an assessment of their satisfaction with a particular EP. The main purpose
of the system's usage is an integrated approach to the internal assessment of the EP quality and
reducing the subjectivity of the final assessment.
The system allows storing information about the assessment according to the system of criteria
defined by regulatory documents. The advantages of the system are centralized storage and the ability
to analyze information obtained during the assessment, if necessary, to carry out a partial assessment
of any element. The adaptability of the system to changes in regulatory legislation is determined by
the ease of changing any component of the evaluation tree and the ability to expand the tree by adding
new nodes. The system is also adaptable to changes in the depth of the evaluation tree. The
information obtained as a result of the analysis can become the basis for identifying problem areas
and improving educational activities for this program.
The system can be used to monitor the satisfaction of the educational process by all stakeholders.
The number of stakeholder groups can also be expanded as needed. Needs further development of the
issue of improving the methods of static analysis of the obtained estimates and the use of expert logic
in the formation of intermediate assessments at different levels of quality assessment of EP.
7. References
[1] Standards and Guidelines for Quality Assurance in the European Higher Education Area
(ESG), 2015. URL: https://www.enqa.eu/wp-content/uploads/2015/11/ESG_2015.pdf.
[2] M. Gaebel, Th. Zhang, Trends 2018. Learning and teaching in the European Higher
Education Area, 2018. URL: https://eua.eu/downloads/publications/trends-2018-learning-
and-teaching-in-the-european-higher-education-area.pdf.
[3] C. Tomas, M. Kelo, ESG 2015–2018 ENQA agency reports: Thematic analysis. ENQA
Occasional Paper 28, Brussles, Belgium, 2020. URL:
http://www.enqa.eu/index.php/home/esg/.
[4] O. Krykliy, K. Kyrychenko, O. Liuta, M. Mazurkiewicz, V. Yuskaiev, The State of the
Development, Structure and Functionality of Internal Quality Assurance Systems in
Ukrainian Higher Education Institutions. Quality Assurance in Higher Education Institutions
in Ukraine 2016 through the prism of European Guidelines and Standards ESG 2015.
EXANTE. Wrocław, 2016. URL: https://projects.lnu.edu.ua/quaere/wp-
content/uploads/sites/6/2018/03/QA-in-Ukraine-reports.pdf.
[5] D. E. Sitnikov, V. M. Demina, Logical approach to the assessment of knowledge according to
the R-point system [in Russian], Bulletin of the National Technical University : Kharkov
Polytechnic Institute Systems analysis, management and information technology, 125,
Kharkiv, KPI, 2000, pp. 41–45.
[6] F. Henard, D. Roseveare, Fostering Quality Teaching in Higher Education: Policies and
Practices, 2012. URL: https://www.oecd.org/education/imhe/QT%20policies%20
and%20practices.pdf.
[7] E. W. Mainardes, H. Alves, M. Raposo, An Exploratory Research on the Stakeholders of a
University, Journal of Management and Strategy, 1, volume 1, 2010, pp. 76–88. URL:
http://www.sciedu.ca/journal/index.php/jms/article/view/15.
[8] I. Marić, Stakeholder analysis of higher education institutions. In Interdisciplinary
Description of Complex Systems, 2, volume 11, 2013, pp. 217–226. URL:
https://hrcak.srce.hr/100637.
[9] O. Baturin, , V. Ivanov, V. Lyfar, S. Mytrokhin, L. Lyhina, Functional model of definition of
integral indicators of quality of educational activity and quality of higher education by
institutions of higher education, Scientific news of Dalev University, volume 18, 2020. URL:
http://nvdu.snu.edu.ua/wp-content/uploads/2020/02/2020-18-3.pdf.
[10] A.I. Baturin, V.O. Lyfar, O.I. Zakhozhay, V.H. Ivanov, «A Uniform Interaction Model Of
Educational Process Agents In The Unified Management System Of A Higher Education
Institution», Information Technologies and Learning Tools, 4, volume 78, 2020, pp. 266–277.
URL: https://journal.iitta.gov.ua/index.php/itlt/article/download/3178/1704.
[11] F. Hénard, Learning our Lesson: Review of Quality Teaching in Higher Education, OECD
Publishing, Paris, 2010. URL: https://www.oecd.org/education/imhe/43961761.pdf.
[12] N. Komleva, V. Liubchenko, S. Zinovatna, Improvement of Teaching Quality in the View
of a Resource-Based Approach, 16th International Conference on ICT in Education,
Research, and Industrial Applications, Kharkiv, Ukraine, 2020, pp. 262–277. URL:
http://ceur-ws.org/Vol-2740/20200262.pdf.
[13] W. Tamrat, The exigencies of student participation in university governance: Lip services
and bottlenecks, School of Graduate Studies, St. Mary's University, Addis Ababa, Ethiopia,
Higher Education Quarterly, 1, volume 74, 2020, pp. 35–47. URL:
https://www.researchgate.net/publication/336695816_The_exigencies_of_student_participati
on_in_university_governance_Lip_services_and_bottlenecks.
[14] A. Sperlich, K. Spraul, Students As Active Partners: Higher Education Management in
Germany Political Science, volume 12, issue 3, article 11, 2007, pp. 1–19 URL:
https://www.researchgate.net/publication/237517180_Students_As_Active_Partners_Higher_
Education_Managem–ent_in_Germany.
[15] Verkhovna Rada of Ukraine: On approval of the Regulation on accreditation of educational
programs for which higher education applicants are trained: The Law “On Education”,
880/33851, 2019. URL: https://zakon.rada.gov.ua/laws/show/z0880-19#Text.