=Paper=
{{Paper
|id=Vol-2300/Paper49
|storemode=property
|title=Research of the Agree of Experts’ Evaluations in the Estimation of Software Systems
|pdfUrl=https://ceur-ws.org/Vol-2300/Paper49.pdf
|volume=Vol-2300
|authors=Svitlana Krepych,Iryna Spivak,Roman Krepych
|dblpUrl=https://dblp.org/rec/conf/acit4/KrepychSK18
}}
==Research of the Agree of Experts’ Evaluations in the Estimation of Software Systems==
https://ceur-ws.org/Vol-2300/Paper49.pdf
203
Research of the Agree of Experts’ Evaluations in the
Estimation of Software Systems
Svitlana Krepych1, Iryna Spivak2, Roman Krepych3
Department of Computer Science, Ternopil National Economic University, UKRAINE, Ternopil, 8 Chekhova str., email:
s.krepych@tneu.edu.ua1, spivak.iruna@gmail.com2, jagmstar@gmail.com3
Abstract: The article deals the task of comparing of
agree of the results of expert evaluation conducted by II. STATEMENT OF THE TASK
several independent groups of experts of the software of The evaluation of the software system quality begins from
teacher’ rating of higher educational institutions. The the early stages of development, including the definition of
resulting evaluation of the expert group calculated by the the specification requirements for the software system,
modified method of expert evaluation of software systems analysis, design, etc. The quality estimation process at these
based on interval data analysis. The resulting evaluation stages usually involves design engineers who themselves
of expert groups will can improve the "weak" aspects of evaluate some part of the system they are developing. After
the software system and can help conduct analyze the that, the results of their estimations are integrating and
expert assessments too. averaging. Of course, the resulting evaluation in this case will
Keywords: expert evaluation, methods of interval data be overestimated [5].
analysis, software system, teacher rating system. Approaches and recommendations for the process of expert
evaluation of the software systems quality in various
I. INTRODUCTION scientific schools, studies by individual scientist cover
One of the important tasks of the present is the research of various aspects of this issue. Some works try to formulate
one of the most important characteristics of software systems recommendations for the evaluation process. In the work [6]
- quality. Under the quality of the software system, we a critical analysis was made to cover all the attributes of the
understand the set of the software product properties, which software systems quality that need to be accounting in the
characterizes its ability to meet the established or predicted expert evaluation of systems. In particular, in the paper [7]
needs of the customer, which he expressed in the form of user we consider the methodology for evaluating the quality of
requirements for software in the early stages of its web-projects. The authors proposed to evaluate quality not by
development [1]. One of the most important attributes of a formal numerical measurements, but in the form of relations
software system's quality is the functional capability, that is, and preferences with the application of the logic of antonyms
the ability of the system to perform its functions over a [8]. The drawback of this method, in our opinion, is the using
certain period of operation within predetermined limits and limit. It only can be use to obtain an end-user evaluation. In
under certain operating conditions [2, 3]. Nowadays, a large the works [9-11] a method of calculating the quality
number of leading scientists of Ukraine and the world is evaluation of software systems by a set of criteria from
involved in the study of these issues and expert evaluation is different groups of participants in the development process is
a common method for estimating the quality of the software proposed. The results of the expert evaluation of software
system. Expert (lat. Expertus - experienced) evaluation systems quality according to various criteria by this method
(expertise) - a method of obtaining summary information by also allowed to developing a method of visualization of the
the way of estimation a situation, event or phenomenon by a information on the estimates using polar diagrams. However,
group of independent experts [4]. Such generalized this method has a number of remarks that discussed in the
information is obtaine through an expert survey, which work [12].
involves specialists in the area that interests the researcher. One of the important drawbacks of the method is the exact
The task of an expert is to formulate his own opinion about formalization of the expert's opinion when evaluating a
the object under research or the phenomenon on a certain particular criterion [12]. The paper [6] provides information
scale in accordance with the prescribed rules. The main on the real economic and environmental impacts that may
problem of expert evaluation is the choice of competent in occur not only in the life of some company but also ordinary
the study area experts, which have an unbiased attitude to the people, with insufficient evaluation of a particular project at
object of research and had a critical attitude towards all that various stages of its development. Therefore, the choice of
evaluates, especially if it is a study of software systems experts and giving him the rights for objectively evaluate the
whose lowly quality and may endanger the life of a person or proposed software systems is extremely actuality.
of humanity [4]. To take into account the possible questionable expert
In view of the above, the task of studying agree of expert’ evaluations according to certain criteria, the method of expert
evaluations of different groups in estimating software evaluation of the software systems quality to be realization
systems of any practical or theoretical direction is actuality. using the methods of interval data analysis are proposed
[3,12-14].
ACIT 2018, June 1-3, 2018, Ceske Budejovice, Czech Republic
204
Interval evaluation of the software system based on the set
up of the upper and lower limits [14] of the permissible
expert estimation according to a certain criterion:
x m,k ∈ [ x min ; x max ] , (1)
where x m,k ∈ [1..10] - the evaluation is set by the expert on a
certain criterion; m - the number of criteria for evaluation; k
- the expert; x min = x m,k − δ ⋅ x m,k ; x max = x m,k ; δ - the
percentage of deviation from evaluation the set by the expert,
which can be determined for each project or expert,
depending on the "degree of trust" to the expert.
The resulting expert evaluation has the form:
[ X k ] = ∑ [ xmin ; xmax ] ⋅ cm , (2)
m
− +
Fig. 1. The process of expert evaluation
where [ X k ] = [ X k ; X k ] - interval evaluation of the expert of
the area, which takes into account the percentage of the
expert's rating deviation from the nominal value; cm -
percentage indicator of importance of the criterion of
software evaluation, ∑ c m = 1 .
m
The resulting evaluation of the software system quality in
this case will take the form:
∑ [ X k ] ⋅ qk
[X ] = k , (3)
∑ qk Fig. 2. The main window of the system of teacher’ rating
k
where q k - the weight indicator of the individual group;
On Fig. 3 is provide the general rating of university
− +
[ X ] = [ X ex ; X ex ] - the resulting evaluation of all groups of teachers.
experts, which is the interval of confidence to the software
developing.
The condition of the agree of the obtained interval
estimation of the software system quality to the admissible is
[3,6]:
− +
[ X ex ; X ex ] ⊂ [ X min ; X max ] , (4)
where [ X min ; X max ] - the established interval of software
evaluation, which is guaranteed to satisfy software
developers.
III. EXAMPLE OF THE EXPERT EVALUATION OF THE
SOFTWARE OF TEACHER’S RATING OF HIGHER
EDUCATIONAL INSTITUTIONS
In Fig. 1 schematically illustrated the process of expert
evaluation the quality of the software system.
As we can see from the figure, the resulting evaluation of Fig. 3. The general rating of university teachers
the software system quality depends on the estimates made Evaluation of the work of the scientific and pedagogical
by experts at the initial stage. workers of the university has a cumulative character, takes
We will conduct a comparative analysis of evaluations put into account data for 5 years and it calculated by the
forward by different groups of experts on the example of the following formula:
software of teacher’ rating of higher educational institute. 4
Shortly about the rating system of teachers. The first version I = P + ∑ 2 −i Pi , (5)
of this system developed by the Master of the Department of i =1
Computer Science in 2017. In Fig. 2 shows the main window where I - the evaluation of the work of the scientific and
of the system. pedagogical workers for the last 5 years; P - the evaluation
of the work of the employee for the reporting year; Pi - the
evaluation of the work for the previous 4 years.
ACIT 2018, June 1-3, 2018, Ceske Budejovice, Czech Republic
205
The work of the teacher evaluated in the context of such one in order to take into account the "doubts" of experts of
activities as research work, educational work, methodical the setting score.
work, organizational work, qualifications and additional TABLE 1. EVALUATION OF THE SOFTWARE SYSTEM BY
criteria. On fig. 4 showed one of the windows for capability EXPERTS FROM THE DEPARTMENT 1
reporting.
№ EA BA SA SA EI EI
1 9 9 8 9 8 8
2 9 9 8 7 8 7
3 9 8 9 9 9 8
4 8 8 9 9 7 9
5 9 9 8 9 8 8
6 8 7 9 9 9 8
7 8 8 8 8 8 8
8 9 9 9 9 8 9
9 9 8 9 7 9 9
10 9 8 8 8 8 9
Weight 7 8 9 9 9 9
indicator of
each expert
The resulting [8,35;8,8] [7,79;8,2] [7,91;8,33] [7,96;8,38]
evaluation of
Fig. 4. Form to fill out the report by the teacher a group of
experts
To evaluate the quality of this software system in an The resulting [7,97;8,39]
objective way and the possibilities for its further perfection evaluation of
and improvement, it decided to conduct its evaluation. the project
Leading departments of the Faculty of Computer Information TABLE 2. EVALUATION OF THE SOFTWARE SYSTEM BY
Technologies were in the role of expert groups. EXPERTS FROM THE DEPARTMENT 2
The software system evaluated according to the following
№ EA BA SA EI
criteria [12]:
1. The correctness of the work (the system must be 1 7 8 8 7
isolated from external influences and the result of performing 2 8 9 7 8
the functions should be correct in all conditions). 3 8 8 7 8
2. Protection from unauthorized access 4 7 8 8 7
3. Program reliability (the system must be resistant to 5 7 9 7 7
various user-side influences) 6 7 9 7 7
4. Comfortable graphical user interface 7 7 9 8 7
5. Low cost of hardware resources (the system should not 8 7 7 7 8
require high hardware costs of the computer) 9 8 8 7 7
6. Mobility (the system should have a small amount of 10 7 8 7 7
memory, a small amount of processor time, etc., so that it can Weight indicator 7 8 9 9
of each expert
be used on any PC)
The resulting [6,84;7,2] [7,7;8,1] [6,8;7,2] [6,8;7,2]
7. Scalability (improving the capabilities of the system by evaluation of a
introducing a new functional) group of experts
8. Convenience of use The resulting [7,1;7,4]
9. Speed evaluation of the
10. Completeness of functional requirements (the system project
must meet all of its functional requirements from the side of
TABLE 3. EVALUATION OF THE SOFTWARE SYSTEM BY
the subject area). EXPERTS FROM THE DEPARTMENT 3
The software system was evaluated by the experts of the
following categories: expert in the area (in the table - EA), №
business analyst (BA), software architect (SA) and expert of
BA
EA
EA
EA
SA
SA
SA
EI
EI
EI
user interface (EI). Percentage coefficient importance of the
criterion cm for each of the criteria listed above, accordingly 1 9 10 8 8 9 9 8 7 9 9
0,05; 0,05; 0,05; 0,1; 0,05; 0,05; 0,05; 0,1; 0,1; 0,4. System 2 8 9 7 4 8 3 7 9 7 8
developers want to achieve system quality with a minimum 3 8 9 5 4 8 5 8 7 7 8
threshold of 80%. Below in the tables 1-3 shows the upper 4 8 7 7 4 9 8 6 7 9 8
limit for the expert's estimate. The lower limit of the 5 9 10 10 8 9 8 8 8 9 9
evaluation will formed with a deviation 5% from the upper 6 7 10 8 4 7 8 6 4 6 7
7 8 10 10 7 8 9 7 5 8 8
ACIT 2018, June 1-3, 2018, Ceske Budejovice, Czech Republic
206
8 8 9 8 7 9 8 7 6 7 8 [3] S. Krepych, A. Dyvak, M. Dyvak, I. Spivak, “The
9 8 10 10 8 8 7 8 8 6 7 method of providing of functional suitability of
10 8 10 7 7 7 9 8 7 6 7 elements of the device of formation of signal in
The [7,6;8,1] [7,5;7,9] [7,4;7,8] [6,8;7,1] electrophysiological way of classification tissues
result- surgical wound”, 13th International Conference
ing Perspective Technologies and Methods in MEMS
evalu- Design, MEMSTECH 2017 Proceedings, pp.183-186,
ation 2017.
of a [4] I. Galyan, "Psychodiagnostics: Textbook," Kyiv
group
Academic Edition, p. 464, 2011.
of
exp-
[5] Y. Ryabokin, “Software cost estimation”, Electrical and
erts Automation system, vol. 1(82), pp.117-124, 2015.
The [7,2;7,6] [6] Grytsyuk Yu., Gritsyuk P., "Modern problems of
result- scientific evaluation of the applied software quality",
ing Scientific Bulletin of the NLTU of Ukraine "Information
evalu- Technologies and Modeling in Economics", №. 25/7, pp.
ation 284-294, 2015
of the [7] Berko, A., Alekseeva, K., "Estimation of the
pro- information resources quality in WEB-projects",
ject
“Actual problems of economy”, №10 (136), pp.226-234,
Consequently, from the expert evaluation of the rating
2012.
system of the university teachers, the main quantitative
[8] Golota Y., Tysenko V., Falkov D., "The logic of
estimates of which given in the tables, we can conclude that
antonyms is the theoretical basis for the formation of
in fact the results of the evaluation of only one of the three
complex assessments based on expert estimates of
expert groups are consistent with the initially established
individual parameters," Modeling of intellectual
interval of evaluation of the software system, which satisfies
processes of design and production: Materials II
the developers.
internationally scientific-practical conference, Minsk,
IV. CONCLUSION p.166-167, 1998.
The paper is devoted to the problem of studying and [9] I. Morgun, “Method of expert evaluation of software
evaluating the quality of software systems. The methods of quality”, Materials of the International Scientific and
expert evaluation are considered. It indicated that most of Practical Conference of Postgraduate Students and
them carry only theoretical recommendations for improving Students “Software Engineering 2011”, vol.2(6),
the evaluation process. Some methods aimed at improving pp.117-124, 2011.
the visual presentation of the evaluation results. The results [10] Morgun I., Botsula M., "New method and information
of the analysis showed the actuality and importance of paying technology for data processing for quality management
special attention to the selection of experts who would of electronic training courses", International scientific
evaluate software systems so that their opinion was objective and technical magazine "Information technologies and
and unbiased. It proved that, in order to avoid "doubts" computer engineering", №3, pp.25- 33, 2014.
regarding the evaluation of a particular criterion, use the [11] Grytsyuk Yu., Buchkovskaya A., "Visualization of the
methods of interval data analysis. The method of expert results of expert evaluation of software quality using
evaluation of software systems based on the analysis of polar diagrams", Scientific Bulletin of NLTU of Ukraine,
interval data is developed, the result of which is to check the Vol.27, No.10, pp.137- 145, 2017.
agree of the resulting interval estimation according to the [12] I.Spivak, S.Krepych, S. Budenchuk, “Methods and
project proposed by an independent group of experts and the means of expert evaluation of software systems on the
set the interval of evaluation of the software system that basis of interval data analysis”, 14th International
satisfies the developers. On the example of the expert Conference on Advanced Trends in Radioelectronics,
evaluation of the rating system of the university teachers, the Telecommunications and Computer Engineering
diversity of opinions of different groups of experts shown. pp.101-127, 2018
[13] S. Krepych, I.Spivak, “Estimation of the time
REFERENCES complexity of the Monte Carlo method and interval
[1] Pomorova, O.V., Hovorushchenko, T.O, “Modern analysis of data for determining the functional
suitability of REC”, Modern Computer Information
problems of software quality assessment”,
Technologies: Materials of the Third All-Ukrainian
Radioelektronni I kompiuterni systemy, no.5, pp.319-
School-Seminar for Young Scientists and Students
327, 2013.
ACIT’2013, Ternopil, pp.36-37, 2013.
[2] International Standard ISO/IEC 9126. Information
[14] I. Spivak, M. Dyvak, “Tolerance estimation of the
technology – Software product evaluation – Quality
parameters of “input-output” dynamic model on the
characteristics and guidelines for their use. International
basis of interval data analysis”, Proceeding of
Organization for Standartization International
International Conference CADSM’2005, Lviv-Polyana,
Electrotechnical Commission, Geneva, 1991.
pp.151-153, 2005.
ACIT 2018, June 1-3, 2018, Ceske Budejovice, Czech Republic