=Paper=
{{Paper
|id=Vol-2577/paper8
|storemode=property
|title=A posteriori determination of expert competence under uncertainty
|pdfUrl=https://ceur-ws.org/Vol-2577/paper8.pdf
|volume=Vol-2577
|authors=Hryhorii Hnatiienko,Vitaliy Snytyuk
|dblpUrl=https://dblp.org/rec/conf/its2/HnatiienkoS19
}}
==A posteriori determination of expert competence under uncertainty==
https://ceur-ws.org/Vol-2577/paper8.pdf
82
A posteriori determination of expert competence under
uncertainty
© Hryhorii Hnatiienko[0000-0002-0465-5018] and © Vitaliy Snytyuk[0000-0002-9954-8767]
Taras Shevchenko National University of Kyiv, Kyiv, Ukraine
g.gna5@ukr.net, snytyuk@gmail.com
Abstract. The problem of determining the relative competence of experts in ex-
pert assessment tasks is considered. The classification of competence determina-
tion methods is offered and the use of objective approaches to a posteriori deter-
mination of these coefficients is substantiated. The problem of determining the
objects resultant ranking and ways of getting its solutions are given. Based on the
analysis of obtained solutions in the group ranking problem, we propose a method
of relative competence coefficients determining for experts in the form of fuzzy
set membership functions. An example that demonstrates the effect and features
of the described method application is shown.
Keywords: expert evaluation, resultant ranking, expert competence, decision
making, fuzzy set membership function.
1 Introduction
Determining expert competence ratios is an important element in solving expert assess-
ment problems. The accuracy of determining the relative competence of experts affects
the outcome of the task, the reliability of the solution, and the credibility of result. Tak-
ing into account competence of experts is the key to the quality of decision making and
has a significant impact on the outcome of solution in the expert assessment problems.
Therefore, determining of experts competence and their reasoned consideration is an
actual area of research and can greatly contribute to improving the adequacy of the
analysis of expert evaluation results and their validity.
2 Classification of methods for determining of experts
competence
Determining the experts competence is an important element of expert assessment. This
factor significantly influences to decision making results and requires in-depth study,
research and development. Today, there are several approaches to determining the rel-
ative competence of experts, but each of directions is not perfect:
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License
Attribution 4.0 International (CC BY 4.0).
83
– documentary is often distrustful to ways of formalizing and taking into account
objective data on experts, and is also determined by the current tendency to devalue
official documents and the uncriticality of some institutional decisions;
– self-assessment, during which often yields information about the level of self-con-
fidence of the expert, rather than his real competence;
– mutual evaluation, by which in some problems it is possible to detect confrontation
in the expert committee and the existence of coalitions among its members, which
sometimes distort the true individual competence of experts, which really has to influ-
ence the results;
– sometimes combined methods, instead of synergistic effects, lead to a cumulative
error;
– objective approaches, which are easy to apply and reasonably justified, are the
most reliable to calculate adequate indicators of the relative competence of experts.
Objective approaches distinguish between a priori and a posteriori methods of cal-
culating competence. Let us dwell on the methods of a posteriori determination of the
experts competence developed by the authors. Objective approaches to the posteriori
determination of expert competence ratios include:
– study of the expert's participation effectiveness in previous examinations;
– calculation of competence based on the results of control examination;
– study of the expert's participation results in a specific examination.
The latter approach, which is to investigate the participation of an expert group mem-
bers in a particular examination, may in turn be regarded as:
– indirect analysis of indicators without first aggregating results;
– the ratio of initial individual expert data and calculated integral indicators.
In the problems of determining the group ranking of objects as components of an
approach in which the relationship of individual expert data and calculated integral in-
dicators are investigated are:
– calculation of distances to pair comparison matrix (MCM) based on the found re-
sultant ranking (in different metrics and by different criteria);
– a posteriori determination based on the objects resultant ranking by given individ-
ual expert rankings:
= analysis of the inversions number in the expert ranking relative to the resulting
ranking;
= determination of the number of objects consistently ranked by the expert;
= calculating the sum of difference modules between the objects rank in the re-
sulting ranking and the expert's given rankings;
– determination of individual relative estimates for collectively specified relative es-
timates:
= calculating the intervals sum of relative estimates;
= determining the maximum interval among relative estimates;
= calculating the "volume" of the intervals of the determined relative estimates;
= calculation of the "area" of the intervals of relative estimates;
= determining the deviation level of the MCM from a given "ideal" vector of rel-
ative estimates;
– distance to the group vector of weighting coefficients when given by the experts
of MCM in cardinal scales;
– the relative distance from expert rankings to the resultant rankings.
84
The modeling and study of the last two approaches are discussed in this paper. Such
approaches are characterized by a situation of uncertainty. It is generated by the pres-
ence of a large number of methods developed and substantiated by different authors for
a posteriori calculation of the coefficients of relative competence of experts. At the
same time, it is not possible to determine for certain practical situations precisely and
precisely which of the methods developed is the most appropriate. Therefore, the au-
thors propose to apply in the previous stages of the analysis all known methods of de-
termining the relative competence of experts and only in the last stages of the analysis
to aggregate the information obtained by different methods.
3 Formulation of the task of determining the resultant ranking
by group evaluation
The methods of processing expert information are divided into three main groups [1]:
– statistical methods;
– scaling methods;
– algebraic methods.
The essence of algebraic methods is that the set of admissible estimates sets the dis-
tance and the resulting score is defined as the distance to which the expert estimates for
the selected criterion are minimal. On the basis of algebraic approach we will determine
the coefficients of relative competence of experts.
l ∈ L = {1,..., k }, k > 1,
Let k experts with index set A their preferences for the
set of n objects in the form of object rankings R , l ∈ L.
l
To determine the distance between object rankings, researchers use:
– Cook metric for mismatchingj of lobject rankings in individual rankings
d (R , R ) = ri j − ri l ,
i∈ I (1)
ri l Rl , l ∈ L, 1 ≤ ril ≤ n,
where – is the rank of the i-th object in the ranking of the l-th expert
– Hamming's metric
( )
d B j , B l = 0,5 bisj − bisl ,
i∈ I s ∈ I (2)
where
( )
B l = bisl , l ∈ L, i, s ∈ I ,−
the MCM corresponding to the rankings R , l ∈ L.
l
Sometimes the Euclidean metric is used to determine the resulting ranking.
Well known is the problem of collective expert assessment. Necessary to find the re-
sultant (collective, group, compromise, integral, aggregated, agreed, collapsed, synthe-
sized, generalized, global, etc.) ranking of objects that by some criterion is “closest” to
all expert rankings. The most reasonable method of finding the resultant ranking of
objects is to calculate the median of given rankings.
The most common method of finding the resultant ranking of objects is to calculate
the median of the given rankings.
85
We denote the set of all possible n object rankings by Ω , and the set of MCMs
R
corresponding to all possible n object rankings by Ω . We will denote the set of
B
B
rankings given by the experts as R , and the set of the corresponding MCMs as R .
A
A B
For the case considered in this paper, the capacity of the sets R and R is the same:
R A = R B = n Rl ∈ R A , Bl ∈ R B , l ∈ L R A ⊂ ΩR , R B ⊂ ΩB
, . It is clear that . In
Ω B = 2 n (n−1)/ 2
the general case, the set capacity of the . But for the method described
Ω R = Ω B = n!
in this paper, we will assume that since we are not interested in non-
B
transitive elements of the solution space Ω .
For Cook metric (object rank mismatch) (1), using the utilitarian criterion, the Cook-
Sayford median is calculated [1]:
R CS
∈Ω CS
= Arg minR d R, R .
R∈Ω
l
( )
l∈L (3)
Modified Cook-SayfordМCS
median:МCS
R ∈Ω = Arg minA d R, R l .
R∈R
( )
l∈L (4)
When using the egalitarian criterion, the GV-median (compromise) is calculated [1]:
R ГВ ∈ Ω ГВ = Arg minR max d R, R l .
R∈Ω l∈L
( )
(5)
Modified GV median:
R МГВ ∈ Ω МГВ = Arg minA max d R, R l .
R∈R l∈L
( )
(6)
For the Hamming’s metric (2), the Kemeni-Snell median is calculated using the
utilitarian criterion: КС
R КС
∈Ω = Arg minB d B, B .
B∈Ω
l
( )
l∈L (7)
Modified Kemeni-Snell medianМКС
[2]:
МКС
R ∈Ω = Arg minB d B , B l .
B∈R
( )
l∈L (8)
When using the egalitarian criterion, the HG-median (compromise) is calculated [3]:
R ВГ ∈ Ω ВГ = Arg minB max d B , B l .
B∈Ω l∈L
( )
(9)
Modified VG median:
R МВГ ∈ Ω МВГ = Arg minB max d B , B l . ( )
B∈R l∈L (10)
86
Into account can be taken expert competence ratios:
ρ1 ,..., ρ k .
Methods for determining the medians of species (3), (5), (7), (9) and their features
are considered in the monograph [3]. Modified medians (4), (6), (8), (10) are proposed
in some papers [2], but their use in many practical tasks is inappropriate and sometimes
unreasonable and unjustified.
The modified median significantly limits the choice space, so when applying these
criteria, we usually find ineffective solutions: they are dominated, in particular, by the
medians of (4), (6), (8), (10). In addition, the method described in this paper cannot be
applied to modified medians, but finding modified medians is inappropriate and unrea-
sonable in most cases.
The criteria functions used to determine the median (3) - (10) are related to the dis-
tances from the expert's given rankings to calculated solutions of the problem. There-
fore, the criteria minimum (5), (6), (9), (10) for the corresponding formulations of the
problem can be an efficiency feature of the problem obtained solution. Obviously, so-
lutions that do not meet the minimum of the specified criteria are ineffective and they
are dominated by solutions that deliver the minimum of the criteria (5), (6), (9), (10).
The metrics and criteria for determining the median of given object rankings can
have varying degrees of popularity among researchers, have unequal levels of trust, and
be reasonably different. But all of these tools are well-established ways of determining
the resultant ranking, and it makes no sense to ignore any of the well-known approaches
that have proven themselves over the decades.
Classical methods of choice theory (Condorcet, Borda, Simpson, Copland, Nanson,
alternative votes, relative majority, etc.) can also be limited in determining the resulting
ranking. Such use is described, for example, in a monograph [3]. Investigation of ap-
plying classical methods of choice theory to determine the weights should be investi-
gated with the further development of the method described in this paper.
On the basis of calculation and analysis of the medians above (3), (5), (7), (9) it is
proposed to calculate the coefficients of the relative competence of experts in the form
of fuzzy set membership function (MF). The substantiation of such approaches was
provided by the authors in [4-6].
4 The task of determining the experts competence based on the
analysis of the given individual expert rankings of objects
To date, a considerable number of methods for defining collective expert assessments
and calculating the relative competence of experts have been developed and substanti-
ated in the expert evaluation direction. They are all entitled to existence and can be used
to make decisions. To determine the experts competence by solving the specific prob-
lem of expert evaluation, the authors suggest to use the analysis of existing methods of
a posteriori calculation of coefficients of experts relative competence. The developed
apparatus for determining the group estimates of the form (3) - (8) will be used and
"collapsed" into one MF.
In [3] one of the most important problems of expert evaluation, which is to determine
the experts competence, is considered. The proposed methods of determining these co-
efficients are based on the axiom of immutability: "Conclusions of the majority are
87
competent" and the consequence that "the most competent is the expert whose conclu-
sions in most cases coincided with the conclusions of the majority of experts".
The feature of mismatched MCMs and object rankings set is the large number of
effective solutions in the problem of defining group object ranking. In particular, each
of the calculated medians (3) - (8) may not be unique.
It will be assumed that the experts competence is determined on the basis of the
assigned object rankings.
The number of solutions to the task of resulting ranking determining is influenced
by the following factors: 1
{
– set of used metrics S = d , d ,... ;
1 2
}
–criteria set for determining the closeness between the rankings S = δ , δ ,... ;
2
{ 1 2
}
– is not the only solution of the median determining problem of type (3) - (5);
– use to determine the resulting ranking of a set of classical choice rules: Condorcet,
Bord, Simpson, Copland, Nanson, alternative votes, relative majority, etc. [1], the set
of which is denoted by S .
3
Thus, the number of1 task solutions for collective ranking of objects determining will
s = si s j + s ,
2 3
be equal to i⇔ j where the symbol
" i ⇔ j" means "metric indexes
i ∈ S1, for which there are indexes of proximity criteria
j ∈ S 2 ",
s t = S t , t = 1,2,3, ⋅ −
and is the number of set elements.
As a result of applying all the methods of calculating the relative coefficients of
expert competence, we obtain a set of object rankings, which are the median of the
giveni expertК rankings:
CS ГВ КС ВГ
R ∈Ω = Ω ∪Ω ∪Ω 3
∪Ω ∪ Ω , i = 1,..., s.
According to [3], when a posteriori determination of the relative coefficients of
competence in the ranking tasks, the distance from expert rankings to the calculated
medians serves as a measure of competence. That is, the normalized weights of the
experts relative competence on the basis of the found s resultant rankings that make
К
up the set Ω , will be calculated s by the formula:
γ ij = (1 / d (R i , R j ))/ (1 / d (R t , R j )), i
R ∈ Ω К , i = 1,..., s, j ∈ I .
t =1 (11)
The idealized weight coefficients [7] of the relative competence of the criteria will be
calculated by the formulas:
γ ijід = γ ij / max γ it , i = 1,..., s, j ∈ I .
t∈I (12)
Since the variants of determining the resulting ranking can be tens, the coefficients of
competence can be calculated in the form of MF.
88
The algorithms for calculating MF in such cases are substantiated and described in
[3, 4, 6]. The need to determine the coefficients of experts relative competence in the
form of MF is due to various reasons. Although the use of different metrics and criteria
to determine the resulting ranking are justified to varying degrees, they are all
successfully applied in different application situations. Failure to use some of the tools
entails the risk of losing information about the structure of preferences given by experts;
in our case, linear ordering of objects. In order not to do so, all possible solutions
obtained by different approaches are combined into a single array for their further
analysis and conclusion.
This is the situation that Blaise Pascal wrote in [8]: «Justice and truth are two such thin
points that our instruments turn out to be too coarse to touch. But if they are touched, they
open the edge and lean on the environment, rather than a flaw than a truth».
5 Method for determining fuzzy set membership function by
analyzing frequency of values
To apply the method of determining the MF for frequency values must be determined
with the universal set. For the requirements of this research, the universal set for deter-
mining the MF is the values from the interval (0,1), namely, the results of determining
the corresponding normalized coefficients of experts relative competence on the basis
of calculating values (11) inverted to distances between rankings. For the idealized val-
ues of the weight coefficients in the form (12), the universal set is the idealized values
of the coefficients, which are also chosen from the interval (0,1).
It is necessary to analyze the results s of numerical estimates or the results of meas-
urement or estimation of some value and determine the MF for the measured or esti-
mated values. Typically, data is grouped at 10-20 intervals. However, the number of
values that fall into each interval does not exceed 15-20% of their total. This is suffi-
cient to identify all the properties of the magnitude and reliably calculate by group fre-
quency the basic characteristics of the fuzzy set. In cases where the number of intervals
exceeds 20, the MF may be polymodal and the information displayed will not be a clear
representation of the fuzzy set. Grouping data at too large intervals can lead to the loss
of the expert's view of the behavior of the MF, as well as to gross errors in its applica-
tion.
Obviously, by analyzing the frequency of values, the number of values of the esti-
mated values that belongs to each selected interval indicates the measure with which
this value belongs to each interval. In this case, only the points are classified according
to their values: whether or not the point belongs to the selected interval. The number of
points included in the interval indicates the degree of optimality or quasi-optimality of
this interval [6]. This justifies the use of frequency algorithms in determining the MF.
It is known [2] that for nominal features, that is, measured in the scale of names, as
mean used the mode. For data measured in the ordinal scale, the median is a valid mean.
When examining the information measured in the interval scale, only the arithmetic
mean can be used. And for the data analysis specified in the scale of relations, stepwise
averages and geometric averages are used.
In constructing the MF on the frequency of values, we actually use the results of
89
mode analysis of the estimated meanings of the studied value. In [4, 6], several algo-
rithms for the classification of the obtained solutions are constructed for constructing
the MF of the investigated value on the analysis basis of the measured magnitude values
frequency at some intervals chosen by the researcher.
6 Experiment results
To illustrate the method described in this paper, we give an example of solving a small
dimension problem for the number of objects n = 5 and the number of experts k = 6.
The algorithm will be described in two steps. In the first stage we will present the meth-
ods and results of obtaining the median of the given rankings. Particular attention will
be paid to comparing the modified medians that we propose to calculate in [2] with the
actual medians found on the set of all possible rankings. We formulate reservations
about the limited use of such a simplified aggregation tool set by expert rankings.
In the second stage of describing of the method for coefficients determining of
experts relative competence in the form of MF, the procedure of obtaining the resulting
values of normalized MF is described in detail, which characterizes the degree of
belonging of different values of weight coefficients to fuzzy sets of possible normalized
or idealized values of weight criteria.
A = {a1 , a2 , a3 , a4 , a5 }.
Consider a set of five objects Each expert determines with
indexes set I = {1,2,... 6}, the individual ranking of the objects
Ri , i ∈ I :
R1 = {a4 a1 a3 a2 a5 }
;
R 2 = {a3 a2 a4 a5 a1 }
;
R 3 = {a1 a4 a3 a5 a2 }
;
R 4 = {a2 a5 a1 a4 a3 }
;
R 5 = {a5 a1 a3 a4 a2 }
;
R 6 = {a2 a3 a4 a1 a5 }
.
R
The space of all possible strict five-object rankings Ω consists of 5! = 120
rankings. The set of valid solutions to determine modified medians consists of only six
expert rankings.
6.1 Calculation of medians set for given expert rankings
МКС МКС
We compute the modified medians for the Hamming metric R ∈Ω of the
МВГ МВГ
form (8) and R ∈ Ω in the form (10).
Modified Kemeny-Snell medians on a given set of expert rankings revealed three,
i.e. 50% of the given:
R 2 = {a3 a2 a4 a5 a1 }
;
R 3 = {a1 a4 a3 a5 a2 }
;
R 5 = {a5 a1 a3 a4 a2 }
.
90
The total distance by Hamming's metric from each of the calculated medians to the
6
six given medians is minimal and equal 54:
R МКС ∈ Ω МКС = Arg minB d B , B l = 54 .
B∈R
( )
l =1
The modified VG median for this example is the only one
R 2 = {a3 a2 a4 a5 a1 }
R МВГ ∈ Ω МВГ = Arg minB max d B , B l = 14 .
B∈R l ={1, 2 ,..., 6}
( )
and the maximum distance from it to all given is 14:
Then compute the modified medians for the Cook metric of mismatch
R МCS ∈ Ω МCS of type (4) and R МГВ ∈ Ω МГВ of type (6).
Modified Cook-Sayford medians on a given set of expert rankings also revealed
three, that is, 50% of the given ones, and for our example they coincide with the
modified median of Kemeny-Snell:
R 2 = {a3 a2 a4 a5 a1 }
;
R 3 = {a1 a4 a3 a5 a2 }
;
R 5 = {a5 a1 a3 a4 a2 }
.
The total distance by Cook's metric of the rank mismatch from each of the calculated
medians to the six given is minimal and equal 42:
( )
6
R МCS ∈ Ω МCS = Arg minA d R, R l = 42.
R∈R l =1
The modified GV medians (two) for our example are calculated
R 4 = {a2 a5 a1 a4 a3 } R 5 = {a5 a1 a3 a4 a2 }
and , that is, 33% of
the Rgiven,
МГВ andМГВ
∈ Ω the =maximum
Arg min distance
max dfrom
R , Rthem
l
(
to .all given is 10:
= 10
B∈R A l ={1, 2 ,..., 6 }
)
Let's solve the problems of determining the median on the space of all possible five
object rankings for the given six rankings.
МCS
Then compute the medians for the Hamming metric R ∈ Ω МCS in the form (7)
МГВ МГВ
and R ∈Ω in the form (9).
The Kemeny-Snell medians for a given set of expert rankings in the space of all
possible rankings of five objects is calculated six, that is 5% of the possible 120
rankings of five objects:
R КС (1) = {a1 a 2 a3 a5 a 4 }
;
R КС ( 2 ) = {a1 a 2 a 4 a5 a3 }
;
R КС ( 3) = {a1 a3 a 4 a5 a 2 }
;
R КС ( 4 ) = {a 2 a1 a3 a5 a 4 }
;
R КС ( 5 ) = {a 2 a1 a 4 a5 a3 }
;
R КС ( 6 ) = {a3 a1 a 2 a5 a 4 }
.
The total distance by Hamming's metric from each of the calculated medians to the
six given is minimal and is 50:
91
R КС
∈Ω КС
B∈Ω
(
= Arg minB d B, B = 50. l
)
l∈L
That is, the Kemeny-Snell medians have a better value of the additive criterion in
form (7) than the value of the additive criterion of type (8), and thus the modified
Kemeny-Snell medians dominate. In other words, modified medians in form (8) are
ineffective. And since these medians are not Pareto optimal, you can use them with
considerable caution to solve group object ranking tasks.
The compromise median, that is, the VG median, for our example, is calculated by
R ВГ (1) = {a1 a2 a3 a5 a4 } R ВГ ( 2 ) = {a 2 a1 a3 a5 a 4 }
two: and .
TheRmaximum
ВГ
∈ Ω ВГ distance
= Arg minfrom these
max d B
B∈Ω B
medians
l∈L
(
to .all given is 10:
, B l = 10 )
Thus, the compromise medians on the Hamming metric in form (9) are also
dominated by the modified VG-medians in form (10). This means that modified
medians and for the minimax criterion can only be used with significant caveats, used
as a reference, or to manipulate choices and to demonstrate sci-fi.
CS CS
We compute the medians for the Cook metric of ranks mismatch R ∈ Ω in form
ГВ ГВ
(3) and R ∈ Ω in form (5) in the space of all possible medians of five objects.
The Cook-Sayford medians of a given set of six expert rankings is five, that is, 4%
of all CS
possible strict rankings generated by five objects:
R (1) = {a 2 a1 a3 a 4 a5 }
;
R CS ( 2 ) = {a 2 a1 a3 a5 a 4 }
;
R CS ( 3) = {a 2 a1 a 4 a5 a3 }
;
R CS ( 4 ) = {a a1 a4 a5 a2 }
;
R CS ( 5 )
= {a 4 a1 a3 a5 a 2 }
.
The total distance by Cook's metric of the rank mismatch from each of the calculated
R CS ∈toΩthe
medians
CS
=sixArg R
minis
given minimum
R∈Ω
(
d R , R l and
= 40
is. 40: )
l∈L
That is, each of the calculated five rankings is the median of Cook-Sayford given six
expert rankings. Obviously, they are dominated by the above Cook-Sayford modified
medians, the value of criterion (3) of which is 42.
As a result of calculating the GV medians, six examples were identified in our
example, that is, 5% of all possible rankings of five objects:
R ГВ (1) = {a1 a 2 a3 a 4 a5 }
;
R ГВ ( 2 ) = {a 2 a1 a3 a 4 a5 }
;
R ГВ ( 3) = {a1 a 2 a3 a5 a 4 }
;
R ГВ ( 4 )
= {a 2 a1 a3 a5 a 4 }
;
R ГВ ( 5 )
= {a 2 a1 a 4 a5 a3 }
;
R ГВ ( 6 )
= {a 2 a1 a5 a 4 a3 }
.
92
The maximum distance from the calculated GV medians to all six rankings set by
experts
R ГВ is∈8,Ωthat
ГВ is:
= Arg min max d R , R l = 8.
R∈Ω R l∈L
( )
Thus, the GV medians are dominated by the above modified GV medians, for which
the value of the minimax criterion is 10. This means that the modified medians can only
be used in exceptional cases and it is better to refrain from this formulation.
In addition, choosing a solution among the rankings given by experts, we
significantly narrow the space of choice and the dictator chosen in this way does not
satisfy many of Arrow's axioms [1], and, above all, is not effective.
6.2 Determination of the experts’ relative competence
Based on the medians calculated above for different metrics and different criteria, let
us determine the competence coefficients of the experts who set the ranking of the
objects based on the axiom (heuristic) of non-bias: the conclusions of most experts are
competent. Against this background, we assume that expert competence is inversely
proportional to the median defined in the space of all possible object rankings.
Since there are several ways to determine the median of given rankings in the
algebraic approach, and each solution is often not unique, a whole family of relative
competence factors of experts is generated that needs to be adequately considered,
analyzed and reasoned appropriately.
To determine the coefficients of experts relative competence on the results of
calculating the medians that satisfy the criteria (3), (5), (7), (9), we summarize all the
obtained values in a single Table 1.
Table 1. Results of the median calculation that satisfy the criteria (3), (5), (7), (9)
Frequency
Distances from expert rankings to medians
Ranking objects
1 2 3 4 5 6
{a1 a2 a3 a4 a5 }
1 8 10 8 10 12 6
{a 2 a1 a3 a 4 a5 }
2 6 8 10 12 10 8
{a1 a 2 a3 a5 a 4 }
3 10 8 6 8 10 8
{a1 a 2 a 4 a5 a 3 }
1 12 6 4 10 8 10
{a1 a3 a 4 a5 a 2 }
1 14 4 2 8 10 12
{a 2 a1 a3 a5 a 4 }
4 8 6 8 10 8 10
{a 2 a1 a 4 a5 a 3 }
3 10 4 6 12 6 12
{a 3 a1 a 2 a5 a 4 }
1 6 8 10 8 6 12
{a 3 a1 a 4 a5 a 2 }
1 12 2 8 14 4 14
{a 4 a1 a3 a5 a 2 }
1 10 4 10 12 2 16
{a 2 a1 a5 a 4 a3 }
1 8 6 8 14 8 10
93
The "frequency" row indicates the number of times that the appropriate ranking became
the median by some criterion on a certain metric, or simultaneously by several criteria
and metrics.
You can see that among the medians calculated by formulas (3), (5), (7), (9), there
is Дone that is present in each of the subsets of solutions. This ranking
R = {a 2 a1 a3 a5 a 4 }
, which is a median that delivers minimum
simultaneously to all the criteria for all metrics and, accordingly, can be substantiated
more comprehensively and substantiated than any of those that also fall into one or
more subsets of equivalents by given solution criteria group ranking problems. We call
this solution "statistically significant" or "perfect" median.
Based on the data in formulas (11) in the previous table, we determine the
normalized values of the weights of the experts relative competence, depending on
which median is the solution of the task of determining group ranking. We present all
the calculated coefficients in the form of Table 2.
Table 2. The normalized values of the experts relative competence weights, depending on
which median is the promlem solution of determining group ranking
Normalized competency weights
Frequency
Ranking objects
1 2 3 4 5 6
{a1 a2 a3 a4 a5 }
1 0,179 0,143 0,179 0,143 0,119 0,238
{a 2 a1 a3 a 4 a5 }
2 0,238 0,179 0,143 0,119 0,143 0,179
{a1 a 2 a3 a5 a 4 }
3 0,135 0,169 0,225 0,169 0,135 0,169
{a1 a 2 a 4 a5 a 3 }
1 0,101 0,202 0,303 0,121 0,152 0,121
{a1 a3 a 4 a5 a 2 }
1 0,063 0,221 0,443 0,111 0,089 0,074
{a 2 a1 a3 a5 a 4 }
4 0,169 0,225 0,169 0,135 0,169 0,135
{a 2 a1 a 4 a5 a 3 }
3 0,118 0,294 0,196 0,098 0,196 0,098
{a 3 a1 a 2 a5 a 4 }
1 0,217 0,163 0,130 0,163 0,217 0,109
{a 3 a1 a 4 a5 a 2 }
1 0,076 0,454 0,114 0,065 0,227 0,065
{a 4 a1 a3 a5 a 2 }
1 0,091 0,228 0,091 0,076 0,456 0,057
{a 2 a1 a5 a 4 a3 }
1 0,175 0,234 0,175 0,100 0,175 0,140
We define the idealized values of the weight coefficients of the experts relative
competence [7, 9,] and present them in the form of Table 3. Unlike the normalized
values, for which the sum of the coefficients is equal to one, among the idealized values
of the coefficients, the greatest is equal to one, and others are determined
proportionally.
We construct a Table 4 with normalized values of the weights of the experts relative
competence, taking into account the frequency of values, and carry out the ordering of
the thus obtained normalized weights. Table 4 will have 19 rows, each row corresponds
94
to the expert competence coefficients that were determined based on the median that
was determined to be effective in solving one of the problems of (3), (5), (7), (9).
Table 3. Idealized values of experts' relative competence weights
Frequency
Idealized weighted competency coefficients
Ranking objects
1 2 3 4 5 6
{a1 a2 a3 a4 a5 }
1 0,75 0,60 0,75 0,60 0,50 1,00
{a 2 a1 a3 a 4 a5 } 2 1,00 0,75 0,60 0,50 0,60 0,75
{a1 a 2 a3 a5 a 4 }
3 0,60 0,75 1,00 0,75 0,60 0,75
{a1 a 2 a 4 a5 a 3 } 1 0,33 0,67 1,00 0,40 0,50 0,40
{a1 a3 a 4 a5 a 2 } 1 0,14 0,50 1,00 0,25 0,20 0,17
{a 2 a1 a3 a5 a 4 }
4 0,75 1,00 0,75 0,60 0,75 0,60
{a 2 a1 a 4 a5 a 3 } 3 0,40 1,00 0,67 0,33 0,67 0,33
{a 3 a1 a 2 a5 a 4 } 1 1,00 0,75 0,60 0,75 1,00 0,50
{a 3 a1 a 4 a5 a 2 }
1 0,17 1,00 0,25 0,14 0,50 0,14
{a 4 a1 a3 a5 a 2 } 1 0,20 0,50 0,20 0,17 1,00 0,13
{a 2 a1 a5 a 4 a3 } 1 0,75 1,00 0,75 0,43 0,75 0,60
Table 4. The normalized values of the weights of the experts relative competence, taking into
account the frequency of values
Normalized competency weights
Item
number 1 2 3 4 5 6
1 0,063 0,143 0,091 0,065 0,089 0,057
2 0,076 0,163 0,114 0,076 0,119 0,065
3 0,091 0,169 0,130 0,098 0,135 0,074
4 0,101 0,169 0,143 0,098 0,135 0,098
5 0,118 0,169 0,143 0,098 0,135 0,098
6 0,118 0,179 0,169 0,100 0,143 0,098
7 0,118 0,179 0,169 0,111 0,143 0,109
8 0,135 0,202 0,169 0,119 0,152 0,121
9 0,135 0,221 0,169 0,119 0,169 0,135
10 0,135 0,225 0,175 0,121 0,169 0,135
11 0,169 0,225 0,179 0,135 0,169 0,135
12 0,169 0,225 0,196 0,135 0,169 0,135
13 0,169 0,225 0,196 0,135 0,175 0,140
14 0,169 0,228 0,196 0,135 0,196 0,169
15 0,175 0,234 0,225 0,143 0,196 0,169
16 0,179 0,294 0,225 0,163 0,196 0,169
17 0,217 0,294 0,225 0,169 0,217 0,179
18 0,238 0,294 0,303 0,169 0,227 0,179
19 0,238 0,454 0,443 0,169 0,456 0,238
95
To determine the membership functions of the relative normalized weighting
coefficients of the experts' relative competence, we shall classify the values obtained
by formula (11) according to their belonging to the selected intervals. All values of the
γ ∈ [0,057;0,456],
normalized coefficients are in the range from 0.057 to 0.456: ij
i = 1,...,19, j = 1,...,6.
To apply the method of determining the MF coefficients values of their frequency,
we introduce two heuristics.
γ ,
Heuristics E1. The interval in which all the normalized values of the weights ij
i = 1,...,19, j = 1,...,6,
are found is broken down into intervals with a width of 0.05.
Heuristics E2. The fixed value represented by each interval will be the rounded
upper bound of each such interval.
γ , i = 1,...,19, j = 1,...,6,
As a result of the classification of points ij according to the
eight intervals selected with regard to heuristics E1 and E2, we obtain Table 5.
Table 5. Frequency of normalized values of the experts relative competence weights in selected
intervals
The value The number of values of normalized weights of expert competence that fall into the
representing selected interval
the selected
interval 1 2 3 4 5 6
0,1 3 1 1 1 6
0,15 7 1 4 10 6 7
0,2 6 6 9 4 9 5
0,25 3 8 3 2 1
0,3 3
0,35 1
0,4
0,45 1 1 1
As a result of Table 5 in terms of MF, we obtain the following values of singletons of
frequency for normalized values of weighting coefficients:
μ1 (0,1) / 3 + μ1 (0,15) / 7 + μ1 (0,2) / 6 + μ1 (0,25) / 3;
μ 2 (0,15) / 1 + μ 2 (0,2) / 6 + μ 2 (0,25) / 8 + μ 2 (0,3) / 3 + μ 2 (0,45) / 1;
μ 3 ( 0 ,1) / 1 + μ 3 ( 0 ,15 ) / 4 + μ 3 ( 0 , 2 ) / 9 + μ 3 ( 0 , 25 ) / 3 + μ 3 ( 0 ,35 ) / 1 + μ 3 ( 0 , 45 ) / 1;
μ 4 (0,1) / 5 + μ 4 (0,15) / 10 + μ 4 (0,2) / 4;
μ 5 (0,1) / 1 + μ 5 (0,15) / 6 + μ 5 (0,2) / 9 + μ 5 (0,25) / 2 + μ 5 (0,45) / 1;
μ 6 (0,1) / 6 + μ 6 (0,15) / 7 + μ 6 (0,2) / 5 + μ 6 (0,25) / 1.
Let us normalize the membership functions and present the fuzzy values of the relative
coefficients of experts' competence in the form of normal membership functions.
μ1 (0,1) / 0,43 + μ1 (0,15) / 1 + μ1 (0,2) / 0,86 + μ1 (0,25) / 0,43;
μ 2 (0,15) / 0,13 + μ 2 (0,2) / 0,75 + μ 2 (0,25) / 1 + μ 2 (0,3) / 0,38 + μ 2 (0,45) / 0,13;
μ3 (0,1) / 0,11+ μ3 (0,15) / 0,44+ μ3 (0,2) /1+ μ3 (0,25) / 0,33+ μ3 (0,35) / 0,11+ μ3 (0,45) / 0,11;
μ 4 (0,1) / 0,5 + μ 4 (0,15) / 1 + μ 4 (0,2) / 0,4;
μ 5 (0,1) / 0,11 + μ 5 (0,15) / 0,67 + μ 5 (0,2) / 1 + μ 5 (0,25) / 0,22 + μ 5 (0,45) / 0,11;
μ 6 (0,1) / 0,86 + μ 6 (0,15) / 1 + μ 6 (0,2) / 0,71 + μ 6 (0,25) / 0,14.
96
The idealized [7, 9] values of the experts relative competence, which correspond to
the table of normalized coefficient values and calculated by the formula (12), can be
represented in the form of Table 6.
Table 6. Idealized values of experts' relative competence weights taking into account frequency
of values
Idealized weighted competency coefficients
Item
number
1 2 3 4 5 6
1 0,143 0,500 0,200 0,143 0,200 0,125
2 0,167 0,500 0,250 0,167 0,500 0,143
3 0,200 0,600 0,600 0,250 0,500 0,167
4 0,333 0,667 0,600 0,333 0,500 0,333
5 0,400 0,750 0,600 0,333 0,600 0,333
6 0,400 0,750 0,667 0,333 0,600 0,333
7 0,400 0,750 0,667 0,400 0,600 0,400
8 0,600 0,750 0,667 0,429 0,600 0,500
9 0,600 0,750 0,750 0,500 0,600 0,600
10 0,600 0,750 0,750 0,500 0,667 0,600
11 0,750 1,000 0,750 0,600 0,667 0,600
12 0,750 1,000 0,750 0,600 0,667 0,600
13 0,750 1,000 0,750 0,600 0,750 0,600
14 0,750 1,000 0,750 0,600 0,750 0,750
15 0,750 1,000 1,000 0,600 0,750 0,750
16 0,750 1,000 1,000 0,750 0,750 0,750
17 1,000 1,000 1,000 0,750 0,750 0,750
18 1,000 1,000 1,000 0,750 1,000 0,750
19 1,000 1,000 1,000 0,750 1,000 1,000
To determine the membership functions of the relative idealized weighting coefficients
of the experts relative competence, classification of the values obtained by formula (12)
according to their belonging to the selected intervals should be carried out. All values
γ ід ∈ [0,125;1],
of the idealized coefficients are in the range of 0.125 to 1: ij
i = 1,...,19, j = 1,...,6.
To apply to the analysis of idealized values of MF values coefficients determining
method by their frequency, we introduce two more heuristics.
Heuristics E3. The interval [0;1], in which all the idealized values of the weights
γ ijід , i = 1,...,19, j = 1,...,6,
are found is divided into five intervals of width 0.2.
Heuristics E4. A fixed value representing each selected interval will be considered
the upper bound of each such interval.
Let us classify the obtained 19 points, which correspond to the idealized values ідof
γ ,
the coefficients, at five intervals. As a result of the classification of points ij
i = 1,...,19, j = 1,...,6,
according to the five intervals selected with regard to heuristics
E3 and E4, we obtain Table 7.
97
Table 7. Frequency of the weights idealized values of experts relative competence in selected
intervals
The value The number of values of the idealized weights of expert competence that fall into the
representing selected interval
the selected
interval 1 2 3 4 5 6
0,2 1 2 3 4 5 6
0,4 4 2 6 1 3
0,6 3 2 4
0,8 3 4 6 7 11 6
1 6 6 6 4 5 5
Based on construction method the MF to fuzzy set by analyzing the frequency of values,
it is possible to present fuzzy normalized weights of experts' competence in the form of
singletons of weighting coefficients values frequency:
μ1 (0,2) / 3 + μ1 (0,4) / 4 + μ1 (0,6) / 3 + μ1 (0,8) / 6 + μ1 (1) / 3;
μ 2 (0,4) / 2 + μ 2 (0,6) / 2 + μ 2 (0,8) / 6 + μ 2 (1) / 9;
μ 3 (0,4) / 2 + μ 3 (0,6) / 3 + μ 3 (0,8) / 9 + μ 3 (1) / 5;
μ 4 (0,2) / 2 + μ 4 (0,4) / 5 + μ 4 (0,6) / 8 + μ 4 (0,8) / 4;
μ 5 (0,4) / 1 + μ 5 (0,6) / 8 + μ 5 (0,8) / 8 + μ 5 (1) / 2;
μ 6 (0,2) / 3 + μ 6 (0,4) / 4 + μ 6 (0,6) / 6 + μ 6 (0,8) / 5 + μ 6 (1) / 1.
As a result of membership functions normalization, we finally obtain singletons to
express the fuzzy values of the experts relative competence weights for our example.
μ1 (0,2) / 0,5 + μ1 (0,4) / 0,67 + μ1 (0,6) / 0,5 + μ1 (0,8) / 1 + μ1 (1) / 0,5;
μ 2 (0,4) / 0,2 − 2 + μ 2 (0,6) / 0,22 + μ 2 (0,8) / 0,67 + μ 2 (1) / 1;
μ 3 (0,4) / 0,22 + μ 3 (0,6) / 0,33 + μ 3 (0,8) / 1 + μ 3 (1) / 0,56;
μ 4 (0,2) / 0,25 + μ 4 (0,4) / 0,63 + μ 4 (0,6) / 1 + μ 4 (0,8) / 0,5;
μ 5 (0,4) / 0,13 + μ 5 (0,6) / 1 + μ 5 (0,8) / 1 + μ 5 (1) / 0,25;
μ 6 (0,2) / 0,5 + μ 6 (0,4) / 0,67 + μ 6 (0,6) / 1 + μ 6 (0,8) / 0,83 + μ 6 (1) / 0,17.
All constructed normal MFs are triangular, with a tendency to bell-shaped, except
for the fifth, which is trapezoidal.
As a result of the method developed by the authors and described in this paper, the
normalized and idealized values of the coefficients of experts relative competence in
the problems of objects collective ranking are calculated. These values are fuzzy and
are represented by the MF, which reflects the degree of correlation of the weights se-
lected values on the basis of the heuristics entered, to a fuzzy set of normalized or ide-
alized values.
7 Prospects for further research
It is obvious that, depending on the content of the problem being solved, experts can be
interpreted as methods that generate the ranking of measured or evaluated objects, or
other sources of information that are used to rank objects of a non-numerical nature.
98
Accordingly, not only competence but also weight, reliability, adequacy, convenience,
robustness, speed of calculation, convergence and other parameters obtained from
different sources of information can be investigated.
Methods for determining the coefficients of the experts relative competence in rank-
ing problems can be interpreted as coefficients of the relative importance of sources of
information [10]. In the following, we can consider the formulation of the problem and
the approaches to its solution in view of its stated interpretation.
The approach described in this paper can be improved in the following areas:
– formulation of determining problem statement for the experts competence who
specify non-strict rankings [3, 11] (perfect quasi-orders [12], ordering [13], quasi-series
[14], ranking with links [15], quasi-orders [16], clustered rankings [2]);
– study the impact of the opinion consistency level of the expert group on the number
of effective solutions of problems in different metrics and for different criteria;
– identification of experts coalitions on the basis of expert rankings clustering with
a considerable number of expert group members;
– study of resistance to manipulation with different metrics and different criteria for
determining the median of given rankings;
– calculating the impact of coalitions' relative weight on the resulting ranking and
sensitivity of metrics and criteria to changes the weight of expert coalitions;
– taking into account expertly specified coefficients that reflect the influence of the
methods of determining the median on final problem solution: the methods reliability,
increasing confidence in them from the research or taking into account the experimental
data;
– study of the conclusions bias of corrupt experts and determine the likelihood of a
corrupt component in solving the problem of expert evaluation will be a development
of the studies described in [17];
– study of the relationship between the intervals of normalized and idealized
coefficients of relative competence of experts;
– the use of a heuristic method for determining the median of expert rankings or the
genetic algorithm for finding medians in the presence of a large number of objects.
8 Conclusions
The classification of methods for determining the experts competence is proposed. The
prospect of applying a posteriori approach to calculating the coefficients of experts rel-
ative competence is substantiated. Based on the developed application for this ap-
proach, and the analysis of them using the method of determining the MF based on the
frequency of values, an approach is proposed for the construction of weight coefficients
of experts relative competence in the form of the MF.
An example illustrating the method proposed by the authors is discussed. Based on
the example, the insufficient validity of the use of modified medians in ranking prob-
lems is illustrated. The concept of statistically significant or perfect median of expert
rankings was introduced. The coefficients of the experts relative competence in the
form of the membership function to a fuzzy set are also calculated.
99
The prospects of investigating the problems of determining the coefficients of the
experts relative competence or the weighting coefficients of the relative importance of
sources of information are outlined.
References
1. Voloshyn, O. F., Mashchenko, S. O.: Decision theory: tutorial. Kiev: Publishing and Print-
ing Center "Kyivskyi universytet", 366 p. (2010).
2. Orlov,A. I. Management decision-making methods: a textbook. M: KNORUS 286p. (2018).
3. Hnatiienko, H. M., Snytyuk, V. E.: Expert Technologies for Decision Making. – K.: Com-
pany ”Maklaut”. – 444 p. (2008).
4. Hnatiienko, H. M. Algorithms for determining the function of belonging by analyzing the
time-accuracy of values // Proceedings of the III International School-Seminar "Decision-
making Theory", Uzhgorod, UzhNu.-P.32-34. (2006).
5. Snytyuk, V. E., Hnatiienko H. M. Optimization of the estimation process under uncertainty
conditions based on the structure of the domain and the axiom of unbiased // Artificial In-
tellect. – No.3. – Pp. 217–222. (2008).
6. Hnatiienko H. M. Determination of the weight coefficients of the criteria of the multicriteria
optimization problem in the form of functions of fuzzy set membership // Proceedings of the
5th International Scientific and Practical Conference "Information Technologies and Inter-
actions" (IT&I - 2018). - K. VOC "Kyiv University". Pp.29-30. (2018).
7. Saaty T. L. The relative dimension and its generalization in decision making. Why pairwise
comparisons are key in mathematics for measuring intangible factors / T.L. Saaty // Cloud
of Science. - Vol. 3, No. 2. - Pp. 171–262. (2016).
8. Pascal Blaise. Thoughts. / Trans. with fr. - K .: SPIRIT AND LETTER. - 704 p. (2009).
9. Saaty T. L. About measuring the intangible. An approach to relative measurements on the
basis of the principal eigenvector of a matrix of pairwise comparisons / T. L. Saaty // Cloud
of Science. - Vol. 2, No. 1. - Pp. 5–39. (2015).
10. Dodonov, A., Lande, D., Tsyganok, V., Andriichuk, O., Kadenko, S., Graivoronskaya, A.:
Information Operations Recognition. From Nonlinear Analysis to Decision-Making. Lam-
bert Academic Publishing, (2019).
11. Makarov, I. M., Vinogradskaya, T. M., Rubchinsky, A. A., Sokolov, V. B. The theory of
choice and decision-making. Moscow: Nauka. – 328 p. (1982).
12. Schrader, Yu.A. Equality, similarity, order. - M.: Science. - 256 p. (1971).
13. Kemeny, J. G., Snell, J. L.: Cybernetic modeling: Some applications. M.: Soviet Radio. -
192p. (1972).
14. Mirkin, B. G. The problem of group choice.. M.: Nauka. - 256 p. (1973).
15. Hollander, M., Wolf, D. Nonparametric methods of statistics. M.: Finance and Statistics. -
518 p. (1983).
16. Voloshin, O. F, Hnatiienko, H. M, Kudin, V. I. Consistent analysis of options. Technology
and application. – K.: Stylos, 304 p. (2013).
17. Gnatiyenko, G. M, Ledomska, L. S. Some aspects of the phenomenon of corruption and its
impact on the economy of Ukraine // Economic Journal. No.9, Pp.29-34. (1997).