The problem of determining the relative competence of experts in expert assessment tasks is considered. The classification of competence determination methods is offered and the use of objective approaches to a posteriori determination of these coefficients is substantiated. The problem of determining the objects resultant ranking and ways of getting its solutions are given. Based on the analysis of obtained solutions in the group ranking problem, we propose a method of relative competence coefficients determining for experts in the form of fuzzy set membership functions. An example that demonstrates the effect and features of the described method application is shown.
Introduction Determining expert competence ratios is an important element in solving expert assessment problems. The accuracy of determining the relative competence of experts affects the outcome of the task, the reliability of the solution, and the credibility of result. Taking into account competence of experts is the key to the quality of decision making and has a significant impact on the outcome of solution in the expert assessment problems. Therefore, determining of experts competence and their reasoned consideration is an actual area of research and can greatly contribute to improving the adequacy of the analysis of expert evaluation results and their validity.
Classification of methods for determining of experts competence Determining the experts competence is an important element of expert assessment. This factor significantly influences to decision making results and requires in-depth study, research and development. Today, there are several approaches to determining the relative competence of experts, but each of directions is not perfect: – documentary is often distrustful to ways of formalizing and taking into account objective data on experts, and is also determined by the current tendency to devalue official documents and the uncriticality of some institutional decisions; – self-assessment, during which often yields information about the level of self-confidence of the expert, rather than his real competence;
– mutual evaluation, by which in some problems it is possible to detect confrontation in the expert committee and the existence of coalitions among its members, which sometimes distort the true individual competence of experts, which really has to influence the results;
– sometimes combined methods, instead of synergistic effects, lead to a cumulative error;
– objective approaches, which are easy to apply and reasonably justified, are the most reliable to calculate adequate indicators of the relative competence of experts.
Objective approaches distinguish between a priori and a posteriori methods of calculating competence. Let us dwell on the methods of a posteriori determination of the experts competence developed by the authors. Objective approaches to the posteriori determination of expert competence ratios include: – study of the expert's participation effectiveness in previous examinations; – calculation of competence based on the results of control examination; – study of the expert's participation results in a specific examination.
The latter approach, which is to investigate the participation of an expert group members in a particular examination, may in turn be regarded as: – indirect analysis of indicators without first aggregating results; – the ratio of initial individual expert data and calculated integral indicators.
In the problems of determining the group ranking of objects as components of an approach in which the relationship of individual expert data and calculated integral indicators are investigated are:
– calculation of distances to pair comparison matrix (MCM) based on the found resultant ranking (in different metrics and by different criteria);
– a posteriori determination based on the objects resultant ranking by given individual expert rankings:
= analysis of the inversions number in the expert ranking relative to the resulting ranking; = determination of the number of objects consistently ranked by the expert; = calculating the sum of difference modules between the objects rank in the resulting ranking and the expert's given rankings;
– determination of individual relative estimates for collectively specified relative estimates: = calculating the intervals sum of relative estimates; = determining the maximum interval among relative estimates; = calculating the "volume" of the intervals of the determined relative estimates; = calculation of the "area" of the intervals of relative estimates; = determining the deviation level of the MCM from a given "ideal" vector of relative estimates;
– distance to the group vector of weighting coefficients when given by the experts of MCM in cardinal scales;
– the relative distance from expert rankings to the resultant rankings.
The modeling and study of the last two approaches are discussed in this paper. Such approaches are characterized by a situation of uncertainty. It is generated by the presence of a large number of methods developed and substantiated by different authors for a posteriori calculation of the coefficients of relative competence of experts. At the same time, it is not possible to determine for certain practical situations precisely and precisely which of the methods developed is the most appropriate. Therefore, the authors propose to apply in the previous stages of the analysis all known methods of determining the relative competence of experts and only in the last stages of the analysis to aggregate the information obtained by different methods. 3
Formulation of the task of determining the resultant ranking by group evaluation The methods of processing expert information are divided into three main groups [1]: – statistical methods; – scaling methods; – algebraic methods.
The essence of algebraic methods is that the set of admissible estimates sets the distance and the resulting score is defined as the distance to which the expert estimates for the selected criterion are minimal. On the basis of algebraic approach we will determine the coefficients of relative competence of experts.
Let k experts with index l ∈ L = {1,..., k }, k > 1, set A their preferences for the set of n objects in the form of object rankings Rl , l ∈ L.
To determine the distance between object rankings, researchers use:
– Cook metric for mismatcdhi(nRgj ,oRflo)b=ject rrai njk−inrigls, in individual rankings
i∈I
r l
where i – is the rank of the i-th object in the ranking of the l-th expert
Rl ,l ∈ L, 1 ≤ ril ≤ n,
– Hamming's metric
d (B j , Bl ) = 0,5 bisj − bils ,
i∈I s∈I
(
Bl = (bils ), l ∈ L, i, s ∈ I ,− the MCM corresponding to the rankings Rl , l ∈ L. Sometimes the Euclidean metric is used to determine the resulting ranking.
Well known is the problem of collective expert assessment. Necessary to find the resultant (collective, group, compromise, integral, aggregated, agreed, collapsed, synthesized, generalized, global, etc.) ranking of objects that by some criterion is “closest” to all expert rankings. The most reasonable method of finding the resultant ranking of objects is to calculate the median of given rankings.
The most common method of finding the resultant ranking of objects is to calculate the median of the given rankings. in this paper, we will assume transitive elements of the solution space ΩB .
ΩR = ΩB = n! Sayford median is calRcuClSat∈ed Ω[1C]S: = Arg min d (R, R l ). the general case, the set capacity of the . But for the method described that since we are not interested in non,
We denote the set of all possible n object rankings by Ω R , and the set of MCMs corresponding to all possible n object rankings by Ω B . We will denote the set of
A B rankings given by the experts as R , and the set of the corresponding MCMs as R . A B R AFo=r tRheBcase consli∈deRredA,inBthl i∈spRapBer, the capacity of the setsR RA⊂aΩndR ,RR Bis ⊂theΩsBame: = n R , l ∈ L Ω B = 2n(n−1)/ 2
. It is clear that R∈ΩR l∈L
R∈RA l∈L Modified Cook-SayforRdМmCSed∈iaΩn:МCS = Arg min d (R, Rl ).
R МГВ ∈ Ω МГВ = Arg min max d (R, Rl ).
R∈RA l∈L
For Cook metric (object rank mismatch) (
R∈ΩR l∈L
For the Hamming’s metric (
R КС ∈ ΩКС = Arg min d (B, Bl ).
B∈ΩB l∈L Modified Kemeni-SRneМllКСme∈diΩanМ[К2С]: = Arg min d (B, B l ).
B∈RB l∈L
R МВГ ∈ Ω МВГ = Arg min max d (B, B l ).
B∈RB l∈L When using the egalitarian criterion, the HG-median (compromise) is calculated [3]: R ВГ ∈ Ω ВГ = Arg min max d (B, Bl ).
B∈ΩB l∈L Into account can be taken expert competence ratios: ρ1,..., ρ k .
Methods for determining the medians of species (
The modified median significantly limits the choice space, so when applying these
criteria, we usually find ineffective solutions: they are dominated, in particular, by the
medians of (
The criteria functions used to determine the median (
The metrics and criteria for determining the median of given object rankings can have varying degrees of popularity among researchers, have unequal levels of trust, and be reasonably different. But all of these tools are well-established ways of determining the resultant ranking, and it makes no sense to ignore any of the well-known approaches that have proven themselves over the decades.
Classical methods of choice theory (Condorcet, Borda, Simpson, Copland, Nanson, alternative votes, relative majority, etc.) can also be limited in determining the resulting ranking. Such use is described, for example, in a monograph [3]. Investigation of applying classical methods of choice theory to determine the weights should be investigated with the further development of the method described in this paper.
On the basis of calculation and analysis of the medians above (
The task of determining the experts competence based on the
analysis of the given individual expert rankings of objects
To date, a considerable number of methods for defining collective expert assessments
and calculating the relative competence of experts have been developed and
substantiated in the expert evaluation direction. They are all entitled to existence and can be used
to make decisions. To determine the experts competence by solving the specific
problem of expert evaluation, the authors suggest to use the analysis of existing methods of
a posteriori calculation of coefficients of experts relative competence. The developed
apparatus for determining the group estimates of the form (
In [3] one of the most important problems of expert evaluation, which is to determine the experts competence, is considered. The proposed methods of determining these coefficients are based on the axiom of immutability: "Conclusions of the majority are competent" and the consequence that "the most competent is the expert whose conclusions in most cases coincided with the conclusions of the majority of experts".
The feature of mismatched MCMs and object rankings set is the large number of
effective solutions in the problem of defining group object ranking. In particular, each
of the calculated medians (
It will be assumed that the experts competence is determined on the basis of the assigned object rankings.
The number of solutions to the task of resulting ranking determining is influenced
by the following factors:
– set of used metrics S 1 = {d 1, d 2 ,...};
–criteria set for determining the closeness between the rankings S
2 = {δ 1 ,δ 2 ,...};
– is not the only solution of the median determining problem of type (
Thus, the number of task solutions for collective ranking of objects determining will s = si1s 2j + s 3 ,
"i ⇔ j" where the symbol means "metric indexes be equal to
i⇔ j i ∈ S 1 ,
for which there s t = S t , t = 1,2,3,
are indexes ⋅ − of proximity criteria j ∈ S 2 ", and
is the number of set elements.
As a result of applying all the methods of calculating the relative coefficients of expert competence, we obtain a set of object rankings, which are the median of the giveni e∈xpΩerКt ra=nkΩinCgSs∪: Ω ГВ ∪ Ω КС ∪ Ω ВГ ∪ Ω3 , i = 1,..., s.
R
According to [3], when a posteriori determination of the relative coefficients of competence in the ranking tasks, the distance from expert rankings to the calculated medians serves as a measure of competence. That is, the normalized weights of the experts relative competence on the basis of the found s resultant rankings that make up the set ΩК , will be calculasted by the formula: γ ij = (1/ d (Ri , R j ))/ (1/ d (Rt , R j )), Ri ∈ Ω К , i = 1,..., s, j ∈ I.
t=1 The idealized weight coefficients [7] of the relative competence of the criteria will be calculated by the formulas: γ iіjд = γ ij / max γ it , i = 1,..., s, j ∈ I .
t∈I
Since the variants of determining the resulting ranking can be tens, the coefficients of
competence can be calculated in the form of MF.
(
The algorithms for calculating MF in such cases are substantiated and described in [3, 4, 6]. The need to determine the coefficients of experts relative competence in the form of MF is due to various reasons. Although the use of different metrics and criteria to determine the resulting ranking are justified to varying degrees, they are all successfully applied in different application situations. Failure to use some of the tools entails the risk of losing information about the structure of preferences given by experts; in our case, linear ordering of objects. In order not to do so, all possible solutions obtained by different approaches are combined into a single array for their further analysis and conclusion.
This is the situation that Blaise Pascal wrote in [8]: «Justice and truth are two such thin points that our instruments turn out to be too coarse to touch. But if they are touched, they open the edge and lean on the environment, rather than a flaw than a truth». 5
Method for determining fuzzy set membership function by
analyzing frequency of values
To apply the method of determining the MF for frequency values must be determined
with the universal set. For the requirements of this research, the universal set for
determining the MF is the values from the interval (
It is necessary to analyze the results s of numerical estimates or the results of measurement or estimation of some value and determine the MF for the measured or estimated values. Typically, data is grouped at 10-20 intervals. However, the number of values that fall into each interval does not exceed 15-20% of their total. This is sufficient to identify all the properties of the magnitude and reliably calculate by group frequency the basic characteristics of the fuzzy set. In cases where the number of intervals exceeds 20, the MF may be polymodal and the information displayed will not be a clear representation of the fuzzy set. Grouping data at too large intervals can lead to the loss of the expert's view of the behavior of the MF, as well as to gross errors in its application.
Obviously, by analyzing the frequency of values, the number of values of the estimated values that belongs to each selected interval indicates the measure with which this value belongs to each interval. In this case, only the points are classified according to their values: whether or not the point belongs to the selected interval. The number of points included in the interval indicates the degree of optimality or quasi-optimality of this interval [6]. This justifies the use of frequency algorithms in determining the MF.
It is known [2] that for nominal features, that is, measured in the scale of names, as mean used the mode. For data measured in the ordinal scale, the median is a valid mean. When examining the information measured in the interval scale, only the arithmetic mean can be used. And for the data analysis specified in the scale of relations, stepwise averages and geometric averages are used.
In constructing the MF on the frequency of values, we actually use the results of mode analysis of the estimated meanings of the studied value. In [4, 6], several algorithms for the classification of the obtained solutions are constructed for constructing the MF of the investigated value on the analysis basis of the measured magnitude values frequency at some intervals chosen by the researcher. 6
Experiment results To illustrate the method described in this paper, we give an example of solving a small dimension problem for the number of objects n = 5 and the number of experts k = 6. The algorithm will be described in two steps. In the first stage we will present the methods and results of obtaining the median of the given rankings. Particular attention will be paid to comparing the modified medians that we propose to calculate in [2] with the actual medians found on the set of all possible rankings. We formulate reservations about the limited use of such a simplified aggregation tool set by expert rankings.
In the second stage of describing of the method for coefficients determining of experts relative competence in the form of MF, the procedure of obtaining the resulting values of normalized MF is described in detail, which characterizes the degree of belonging of different values of weight coefficients to fuzzy sets of possible normalized or idealized values of weight criteria.
Consider a set of five objects A = {a1, a2 , a3, a4 , a5}. Each expert determines with indexes set I = {1,2,...6}, the individual ranking of the objects Ri , i ∈ I : R1 = {a4 a1 a3 a2 a5}; R2 = {a3 a2 a4 a5 a1}; R5 = {a5 a1 a3 a4 a2 }; R6 = {a2 a3 a4 a1 a5}.
R3 = {a1 a4 a3 a5 a2 }; R5 = {a5 a1 a3 a4 a2 }.
The space of all possible strict five-object rankings Ω R consists of 5! = 120 rankings. The set of valid solutions to determine modified medians consists of only six expert rankings. 6.1
Calculation of medians set for given expert rankings
We compute the modified medians for the Hamming metric R МКС ∈ Ω МКС of the
form (
Modified Kemeny-Snell medians on a given set of expert rankings revealed three, i.e. 50% of the given:
R2 = {a3 a2 a4 a5 a1};
The total distance by Hamming's metric from each of the calculated medians to the six RgiМvКeСn ∈mΩedМiaКnСs=isAmrignimal a6ndd e(Bqu,Ball 5)4=:54.
min
B∈RB l=1 R2T=he{a3 moadified VG median for this example is the only R МВГ ∈ Ω2МВГa=4Aarg5 main1} and thedm(aBx,imB ulm)=d1is4ta.nce from it to all given is 14: max B∈RB l={1,2,...,6} one
Then compute the modified medians for the Cook metric of mismatch
R МCS ∈ Ω МCS of type (
Modified Cook-Sayford medians on a given set of expert rankings also revealed three, that is, 50% of the given ones, and for our example they coincide with the modif2ie=d{maedian of Kemeny-Snell:
R 3 a2 a4 a5 a1}; medRiaМnCsS ∈toΩthМeCSsix= gAirvgenmiisnm6indim(Ra,lRaln)d=e4q2u.al 42:
R∈R A l=1 R 4T=he{a2 a5 a1 a4 a3} and R5 = {a5 a1 a3 a4 a2 }, that is, 33% of modified GV medians (two) for our example are calculated the RgiМvГeВn∈,aΩndМtГhВe=mAaxrigmmumindimstaanxcedfr(oRm, Rthl e)m= 1to0.all given is 10:
B∈R A l={1,2,...,6}
Let's solve the problems of determining the median on the space of all possible five object rankings for the given six rankings.
Then compute the medians for the Hamming metric R МCS ∈ Ω МCS in the form (
The Kemeny-Snell medians for a given set of expert rankings in the space of all possible rankings of five objects is calculated six, that is 5% of the possible 120 rankRinКgСs(1o)f=fi{vae objects:
1 a2 a3 a5 a4 };
R КС (
3 ;
R КС (
2 ;
R КС (
4 ;
R КС (
3 ;
R КС (
4 .
The total distance by Hamming's metric from each of the calculated medians to the six given is minimal and is 50:
That is, the Kemeny-Snell medians have a better value of the additive criterion in
form (
TheRcВoГ m(
B∈ΩB l∈L
Thus, the compromise medians on the Hamming metric in form (
We compute the medians for the Cook metric of ranks mismatch RCS ∈ ΩCS in form
(
The Cook-Sayford medians of a given set of six expert rankings is five, that is, 4% of all possible strict rankings generated by five objects:
R CS (
4 ;
R CS (
3 ;
RCS (
2 ;
R CS (
2 .
The total distance by Cook's metric of the rank mismatch from each of the calculated medRiaCSns∈toΩtChSe =sixAgrgivmeninismidni(mR,uRml )a=nd4i0s.40:
R∈ΩR l∈L
That is, each of the calculated five rankings is the median of Cook-Sayford given six
expert rankings. Obviously, they are dominated by the above Cook-Sayford modified
medians, the value of criterion (
As a result of calculating the GV medians, six examples were identified in our example, that is, 5% of all possible rankings of five objects:
R ГВ(
5 ;
R ГВ(
4 ;
R ГВ(
4 ;
R ГВ(
3 ;
R ГВ (
3 .
The maximum distance from the calculated GV medians to all six rankings set by expRerГtsВ i∈s8Ω,thГВat=is:Arg min max d (R, R l ) = 8.
R∈ΩR l∈L
Thus, the GV medians are dominated by the above modified GV medians, for which the value of the minimax criterion is 10. This means that the modified medians can only be used in exceptional cases and it is better to refrain from this formulation.
In addition, choosing a solution among the rankings given by experts, we significantly narrow the space of choice and the dictator chosen in this way does not satisfy many of Arrow's axioms [1], and, above all, is not effective. 6.2
Determination of the experts’ relative competence Based on the medians calculated above for different metrics and different criteria, let us determine the competence coefficients of the experts who set the ranking of the objects based on the axiom (heuristic) of non-bias: the conclusions of most experts are competent. Against this background, we assume that expert competence is inversely proportional to the median defined in the space of all possible object rankings.
Since there are several ways to determine the median of given rankings in the algebraic approach, and each solution is often not unique, a whole family of relative competence factors of experts is generated that needs to be adequately considered, analyzed and reasoned appropriately.
To determine the coefficients of experts relative competence on the results of
calculating the medians that satisfy the criteria (
You can see that among the medians calculated by formulas (
Based on the data in formulas (
We construct a Table 4 with normalized values of the weights of the experts relative
competence, taking into account the frequency of values, and carry out the ordering of
the thus obtained normalized weights. Table 4 will have 19 rows, each row corresponds
to the expert competence coefficients that were determined based on the median that
was determined to be effective in solving one of the problems of (
To apply the method of determining the MF coefficients values of their frequency, we introduce two heuristics.
Heuristics E1. The interval in which all the normalized values of the weights γ ij , i = 1,...,19, j = 1,...,6,
are found is broken down into intervals with a width of 0.05.
Heuristics E2. The fixed value represented by each interval will be the rounded upper bound of each such interval.
As a result of the classification of points γ ij , i = 1,...,19, j = 1,...,6, according to the
eight intervals selected with regard to heuristics E1 and E2, we obtain Table 5.
As a result of Table 5 in terms of MF, we obtain the following values of singletons of
frequency for normalized values of weighting coefficients:
μ 1 (
Let us normalize the membership functions and present the fuzzy values of the relative
coefficients of experts' competence in the form of normal membership functions.
μ 1 (
The idealized [7, 9] values of the experts relative competence, which correspond to
the table of normalized coefficient values and calculated by the formula (
To apply to the analysis of idealized values of MF values coefficients determining method by their frequency, we introduce two more heuristics.
Heuristics E3. The interval [0;1], in which all the idealized values of the weights γ iіjд , i = 1,...,19, j = 1,...,6,
are found is divided into five intervals of width 0.2.
Heuristics E4. A fixed value representing each selected interval will be considered the upper bound of each such interval.
Let us classify the obtained 19 points, which correspond to the idealized valueγsiіjдo,f the coefficients, at five intervals. As a result of the classification of points i = 1,...,19, j = 1,...,6,
according to the five intervals selected with regard to heuristics
E3 and E4, we obtain Table 7.
Based on construction method the MF to fuzzy set by analyzing the frequency of values,
it is possible to present fuzzy normalized weights of experts' competence in the form of
singletons of weighting coefficients values frequency:
μ1 (
As a result of membership functions normalization, we finally obtain singletons to
express the fuzzy values of the experts relative competence weights for our example.
μ1 (
All constructed normal MFs are triangular, with a tendency to bell-shaped, except for the fifth, which is trapezoidal.
As a result of the method developed by the authors and described in this paper, the normalized and idealized values of the coefficients of experts relative competence in the problems of objects collective ranking are calculated. These values are fuzzy and are represented by the MF, which reflects the degree of correlation of the weights selected values on the basis of the heuristics entered, to a fuzzy set of normalized or idealized values. 7
Prospects for further research It is obvious that, depending on the content of the problem being solved, experts can be interpreted as methods that generate the ranking of measured or evaluated objects, or other sources of information that are used to rank objects of a non-numerical nature. Accordingly, not only competence but also weight, reliability, adequacy, convenience, robustness, speed of calculation, convergence and other parameters obtained from different sources of information can be investigated.
Methods for determining the coefficients of the experts relative competence in ranking problems can be interpreted as coefficients of the relative importance of sources of information [10]. In the following, we can consider the formulation of the problem and the approaches to its solution in view of its stated interpretation.
The approach described in this paper can be improved in the following areas: – formulation of determining problem statement for the experts competence who specify non-strict rankings [3, 11] (perfect quasi-orders [12], ordering [13], quasi-series [14], ranking with links [15], quasi-orders [16], clustered rankings [2]); – study the impact of the opinion consistency level of the expert group on the number of effective solutions of problems in different metrics and for different criteria; – identification of experts coalitions on the basis of expert rankings clustering with a considerable number of expert group members;
– study of resistance to manipulation with different metrics and different criteria for determining the median of given rankings;
– calculating the impact of coalitions' relative weight on the resulting ranking and sensitivity of metrics and criteria to changes the weight of expert coalitions; – taking into account expertly specified coefficients that reflect the influence of the methods of determining the median on final problem solution: the methods reliability, increasing confidence in them from the research or taking into account the experimental data;
– study of the conclusions bias of corrupt experts and determine the likelihood of a corrupt component in solving the problem of expert evaluation will be a development of the studies described in [17];
– study of the relationship between the intervals of normalized and idealized coefficients of relative competence of experts;
– the use of a heuristic method for determining the median of expert rankings or the genetic algorithm for finding medians in the presence of a large number of objects. 8
Conclusions The classification of methods for determining the experts competence is proposed. The prospect of applying a posteriori approach to calculating the coefficients of experts relative competence is substantiated. Based on the developed application for this approach, and the analysis of them using the method of determining the MF based on the frequency of values, an approach is proposed for the construction of weight coefficients of experts relative competence in the form of the MF.
An example illustrating the method proposed by the authors is discussed. Based on the example, the insufficient validity of the use of modified medians in ranking problems is illustrated. The concept of statistically significant or perfect median of expert rankings was introduced. The coefficients of the experts relative competence in the form of the membership function to a fuzzy set are also calculated.
The prospects of investigating the problems of determining the coefficients of the experts relative competence or the weighting coefficients of the relative importance of sources of information are outlined.