82 A posteriori determination of expert competence under uncertainty © Hryhorii Hnatiienko[0000-0002-0465-5018] and © Vitaliy Snytyuk[0000-0002-9954-8767] Taras Shevchenko National University of Kyiv, Kyiv, Ukraine g.gna5@ukr.net, snytyuk@gmail.com Abstract. The problem of determining the relative competence of experts in ex- pert assessment tasks is considered. The classification of competence determina- tion methods is offered and the use of objective approaches to a posteriori deter- mination of these coefficients is substantiated. The problem of determining the objects resultant ranking and ways of getting its solutions are given. Based on the analysis of obtained solutions in the group ranking problem, we propose a method of relative competence coefficients determining for experts in the form of fuzzy set membership functions. An example that demonstrates the effect and features of the described method application is shown. Keywords: expert evaluation, resultant ranking, expert competence, decision making, fuzzy set membership function. 1 Introduction Determining expert competence ratios is an important element in solving expert assess- ment problems. The accuracy of determining the relative competence of experts affects the outcome of the task, the reliability of the solution, and the credibility of result. Tak- ing into account competence of experts is the key to the quality of decision making and has a significant impact on the outcome of solution in the expert assessment problems. Therefore, determining of experts competence and their reasoned consideration is an actual area of research and can greatly contribute to improving the adequacy of the analysis of expert evaluation results and their validity. 2 Classification of methods for determining of experts competence Determining the experts competence is an important element of expert assessment. This factor significantly influences to decision making results and requires in-depth study, research and development. Today, there are several approaches to determining the rel- ative competence of experts, but each of directions is not perfect: Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 83 – documentary is often distrustful to ways of formalizing and taking into account objective data on experts, and is also determined by the current tendency to devalue official documents and the uncriticality of some institutional decisions; – self-assessment, during which often yields information about the level of self-con- fidence of the expert, rather than his real competence; – mutual evaluation, by which in some problems it is possible to detect confrontation in the expert committee and the existence of coalitions among its members, which sometimes distort the true individual competence of experts, which really has to influ- ence the results; – sometimes combined methods, instead of synergistic effects, lead to a cumulative error; – objective approaches, which are easy to apply and reasonably justified, are the most reliable to calculate adequate indicators of the relative competence of experts. Objective approaches distinguish between a priori and a posteriori methods of cal- culating competence. Let us dwell on the methods of a posteriori determination of the experts competence developed by the authors. Objective approaches to the posteriori determination of expert competence ratios include: – study of the expert's participation effectiveness in previous examinations; – calculation of competence based on the results of control examination; – study of the expert's participation results in a specific examination. The latter approach, which is to investigate the participation of an expert group mem- bers in a particular examination, may in turn be regarded as: – indirect analysis of indicators without first aggregating results; – the ratio of initial individual expert data and calculated integral indicators. In the problems of determining the group ranking of objects as components of an approach in which the relationship of individual expert data and calculated integral in- dicators are investigated are: – calculation of distances to pair comparison matrix (MCM) based on the found re- sultant ranking (in different metrics and by different criteria); – a posteriori determination based on the objects resultant ranking by given individ- ual expert rankings: = analysis of the inversions number in the expert ranking relative to the resulting ranking; = determination of the number of objects consistently ranked by the expert; = calculating the sum of difference modules between the objects rank in the re- sulting ranking and the expert's given rankings; – determination of individual relative estimates for collectively specified relative es- timates: = calculating the intervals sum of relative estimates; = determining the maximum interval among relative estimates; = calculating the "volume" of the intervals of the determined relative estimates; = calculation of the "area" of the intervals of relative estimates; = determining the deviation level of the MCM from a given "ideal" vector of rel- ative estimates; – distance to the group vector of weighting coefficients when given by the experts of MCM in cardinal scales; – the relative distance from expert rankings to the resultant rankings. 84 The modeling and study of the last two approaches are discussed in this paper. Such approaches are characterized by a situation of uncertainty. It is generated by the pres- ence of a large number of methods developed and substantiated by different authors for a posteriori calculation of the coefficients of relative competence of experts. At the same time, it is not possible to determine for certain practical situations precisely and precisely which of the methods developed is the most appropriate. Therefore, the au- thors propose to apply in the previous stages of the analysis all known methods of de- termining the relative competence of experts and only in the last stages of the analysis to aggregate the information obtained by different methods. 3 Formulation of the task of determining the resultant ranking by group evaluation The methods of processing expert information are divided into three main groups [1]: – statistical methods; – scaling methods; – algebraic methods. The essence of algebraic methods is that the set of admissible estimates sets the dis- tance and the resulting score is defined as the distance to which the expert estimates for the selected criterion are minimal. On the basis of algebraic approach we will determine the coefficients of relative competence of experts. l ∈ L = {1,..., k }, k > 1, Let k experts with index set A their preferences for the set of n objects in the form of object rankings R , l ∈ L. l To determine the distance between object rankings, researchers use: – Cook metric for mismatchingj of lobject rankings in individual rankings d (R , R ) =  ri j − ri l , i∈ I (1) ri l Rl , l ∈ L, 1 ≤ ril ≤ n, where – is the rank of the i-th object in the ranking of the l-th expert – Hamming's metric ( ) d B j , B l = 0,5 bisj − bisl , i∈ I s ∈ I (2) where ( ) B l = bisl , l ∈ L, i, s ∈ I ,− the MCM corresponding to the rankings R , l ∈ L. l Sometimes the Euclidean metric is used to determine the resulting ranking. Well known is the problem of collective expert assessment. Necessary to find the re- sultant (collective, group, compromise, integral, aggregated, agreed, collapsed, synthe- sized, generalized, global, etc.) ranking of objects that by some criterion is “closest” to all expert rankings. The most reasonable method of finding the resultant ranking of objects is to calculate the median of given rankings. The most common method of finding the resultant ranking of objects is to calculate the median of the given rankings. 85 We denote the set of all possible n object rankings by Ω , and the set of MCMs R corresponding to all possible n object rankings by Ω . We will denote the set of B B rankings given by the experts as R , and the set of the corresponding MCMs as R . A A B For the case considered in this paper, the capacity of the sets R and R is the same: R A = R B = n Rl ∈ R A , Bl ∈ R B , l ∈ L R A ⊂ ΩR , R B ⊂ ΩB , . It is clear that . In Ω B = 2 n (n−1)/ 2 the general case, the set capacity of the . But for the method described Ω R = Ω B = n! in this paper, we will assume that since we are not interested in non- B transitive elements of the solution space Ω . For Cook metric (object rank mismatch) (1), using the utilitarian criterion, the Cook- Sayford median is calculated [1]: R CS ∈Ω CS = Arg minR  d R, R . R∈Ω l ( ) l∈L (3) Modified Cook-SayfordМCS median:МCS R ∈Ω = Arg minA  d R, R l . R∈R ( ) l∈L (4) When using the egalitarian criterion, the GV-median (compromise) is calculated [1]: R ГВ ∈ Ω ГВ = Arg minR max d R, R l . R∈Ω l∈L ( ) (5) Modified GV median: R МГВ ∈ Ω МГВ = Arg minA max d R, R l . R∈R l∈L ( ) (6) For the Hamming’s metric (2), the Kemeni-Snell median is calculated using the utilitarian criterion: КС R КС ∈Ω = Arg minB  d B, B . B∈Ω l ( ) l∈L (7) Modified Kemeni-Snell medianМКС [2]: МКС R ∈Ω = Arg minB  d B , B l . B∈R ( ) l∈L (8) When using the egalitarian criterion, the HG-median (compromise) is calculated [3]: R ВГ ∈ Ω ВГ = Arg minB max d B , B l . B∈Ω l∈L ( ) (9) Modified VG median: R МВГ ∈ Ω МВГ = Arg minB max d B , B l . ( ) B∈R l∈L (10) 86 Into account can be taken expert competence ratios: ρ1 ,..., ρ k . Methods for determining the medians of species (3), (5), (7), (9) and their features are considered in the monograph [3]. Modified medians (4), (6), (8), (10) are proposed in some papers [2], but their use in many practical tasks is inappropriate and sometimes unreasonable and unjustified. The modified median significantly limits the choice space, so when applying these criteria, we usually find ineffective solutions: they are dominated, in particular, by the medians of (4), (6), (8), (10). In addition, the method described in this paper cannot be applied to modified medians, but finding modified medians is inappropriate and unrea- sonable in most cases. The criteria functions used to determine the median (3) - (10) are related to the dis- tances from the expert's given rankings to calculated solutions of the problem. There- fore, the criteria minimum (5), (6), (9), (10) for the corresponding formulations of the problem can be an efficiency feature of the problem obtained solution. Obviously, so- lutions that do not meet the minimum of the specified criteria are ineffective and they are dominated by solutions that deliver the minimum of the criteria (5), (6), (9), (10). The metrics and criteria for determining the median of given object rankings can have varying degrees of popularity among researchers, have unequal levels of trust, and be reasonably different. But all of these tools are well-established ways of determining the resultant ranking, and it makes no sense to ignore any of the well-known approaches that have proven themselves over the decades. Classical methods of choice theory (Condorcet, Borda, Simpson, Copland, Nanson, alternative votes, relative majority, etc.) can also be limited in determining the resulting ranking. Such use is described, for example, in a monograph [3]. Investigation of ap- plying classical methods of choice theory to determine the weights should be investi- gated with the further development of the method described in this paper. On the basis of calculation and analysis of the medians above (3), (5), (7), (9) it is proposed to calculate the coefficients of the relative competence of experts in the form of fuzzy set membership function (MF). The substantiation of such approaches was provided by the authors in [4-6]. 4 The task of determining the experts competence based on the analysis of the given individual expert rankings of objects To date, a considerable number of methods for defining collective expert assessments and calculating the relative competence of experts have been developed and substanti- ated in the expert evaluation direction. They are all entitled to existence and can be used to make decisions. To determine the experts competence by solving the specific prob- lem of expert evaluation, the authors suggest to use the analysis of existing methods of a posteriori calculation of coefficients of experts relative competence. The developed apparatus for determining the group estimates of the form (3) - (8) will be used and "collapsed" into one MF. In [3] one of the most important problems of expert evaluation, which is to determine the experts competence, is considered. The proposed methods of determining these co- efficients are based on the axiom of immutability: "Conclusions of the majority are 87 competent" and the consequence that "the most competent is the expert whose conclu- sions in most cases coincided with the conclusions of the majority of experts". The feature of mismatched MCMs and object rankings set is the large number of effective solutions in the problem of defining group object ranking. In particular, each of the calculated medians (3) - (8) may not be unique. It will be assumed that the experts competence is determined on the basis of the assigned object rankings. The number of solutions to the task of resulting ranking determining is influenced by the following factors: 1 { – set of used metrics S = d , d ,... ; 1 2 } –criteria set for determining the closeness between the rankings S = δ , δ ,... ; 2 { 1 2 } – is not the only solution of the median determining problem of type (3) - (5); – use to determine the resulting ranking of a set of classical choice rules: Condorcet, Bord, Simpson, Copland, Nanson, alternative votes, relative majority, etc. [1], the set of which is denoted by S . 3 Thus, the number of1 task solutions for collective ranking of objects determining will s =  si s j + s , 2 3 be equal to i⇔ j where the symbol " i ⇔ j" means "metric indexes i ∈ S1, for which there are indexes of proximity criteria j ∈ S 2 ", s t = S t , t = 1,2,3, ⋅ − and is the number of set elements. As a result of applying all the methods of calculating the relative coefficients of expert competence, we obtain a set of object rankings, which are the median of the giveni expertК rankings: CS ГВ КС ВГ R ∈Ω = Ω ∪Ω ∪Ω 3 ∪Ω ∪ Ω , i = 1,..., s. According to [3], when a posteriori determination of the relative coefficients of competence in the ranking tasks, the distance from expert rankings to the calculated medians serves as a measure of competence. That is, the normalized weights of the experts relative competence on the basis of the found s resultant rankings that make К up the set Ω , will be calculated s by the formula: γ ij = (1 / d (R i , R j ))/  (1 / d (R t , R j )), i R ∈ Ω К , i = 1,..., s, j ∈ I . t =1 (11) The idealized weight coefficients [7] of the relative competence of the criteria will be calculated by the formulas: γ ijід = γ ij / max γ it , i = 1,..., s, j ∈ I . t∈I (12) Since the variants of determining the resulting ranking can be tens, the coefficients of competence can be calculated in the form of MF. 88 The algorithms for calculating MF in such cases are substantiated and described in [3, 4, 6]. The need to determine the coefficients of experts relative competence in the form of MF is due to various reasons. Although the use of different metrics and criteria to determine the resulting ranking are justified to varying degrees, they are all successfully applied in different application situations. Failure to use some of the tools entails the risk of losing information about the structure of preferences given by experts; in our case, linear ordering of objects. In order not to do so, all possible solutions obtained by different approaches are combined into a single array for their further analysis and conclusion. This is the situation that Blaise Pascal wrote in [8]: «Justice and truth are two such thin points that our instruments turn out to be too coarse to touch. But if they are touched, they open the edge and lean on the environment, rather than a flaw than a truth». 5 Method for determining fuzzy set membership function by analyzing frequency of values To apply the method of determining the MF for frequency values must be determined with the universal set. For the requirements of this research, the universal set for deter- mining the MF is the values from the interval (0,1), namely, the results of determining the corresponding normalized coefficients of experts relative competence on the basis of calculating values (11) inverted to distances between rankings. For the idealized val- ues of the weight coefficients in the form (12), the universal set is the idealized values of the coefficients, which are also chosen from the interval (0,1). It is necessary to analyze the results s of numerical estimates or the results of meas- urement or estimation of some value and determine the MF for the measured or esti- mated values. Typically, data is grouped at 10-20 intervals. However, the number of values that fall into each interval does not exceed 15-20% of their total. This is suffi- cient to identify all the properties of the magnitude and reliably calculate by group fre- quency the basic characteristics of the fuzzy set. In cases where the number of intervals exceeds 20, the MF may be polymodal and the information displayed will not be a clear representation of the fuzzy set. Grouping data at too large intervals can lead to the loss of the expert's view of the behavior of the MF, as well as to gross errors in its applica- tion. Obviously, by analyzing the frequency of values, the number of values of the esti- mated values that belongs to each selected interval indicates the measure with which this value belongs to each interval. In this case, only the points are classified according to their values: whether or not the point belongs to the selected interval. The number of points included in the interval indicates the degree of optimality or quasi-optimality of this interval [6]. This justifies the use of frequency algorithms in determining the MF. It is known [2] that for nominal features, that is, measured in the scale of names, as mean used the mode. For data measured in the ordinal scale, the median is a valid mean. When examining the information measured in the interval scale, only the arithmetic mean can be used. And for the data analysis specified in the scale of relations, stepwise averages and geometric averages are used. In constructing the MF on the frequency of values, we actually use the results of 89 mode analysis of the estimated meanings of the studied value. In [4, 6], several algo- rithms for the classification of the obtained solutions are constructed for constructing the MF of the investigated value on the analysis basis of the measured magnitude values frequency at some intervals chosen by the researcher. 6 Experiment results To illustrate the method described in this paper, we give an example of solving a small dimension problem for the number of objects n = 5 and the number of experts k = 6. The algorithm will be described in two steps. In the first stage we will present the meth- ods and results of obtaining the median of the given rankings. Particular attention will be paid to comparing the modified medians that we propose to calculate in [2] with the actual medians found on the set of all possible rankings. We formulate reservations about the limited use of such a simplified aggregation tool set by expert rankings. In the second stage of describing of the method for coefficients determining of experts relative competence in the form of MF, the procedure of obtaining the resulting values of normalized MF is described in detail, which characterizes the degree of belonging of different values of weight coefficients to fuzzy sets of possible normalized or idealized values of weight criteria. A = {a1 , a2 , a3 , a4 , a5 }. Consider a set of five objects Each expert determines with indexes set I = {1,2,... 6}, the individual ranking of the objects Ri , i ∈ I : R1 = {a4  a1  a3  a2  a5 } ; R 2 = {a3  a2  a4  a5  a1 } ; R 3 = {a1  a4  a3  a5  a2 } ; R 4 = {a2  a5  a1  a4  a3 } ; R 5 = {a5  a1  a3  a4  a2 } ; R 6 = {a2  a3  a4  a1  a5 } . R The space of all possible strict five-object rankings Ω consists of 5! = 120 rankings. The set of valid solutions to determine modified medians consists of only six expert rankings. 6.1 Calculation of medians set for given expert rankings МКС МКС We compute the modified medians for the Hamming metric R ∈Ω of the МВГ МВГ form (8) and R ∈ Ω in the form (10). Modified Kemeny-Snell medians on a given set of expert rankings revealed three, i.e. 50% of the given: R 2 = {a3  a2  a4  a5  a1 } ; R 3 = {a1  a4  a3  a5  a2 } ; R 5 = {a5  a1  a3  a4  a2 } . 90 The total distance by Hamming's metric from each of the calculated medians to the 6 six given medians is minimal and equal 54: R МКС ∈ Ω МКС = Arg minB  d B , B l = 54 . B∈R ( ) l =1 The modified VG median for this example is the only one R 2 = {a3  a2  a4  a5  a1 } R МВГ ∈ Ω МВГ = Arg minB max d B , B l = 14 . B∈R l ={1, 2 ,..., 6} ( ) and the maximum distance from it to all given is 14: Then compute the modified medians for the Cook metric of mismatch R МCS ∈ Ω МCS of type (4) and R МГВ ∈ Ω МГВ of type (6). Modified Cook-Sayford medians on a given set of expert rankings also revealed three, that is, 50% of the given ones, and for our example they coincide with the modified median of Kemeny-Snell: R 2 = {a3  a2  a4  a5  a1 } ; R 3 = {a1  a4  a3  a5  a2 } ; R 5 = {a5  a1  a3  a4  a2 } . The total distance by Cook's metric of the rank mismatch from each of the calculated medians to the six given is minimal and equal 42: ( ) 6 R МCS ∈ Ω МCS = Arg minA  d R, R l = 42. R∈R l =1 The modified GV medians (two) for our example are calculated R 4 = {a2  a5  a1  a4  a3 } R 5 = {a5  a1  a3  a4  a2 } and , that is, 33% of the Rgiven, МГВ andМГВ ∈ Ω the =maximum Arg min distance max dfrom R , Rthem l ( to .all given is 10: = 10 B∈R A l ={1, 2 ,..., 6 } ) Let's solve the problems of determining the median on the space of all possible five object rankings for the given six rankings. МCS Then compute the medians for the Hamming metric R ∈ Ω МCS in the form (7) МГВ МГВ and R ∈Ω in the form (9). The Kemeny-Snell medians for a given set of expert rankings in the space of all possible rankings of five objects is calculated six, that is 5% of the possible 120 rankings of five objects: R КС (1) = {a1  a 2  a3  a5  a 4 } ; R КС ( 2 ) = {a1  a 2  a 4  a5  a3 } ; R КС ( 3) = {a1  a3  a 4  a5  a 2 } ; R КС ( 4 ) = {a 2  a1  a3  a5  a 4 } ; R КС ( 5 ) = {a 2  a1  a 4  a5  a3 } ; R КС ( 6 ) = {a3  a1  a 2  a5  a 4 } . The total distance by Hamming's metric from each of the calculated medians to the six given is minimal and is 50: 91 R КС ∈Ω КС B∈Ω ( = Arg minB  d B, B = 50. l ) l∈L That is, the Kemeny-Snell medians have a better value of the additive criterion in form (7) than the value of the additive criterion of type (8), and thus the modified Kemeny-Snell medians dominate. In other words, modified medians in form (8) are ineffective. And since these medians are not Pareto optimal, you can use them with considerable caution to solve group object ranking tasks. The compromise median, that is, the VG median, for our example, is calculated by R ВГ (1) = {a1  a2  a3  a5  a4 } R ВГ ( 2 ) = {a 2  a1  a3  a5  a 4 } two: and . TheRmaximum ВГ ∈ Ω ВГ distance = Arg minfrom these max d B B∈Ω B medians l∈L ( to .all given is 10: , B l = 10 ) Thus, the compromise medians on the Hamming metric in form (9) are also dominated by the modified VG-medians in form (10). This means that modified medians and for the minimax criterion can only be used with significant caveats, used as a reference, or to manipulate choices and to demonstrate sci-fi. CS CS We compute the medians for the Cook metric of ranks mismatch R ∈ Ω in form ГВ ГВ (3) and R ∈ Ω in form (5) in the space of all possible medians of five objects. The Cook-Sayford medians of a given set of six expert rankings is five, that is, 4% of all CS possible strict rankings generated by five objects: R (1) = {a 2  a1  a3  a 4  a5 } ; R CS ( 2 ) = {a 2  a1  a3  a5  a 4 } ; R CS ( 3) = {a 2  a1  a 4  a5  a3 } ; R CS ( 4 ) = {a  a1  a4  a5  a2 } ; R CS ( 5 ) = {a 4  a1  a3  a5  a 2 } . The total distance by Cook's metric of the rank mismatch from each of the calculated R CS ∈toΩthe medians CS =sixArg R  minis given minimum R∈Ω ( d R , R l and = 40 is. 40: ) l∈L That is, each of the calculated five rankings is the median of Cook-Sayford given six expert rankings. Obviously, they are dominated by the above Cook-Sayford modified medians, the value of criterion (3) of which is 42. As a result of calculating the GV medians, six examples were identified in our example, that is, 5% of all possible rankings of five objects: R ГВ (1) = {a1  a 2  a3  a 4  a5 } ; R ГВ ( 2 ) = {a 2  a1  a3  a 4  a5 } ; R ГВ ( 3) = {a1  a 2  a3  a5  a 4 } ; R ГВ ( 4 ) = {a 2  a1  a3  a5  a 4 } ; R ГВ ( 5 ) = {a 2  a1  a 4  a5  a3 } ; R ГВ ( 6 ) = {a 2  a1  a5  a 4  a3 } . 92 The maximum distance from the calculated GV medians to all six rankings set by experts R ГВ is∈8,Ωthat ГВ is: = Arg min max d R , R l = 8. R∈Ω R l∈L ( ) Thus, the GV medians are dominated by the above modified GV medians, for which the value of the minimax criterion is 10. This means that the modified medians can only be used in exceptional cases and it is better to refrain from this formulation. In addition, choosing a solution among the rankings given by experts, we significantly narrow the space of choice and the dictator chosen in this way does not satisfy many of Arrow's axioms [1], and, above all, is not effective. 6.2 Determination of the experts’ relative competence Based on the medians calculated above for different metrics and different criteria, let us determine the competence coefficients of the experts who set the ranking of the objects based on the axiom (heuristic) of non-bias: the conclusions of most experts are competent. Against this background, we assume that expert competence is inversely proportional to the median defined in the space of all possible object rankings. Since there are several ways to determine the median of given rankings in the algebraic approach, and each solution is often not unique, a whole family of relative competence factors of experts is generated that needs to be adequately considered, analyzed and reasoned appropriately. To determine the coefficients of experts relative competence on the results of calculating the medians that satisfy the criteria (3), (5), (7), (9), we summarize all the obtained values in a single Table 1. Table 1. Results of the median calculation that satisfy the criteria (3), (5), (7), (9) Frequency Distances from expert rankings to medians Ranking objects 1 2 3 4 5 6 {a1  a2  a3  a4  a5 } 1 8 10 8 10 12 6 {a 2  a1  a3  a 4  a5 } 2 6 8 10 12 10 8 {a1  a 2  a3  a5  a 4 } 3 10 8 6 8 10 8 {a1  a 2  a 4  a5  a 3 } 1 12 6 4 10 8 10 {a1  a3  a 4  a5  a 2 } 1 14 4 2 8 10 12 {a 2  a1  a3  a5  a 4 } 4 8 6 8 10 8 10 {a 2  a1  a 4  a5  a 3 } 3 10 4 6 12 6 12 {a 3  a1  a 2  a5  a 4 } 1 6 8 10 8 6 12 {a 3  a1  a 4  a5  a 2 } 1 12 2 8 14 4 14 {a 4  a1  a3  a5  a 2 } 1 10 4 10 12 2 16 {a 2  a1  a5  a 4  a3 } 1 8 6 8 14 8 10 93 The "frequency" row indicates the number of times that the appropriate ranking became the median by some criterion on a certain metric, or simultaneously by several criteria and metrics. You can see that among the medians calculated by formulas (3), (5), (7), (9), there is Дone that is present in each of the subsets of solutions. This ranking R = {a 2  a1  a3  a5  a 4 } , which is a median that delivers minimum simultaneously to all the criteria for all metrics and, accordingly, can be substantiated more comprehensively and substantiated than any of those that also fall into one or more subsets of equivalents by given solution criteria group ranking problems. We call this solution "statistically significant" or "perfect" median. Based on the data in formulas (11) in the previous table, we determine the normalized values of the weights of the experts relative competence, depending on which median is the solution of the task of determining group ranking. We present all the calculated coefficients in the form of Table 2. Table 2. The normalized values of the experts relative competence weights, depending on which median is the promlem solution of determining group ranking Normalized competency weights Frequency Ranking objects 1 2 3 4 5 6 {a1  a2  a3  a4  a5 } 1 0,179 0,143 0,179 0,143 0,119 0,238 {a 2  a1  a3  a 4  a5 } 2 0,238 0,179 0,143 0,119 0,143 0,179 {a1  a 2  a3  a5  a 4 } 3 0,135 0,169 0,225 0,169 0,135 0,169 {a1  a 2  a 4  a5  a 3 } 1 0,101 0,202 0,303 0,121 0,152 0,121 {a1  a3  a 4  a5  a 2 } 1 0,063 0,221 0,443 0,111 0,089 0,074 {a 2  a1  a3  a5  a 4 } 4 0,169 0,225 0,169 0,135 0,169 0,135 {a 2  a1  a 4  a5  a 3 } 3 0,118 0,294 0,196 0,098 0,196 0,098 {a 3  a1  a 2  a5  a 4 } 1 0,217 0,163 0,130 0,163 0,217 0,109 {a 3  a1  a 4  a5  a 2 } 1 0,076 0,454 0,114 0,065 0,227 0,065 {a 4  a1  a3  a5  a 2 } 1 0,091 0,228 0,091 0,076 0,456 0,057 {a 2  a1  a5  a 4  a3 } 1 0,175 0,234 0,175 0,100 0,175 0,140 We define the idealized values of the weight coefficients of the experts relative competence [7, 9,] and present them in the form of Table 3. Unlike the normalized values, for which the sum of the coefficients is equal to one, among the idealized values of the coefficients, the greatest is equal to one, and others are determined proportionally. We construct a Table 4 with normalized values of the weights of the experts relative competence, taking into account the frequency of values, and carry out the ordering of the thus obtained normalized weights. Table 4 will have 19 rows, each row corresponds 94 to the expert competence coefficients that were determined based on the median that was determined to be effective in solving one of the problems of (3), (5), (7), (9). Table 3. Idealized values of experts' relative competence weights Frequency Idealized weighted competency coefficients Ranking objects 1 2 3 4 5 6 {a1  a2  a3  a4  a5 } 1 0,75 0,60 0,75 0,60 0,50 1,00 {a 2  a1  a3  a 4  a5 } 2 1,00 0,75 0,60 0,50 0,60 0,75 {a1  a 2  a3  a5  a 4 } 3 0,60 0,75 1,00 0,75 0,60 0,75 {a1  a 2  a 4  a5  a 3 } 1 0,33 0,67 1,00 0,40 0,50 0,40 {a1  a3  a 4  a5  a 2 } 1 0,14 0,50 1,00 0,25 0,20 0,17 {a 2  a1  a3  a5  a 4 } 4 0,75 1,00 0,75 0,60 0,75 0,60 {a 2  a1  a 4  a5  a 3 } 3 0,40 1,00 0,67 0,33 0,67 0,33 {a 3  a1  a 2  a5  a 4 } 1 1,00 0,75 0,60 0,75 1,00 0,50 {a 3  a1  a 4  a5  a 2 } 1 0,17 1,00 0,25 0,14 0,50 0,14 {a 4  a1  a3  a5  a 2 } 1 0,20 0,50 0,20 0,17 1,00 0,13 {a 2  a1  a5  a 4  a3 } 1 0,75 1,00 0,75 0,43 0,75 0,60 Table 4. The normalized values of the weights of the experts relative competence, taking into account the frequency of values Normalized competency weights Item number 1 2 3 4 5 6 1 0,063 0,143 0,091 0,065 0,089 0,057 2 0,076 0,163 0,114 0,076 0,119 0,065 3 0,091 0,169 0,130 0,098 0,135 0,074 4 0,101 0,169 0,143 0,098 0,135 0,098 5 0,118 0,169 0,143 0,098 0,135 0,098 6 0,118 0,179 0,169 0,100 0,143 0,098 7 0,118 0,179 0,169 0,111 0,143 0,109 8 0,135 0,202 0,169 0,119 0,152 0,121 9 0,135 0,221 0,169 0,119 0,169 0,135 10 0,135 0,225 0,175 0,121 0,169 0,135 11 0,169 0,225 0,179 0,135 0,169 0,135 12 0,169 0,225 0,196 0,135 0,169 0,135 13 0,169 0,225 0,196 0,135 0,175 0,140 14 0,169 0,228 0,196 0,135 0,196 0,169 15 0,175 0,234 0,225 0,143 0,196 0,169 16 0,179 0,294 0,225 0,163 0,196 0,169 17 0,217 0,294 0,225 0,169 0,217 0,179 18 0,238 0,294 0,303 0,169 0,227 0,179 19 0,238 0,454 0,443 0,169 0,456 0,238 95 To determine the membership functions of the relative normalized weighting coefficients of the experts' relative competence, we shall classify the values obtained by formula (11) according to their belonging to the selected intervals. All values of the γ ∈ [0,057;0,456], normalized coefficients are in the range from 0.057 to 0.456: ij i = 1,...,19, j = 1,...,6. To apply the method of determining the MF coefficients values of their frequency, we introduce two heuristics. γ , Heuristics E1. The interval in which all the normalized values of the weights ij i = 1,...,19, j = 1,...,6, are found is broken down into intervals with a width of 0.05. Heuristics E2. The fixed value represented by each interval will be the rounded upper bound of each such interval. γ , i = 1,...,19, j = 1,...,6, As a result of the classification of points ij according to the eight intervals selected with regard to heuristics E1 and E2, we obtain Table 5. Table 5. Frequency of normalized values of the experts relative competence weights in selected intervals The value The number of values of normalized weights of expert competence that fall into the representing selected interval the selected interval 1 2 3 4 5 6 0,1 3 1 1 1 6 0,15 7 1 4 10 6 7 0,2 6 6 9 4 9 5 0,25 3 8 3 2 1 0,3 3 0,35 1 0,4 0,45 1 1 1 As a result of Table 5 in terms of MF, we obtain the following values of singletons of frequency for normalized values of weighting coefficients: μ1 (0,1) / 3 + μ1 (0,15) / 7 + μ1 (0,2) / 6 + μ1 (0,25) / 3; μ 2 (0,15) / 1 + μ 2 (0,2) / 6 + μ 2 (0,25) / 8 + μ 2 (0,3) / 3 + μ 2 (0,45) / 1; μ 3 ( 0 ,1) / 1 + μ 3 ( 0 ,15 ) / 4 + μ 3 ( 0 , 2 ) / 9 + μ 3 ( 0 , 25 ) / 3 + μ 3 ( 0 ,35 ) / 1 + μ 3 ( 0 , 45 ) / 1; μ 4 (0,1) / 5 + μ 4 (0,15) / 10 + μ 4 (0,2) / 4; μ 5 (0,1) / 1 + μ 5 (0,15) / 6 + μ 5 (0,2) / 9 + μ 5 (0,25) / 2 + μ 5 (0,45) / 1; μ 6 (0,1) / 6 + μ 6 (0,15) / 7 + μ 6 (0,2) / 5 + μ 6 (0,25) / 1. Let us normalize the membership functions and present the fuzzy values of the relative coefficients of experts' competence in the form of normal membership functions. μ1 (0,1) / 0,43 + μ1 (0,15) / 1 + μ1 (0,2) / 0,86 + μ1 (0,25) / 0,43; μ 2 (0,15) / 0,13 + μ 2 (0,2) / 0,75 + μ 2 (0,25) / 1 + μ 2 (0,3) / 0,38 + μ 2 (0,45) / 0,13; μ3 (0,1) / 0,11+ μ3 (0,15) / 0,44+ μ3 (0,2) /1+ μ3 (0,25) / 0,33+ μ3 (0,35) / 0,11+ μ3 (0,45) / 0,11; μ 4 (0,1) / 0,5 + μ 4 (0,15) / 1 + μ 4 (0,2) / 0,4; μ 5 (0,1) / 0,11 + μ 5 (0,15) / 0,67 + μ 5 (0,2) / 1 + μ 5 (0,25) / 0,22 + μ 5 (0,45) / 0,11; μ 6 (0,1) / 0,86 + μ 6 (0,15) / 1 + μ 6 (0,2) / 0,71 + μ 6 (0,25) / 0,14. 96 The idealized [7, 9] values of the experts relative competence, which correspond to the table of normalized coefficient values and calculated by the formula (12), can be represented in the form of Table 6. Table 6. Idealized values of experts' relative competence weights taking into account frequency of values Idealized weighted competency coefficients Item number 1 2 3 4 5 6 1 0,143 0,500 0,200 0,143 0,200 0,125 2 0,167 0,500 0,250 0,167 0,500 0,143 3 0,200 0,600 0,600 0,250 0,500 0,167 4 0,333 0,667 0,600 0,333 0,500 0,333 5 0,400 0,750 0,600 0,333 0,600 0,333 6 0,400 0,750 0,667 0,333 0,600 0,333 7 0,400 0,750 0,667 0,400 0,600 0,400 8 0,600 0,750 0,667 0,429 0,600 0,500 9 0,600 0,750 0,750 0,500 0,600 0,600 10 0,600 0,750 0,750 0,500 0,667 0,600 11 0,750 1,000 0,750 0,600 0,667 0,600 12 0,750 1,000 0,750 0,600 0,667 0,600 13 0,750 1,000 0,750 0,600 0,750 0,600 14 0,750 1,000 0,750 0,600 0,750 0,750 15 0,750 1,000 1,000 0,600 0,750 0,750 16 0,750 1,000 1,000 0,750 0,750 0,750 17 1,000 1,000 1,000 0,750 0,750 0,750 18 1,000 1,000 1,000 0,750 1,000 0,750 19 1,000 1,000 1,000 0,750 1,000 1,000 To determine the membership functions of the relative idealized weighting coefficients of the experts relative competence, classification of the values obtained by formula (12) according to their belonging to the selected intervals should be carried out. All values γ ід ∈ [0,125;1], of the idealized coefficients are in the range of 0.125 to 1: ij i = 1,...,19, j = 1,...,6. To apply to the analysis of idealized values of MF values coefficients determining method by their frequency, we introduce two more heuristics. Heuristics E3. The interval [0;1], in which all the idealized values of the weights γ ijід , i = 1,...,19, j = 1,...,6, are found is divided into five intervals of width 0.2. Heuristics E4. A fixed value representing each selected interval will be considered the upper bound of each such interval. Let us classify the obtained 19 points, which correspond to the idealized values ідof γ , the coefficients, at five intervals. As a result of the classification of points ij i = 1,...,19, j = 1,...,6, according to the five intervals selected with regard to heuristics E3 and E4, we obtain Table 7. 97 Table 7. Frequency of the weights idealized values of experts relative competence in selected intervals The value The number of values of the idealized weights of expert competence that fall into the representing selected interval the selected interval 1 2 3 4 5 6 0,2 1 2 3 4 5 6 0,4 4 2 6 1 3 0,6 3 2 4 0,8 3 4 6 7 11 6 1 6 6 6 4 5 5 Based on construction method the MF to fuzzy set by analyzing the frequency of values, it is possible to present fuzzy normalized weights of experts' competence in the form of singletons of weighting coefficients values frequency: μ1 (0,2) / 3 + μ1 (0,4) / 4 + μ1 (0,6) / 3 + μ1 (0,8) / 6 + μ1 (1) / 3; μ 2 (0,4) / 2 + μ 2 (0,6) / 2 + μ 2 (0,8) / 6 + μ 2 (1) / 9; μ 3 (0,4) / 2 + μ 3 (0,6) / 3 + μ 3 (0,8) / 9 + μ 3 (1) / 5; μ 4 (0,2) / 2 + μ 4 (0,4) / 5 + μ 4 (0,6) / 8 + μ 4 (0,8) / 4; μ 5 (0,4) / 1 + μ 5 (0,6) / 8 + μ 5 (0,8) / 8 + μ 5 (1) / 2; μ 6 (0,2) / 3 + μ 6 (0,4) / 4 + μ 6 (0,6) / 6 + μ 6 (0,8) / 5 + μ 6 (1) / 1. As a result of membership functions normalization, we finally obtain singletons to express the fuzzy values of the experts relative competence weights for our example. μ1 (0,2) / 0,5 + μ1 (0,4) / 0,67 + μ1 (0,6) / 0,5 + μ1 (0,8) / 1 + μ1 (1) / 0,5; μ 2 (0,4) / 0,2 − 2 + μ 2 (0,6) / 0,22 + μ 2 (0,8) / 0,67 + μ 2 (1) / 1; μ 3 (0,4) / 0,22 + μ 3 (0,6) / 0,33 + μ 3 (0,8) / 1 + μ 3 (1) / 0,56; μ 4 (0,2) / 0,25 + μ 4 (0,4) / 0,63 + μ 4 (0,6) / 1 + μ 4 (0,8) / 0,5; μ 5 (0,4) / 0,13 + μ 5 (0,6) / 1 + μ 5 (0,8) / 1 + μ 5 (1) / 0,25; μ 6 (0,2) / 0,5 + μ 6 (0,4) / 0,67 + μ 6 (0,6) / 1 + μ 6 (0,8) / 0,83 + μ 6 (1) / 0,17. All constructed normal MFs are triangular, with a tendency to bell-shaped, except for the fifth, which is trapezoidal. As a result of the method developed by the authors and described in this paper, the normalized and idealized values of the coefficients of experts relative competence in the problems of objects collective ranking are calculated. These values are fuzzy and are represented by the MF, which reflects the degree of correlation of the weights se- lected values on the basis of the heuristics entered, to a fuzzy set of normalized or ide- alized values. 7 Prospects for further research It is obvious that, depending on the content of the problem being solved, experts can be interpreted as methods that generate the ranking of measured or evaluated objects, or other sources of information that are used to rank objects of a non-numerical nature. 98 Accordingly, not only competence but also weight, reliability, adequacy, convenience, robustness, speed of calculation, convergence and other parameters obtained from different sources of information can be investigated. Methods for determining the coefficients of the experts relative competence in rank- ing problems can be interpreted as coefficients of the relative importance of sources of information [10]. In the following, we can consider the formulation of the problem and the approaches to its solution in view of its stated interpretation. The approach described in this paper can be improved in the following areas: – formulation of determining problem statement for the experts competence who specify non-strict rankings [3, 11] (perfect quasi-orders [12], ordering [13], quasi-series [14], ranking with links [15], quasi-orders [16], clustered rankings [2]); – study the impact of the opinion consistency level of the expert group on the number of effective solutions of problems in different metrics and for different criteria; – identification of experts coalitions on the basis of expert rankings clustering with a considerable number of expert group members; – study of resistance to manipulation with different metrics and different criteria for determining the median of given rankings; – calculating the impact of coalitions' relative weight on the resulting ranking and sensitivity of metrics and criteria to changes the weight of expert coalitions; – taking into account expertly specified coefficients that reflect the influence of the methods of determining the median on final problem solution: the methods reliability, increasing confidence in them from the research or taking into account the experimental data; – study of the conclusions bias of corrupt experts and determine the likelihood of a corrupt component in solving the problem of expert evaluation will be a development of the studies described in [17]; – study of the relationship between the intervals of normalized and idealized coefficients of relative competence of experts; – the use of a heuristic method for determining the median of expert rankings or the genetic algorithm for finding medians in the presence of a large number of objects. 8 Conclusions The classification of methods for determining the experts competence is proposed. The prospect of applying a posteriori approach to calculating the coefficients of experts rel- ative competence is substantiated. Based on the developed application for this ap- proach, and the analysis of them using the method of determining the MF based on the frequency of values, an approach is proposed for the construction of weight coefficients of experts relative competence in the form of the MF. An example illustrating the method proposed by the authors is discussed. Based on the example, the insufficient validity of the use of modified medians in ranking prob- lems is illustrated. The concept of statistically significant or perfect median of expert rankings was introduced. The coefficients of the experts relative competence in the form of the membership function to a fuzzy set are also calculated. 99 The prospects of investigating the problems of determining the coefficients of the experts relative competence or the weighting coefficients of the relative importance of sources of information are outlined. References 1. Voloshyn, O. F., Mashchenko, S. O.: Decision theory: tutorial. Kiev: Publishing and Print- ing Center "Kyivskyi universytet", 366 p. (2010). 2. Orlov,A. I. Management decision-making methods: a textbook. M: KNORUS 286p. (2018). 3. Hnatiienko, H. M., Snytyuk, V. E.: Expert Technologies for Decision Making. – K.: Com- pany ”Maklaut”. – 444 p. (2008). 4. Hnatiienko, H. M. Algorithms for determining the function of belonging by analyzing the time-accuracy of values // Proceedings of the III International School-Seminar "Decision- making Theory", Uzhgorod, UzhNu.-P.32-34. (2006). 5. Snytyuk, V. E., Hnatiienko H. M. Optimization of the estimation process under uncertainty conditions based on the structure of the domain and the axiom of unbiased // Artificial In- tellect. – No.3. – Pp. 217–222. (2008). 6. Hnatiienko H. M. Determination of the weight coefficients of the criteria of the multicriteria optimization problem in the form of functions of fuzzy set membership // Proceedings of the 5th International Scientific and Practical Conference "Information Technologies and Inter- actions" (IT&I - 2018). - K. VOC "Kyiv University". Pp.29-30. (2018). 7. Saaty T. L. The relative dimension and its generalization in decision making. Why pairwise comparisons are key in mathematics for measuring intangible factors / T.L. Saaty // Cloud of Science. - Vol. 3, No. 2. - Pp. 171–262. (2016). 8. Pascal Blaise. Thoughts. / Trans. with fr. - K .: SPIRIT AND LETTER. - 704 p. (2009). 9. Saaty T. L. About measuring the intangible. An approach to relative measurements on the basis of the principal eigenvector of a matrix of pairwise comparisons / T. L. Saaty // Cloud of Science. - Vol. 2, No. 1. - Pp. 5–39. (2015). 10. Dodonov, A., Lande, D., Tsyganok, V., Andriichuk, O., Kadenko, S., Graivoronskaya, A.: Information Operations Recognition. From Nonlinear Analysis to Decision-Making. Lam- bert Academic Publishing, (2019). 11. Makarov, I. M., Vinogradskaya, T. M., Rubchinsky, A. A., Sokolov, V. B. The theory of choice and decision-making. Moscow: Nauka. – 328 p. (1982). 12. Schrader, Yu.A. Equality, similarity, order. - M.: Science. - 256 p. (1971). 13. Kemeny, J. G., Snell, J. L.: Cybernetic modeling: Some applications. M.: Soviet Radio. - 192p. (1972). 14. Mirkin, B. G. The problem of group choice.. M.: Nauka. - 256 p. (1973). 15. Hollander, M., Wolf, D. Nonparametric methods of statistics. M.: Finance and Statistics. - 518 p. (1983). 16. Voloshin, O. F, Hnatiienko, H. M, Kudin, V. I. Consistent analysis of options. Technology and application. – K.: Stylos, 304 p. (2013). 17. Gnatiyenko, G. M, Ledomska, L. S. Some aspects of the phenomenon of corruption and its impact on the economy of Ukraine // Economic Journal. No.9, Pp.29-34. (1997).