Considering Importance of Information Sources during Aggregation of Alternative Rankings

Abstract

The paper outlines several approaches to aggregation of rankings of alternatives, provided by a set of individual information sources (IS). It is shown, that, depending on dimensionality of the set of alternatives, different aggregation methods can be applied. Particular attention is given to weighted aggregation of alternative rankings provided by different IS. Some intuitive assumptions concerning IS weight calculation are set forth. It is assumed that IS weight should reflect both quantity and quality of information it provides, as well as expert estimate of IS credibility based on previous experience of IS usage. Based on these assumptions, expressions for IS weight calculation are suggested. The approaches, suggested in the paper can be applied to information, provided by sources of different nature, i.e., experts, search engines, paper and online documents, etc.

1 Introduction

In order to compile a ranking of alternatives based on all available information, rankings, coming from several information sources, need to be taken into consideration and aggregated in some way. Authors of [2] show that information space is influenced by multiple factors of qualitative nature, which are difficult to formalize and describe in numeric terms. Based on these characteristics, it was shown in [ 3 ] that information space was a typical example of a weakly structured subject domain. In order to provide at least some analytical description of a weakly structured domain, data, coming from any available sources, needs to be taken into consideration. In this paper we will try to address the most general case, when information sources can include paper and online documentation, web search results, and expert judgments.

The problem with aggregation of data, coming from different information sources is that, in the general case, these sources have different weights, depending, again, on several factors. For example, it is reasonable to assume, that more credible information source, providing larger amount of information, should be assigned greater weight prior to data aggregation. While there is, roughly, a dozen of methods of ranking aggregation (proposed by Borda, Condorcet, Kemeny, and others), that are commonly known and described in a multitude of publications, the problem of defining the weights of data sources (voters, judges, experts, search engines etc.) during data aggregation is still relevant and open to discussion.

So, in further sections of our paper, we are going to set forth a method, allowing to aggregate data, presented in the form of alternative rankings, coming from different information sources, particularly focusing on different aspects of information source weight definition problem.

The problem of rank aggregation became relevant back in the ancient times, when democratic societies and voting systems started to emerge. An extensive analysis of voting rules and their evolution is provided, for example, in [ 4 ].

The first rank aggregation methods, that emerged in the process of democratic voting evolution, which are still relevant today, were suggested in the late 18th century by Borda ([ 5 ]) and Condorcet ([ 6 ]). Since then multiple approaches to aggregation of individual preferences using different social welfare functions were devised. We find it most appropriate to mention the period of 1950-s and 1960-s – the time when Arrow’s impossibility theorem was formulated (see [ 7 ]) and attempts were made to bypass its constraints (particularly, by Kemeny [ 8 ] and Copeland [ 9 ]). F.Aleskerov in [ 10 ] summarizes Arrovian aggregation rules in his book.

In the 2000s, with growing popularity of online information search engines, the problem of rank aggregation became even more relevant, as it became evident, that beside data obtained from voters, experts, and analysts, it was necessary to take online information sources into account (sometimes, with minimum human involvement). In this context, we should mention several papers from the 2000s, particularly [ 11-13 ]. In these publications the authors suggest and compare several approaches, targeted at aggregation of data from online information sources.

Definition of weights of information sources and weighted aggregation of rankings represents a separate matter. Methods of Borda and Condorcet can be easily extrapolated to the case when information sources have different weights, as shown by Totsenko in [ 14 ]. Kemeny’s median is a more problematic case when information sources have different weights. Ordinal factorial analysis methods, described in [ 15, 16 ] allow analysts to calculate weights based on available sets of alternative rankings, previously provided by experts. This approach can be extrapolated to calculation of weights of online information sources as well, if evaluators provide some global alternative ranking a priori.

Dwork et al in [ 11 ] stress the importance of involvement of human evaluators in the process of preliminary information source weight definition. However, their paper focuses mostly on comparative analysis of rank aggregation methods and does not address the problem of IS weight calculation directly.

As for IS weights, we can, again, mention Tom Saaty ([ 1 ]) and his academic school (including Ramanathan & Ganesh [ 17 ], Forman & Peniwati [ 18 ], as well as Yang et al [ 19 ]). However, their methods are primarily targeted at aggregation of expert estimates, represented in the form of pair-wise comparison matrices (PCM), provided in some ratio scales (and not rankings).

In our current paper we are going to suggest information source weight calculation methods based on both preliminary human evaluation of information source credibility and amount and quality of information, provided by a given information source.

3 The problem statement

What is given: n information sources (IS1 – ISn). Each of IS provides a ranking Ri, that includes mi alternatives (i=1..n). Information sources can be experts, search engines, analysts who monitor online content, paper or online documents, etc. We should stress, that the number of alternatives, provided by different information sources is, in the general case, different ( mi  m j ;i, j  1..n ).

We should find an aggregated (global) ranking of alternatives.

4 Solution ideas for the case of equal weights of information sources

We suggest building a solution algorithm based on available methods of aggregation of individual rankings (ordinal estimates). The most common ranking aggregation methods were proposed by Borda, Condorcet and, perhaps, Kemeny. We should stress that if we are dealing with experts, the number of alternatives an expert can analyze “in one go” is no more than 7±2. However, search engines can return rankings of hundreds of links. That is why, usage of domination matrix – based methods is not recommended for such dimensionalities. For example, if we are dealing with 10 IS, providing rankings of 1000 alternatives each, then we have to analyze and process 10 matrices with 1000×1000 cells. In such cases Borda method seems to be the most appropriate option, as it operates with ranking vectors and not domination matrices. Besides, it is easier to extrapolate Borda method to the case when IS have different weights (while, for in stance, extrapolation of Markov chain-based methods looks more problematic). If all IS have the same weight, we can use the following algorithm.

1) Find the length of a ranking with all unique alternatives, featured in individual rankings n M  card ({A(ji) ; j  1..mi }) .

i1 2) Form a unified set of alternatives. For this purpose we should add to each ranking Ri of mi alternatives n all the remaining alternatives {A(jk) ; j  1..mk } /{A(ji) ; j  1..mi } with rank mi+1. During unification of k1 alternatives, provided by different IS, we should check whether alternatives are conceptually the same. If IS are experts, then for this purpose we can use the methods of semantic similarity determination, suggested in [20].

3) As a result, we will get M alternatives in each ranking. Each ranking Ri will include mi alternatives with different ranks and (M – mi) alternatives with the same rank (mi+1). (A similar approach was mentioned by Dwork et al in [ 11 ]).

4) Add the ranks of each alternative across all IS.

5) Sort (order) the alternatives in the order of increment of individual rank sums. This sorting will result in the rank order that we should find.

n n  rkj   rlj  rk  rl ,k ,l  1..M (1), j 1 j 1 where rkj и rlj – are the ranks of alternatives Ak and Al in the ranking, provided by IS number j.

5 Usage of different methods under smaller cardinality of the set of alternatives

If the number of alternatives amounts to one or two dozens, or if the decision-maker (analyst) is interested only in the rank order of the first few alternatives, provided by IS, then it makes sense to use other ranking methods (beside or instead of Borda) in order to aggregate the individual ranking results.

For example, let us assume, we have 10-15 IS, and each of them provides a ranking of alternatives, from which we are particularly interested in the top 10-20 items. If all alternatives (items) are different, then we have 300 (1520) different items at most. In fact (especially if we are talking about search engines), many items featured in individual rankings across different IS are the same. So, if we have 15 IS and each of them provides a ranking of 20 alternatives, then the cardinality of the set of unique items will amount to approximately 100 alternatives. Consequently, if we use aggregation methods operating with ordinal pair-wise comparison matrices (PCM) (such as Condorcet rule or Kemeny’s median) and not with ranking vectors (like Borda or Markov Chain based methods), we will have to analyze 15 square matrices containing 100100 cells. As strict rank order relationship is reciprocally symmetrical (if alternative А1 ranks higher than А2, then А2 ranks lower than А1), during aggregation of PCM we will have to analyze just matrix elements above the principal diagonal {dik, i<k}. That is why, even if the aggregate ranking features 100 alternatives, then we will have to analyze (10099)/2 or 4950 cells in each PCM. These calculations demonstrate, that under smaller cardinality of alternative set, we can use Condorcet rule and Kemeny’s median for aggregation of individual ranking results.

In our view, the disadvantage of Borda method is that during aggregation of individual rankings we are witnessing an explicit heuristic transition from ordinal preference scale to ratio scale. However, this transition is inevitable, because representation of alternative estimates in ratio scale is the necessary and sufficient condition of existence of linear global criterion, allowing us to aggregate these estimates (as proven by Litvak in [21]). For example, an alternative with rank 2 does not necessarily have to be exactly 2 times better than alternative with rank 4 (we should remind, that dominating alternatives are assigned smaller ranks).

Condorcet rule, Markov chains, and Kemeny’s method allow us to avoid such an explicit transition from ranks to ratios.

6 Existing methods: a brief overview

Condorcet rule. The key idea of Condorcet method is as follows. If alternative Аi dominates over alternative Ak in the majority of individual rankings, then this preference relationship should be maintained in the global (aggregate) ranking. In order to get an aggregate ranking we should first build individual ordinal PCMs. If in some individual ranking Rj alternative Ai dominates over Ak, then the respective element of ordinal PCM equals 1, if Ai is dominated by Ak – (-1), if the alternatives are equal – 0: 1, Ai  Ak  di(kj )  0, Ai  Ak  1, Ai  Ak

(2) In order to aggregate rankings, provided by several information sources according to Condorcet’s rule, we should limit the number of alternatives to be featured in the global ranking by a fixed number M and build ordinal PCM based on rankings, provided by all IS. { D( j )  { di(k j ) ; i,k  1..M }; j  1..n } . After that we should calculate the sums of all respective elements of individual PCM and fill ordinal PCM, corresponding to the global rank ordering relationship (tournament).

D  { dik ;i,k  1..M } where  n 0,  di(k j )  0  j 1  n dik  1,  di(k j )  0  j 1  n 1,  di(kj )  0  j 1 (3).

After that we should add the elements in each line of the global rank ordering relationship matrix D and rank the obtained line sums.

M M  dik   dlk k 1 l 1  r  rl (4), i where ri and rl are ranks of alternatives Ai and Al in the global ranking R.

The disadvantage of Condorcet’s rule (beside dimensionality limitation) is that equal alternative ranks may emerge and transitivity of the global preference relationship may be violated as a result of the so-called “Condorcet paradox”. The paradox is one of the specific cases of violation of Arrow’s requirements to social choice functions [ 7 ]. If feedback is acceptable, then transitivity of the global preference relationship can be achieved if we use algorithms, described in [ 22, 23 ].

In the context of aggregation of results of online data search, provided by several engines, the advantage of Condorcet’s rule (mentioned by Dwork et al in [ 11 ]) is its ability to filter spam content.

A more “just” aggregation rule in view of Arrow’s impossibility theorem is Kemeny’s rule (sometimes called Kemeny’s median).

Kemeny’s median can be considered the analogue of average mean for ordinal (non-cardinal) estimates.

As ordinal estimates (or ranks) do not bear information on quantitative relation between alternatives, such concepts as Euclidian distance metric or center of mass (center of gravity) do not apply to rank order vectors.

Distance between two rankings of a given set of alternatives depends upon the number of elementary permutations that is needed to obtain one ranking from the other. Kemeny’s distance between two rankings is based on Haming metric. If we have two rankings R1, R2 of a set of alternatives A={A1..Am}, and respective domination matrices D1 and D2 are built based on these rankings according to formula (2), then Kemeny’s distance K is calculated as follows.

If we have a set of n rankings {R1..Rn} of alternatives A={A1..Am}, then, by definition ([ 8 ]), Kemeny’s median of this set of rankings is a ranking R:

m m K ( R1,R2 )    di(k1 )  di(k2 ) i1 k 1

(5), n R  arg min  K ( R, R j )

RT j 1 (6), where T is a set of all possible rankings of alternatives. Kemeny’s median calculation procedure is very laborintensive. Some estimates of complexity of this problem are provided in [ 11 ]. One of the “simplest” algorithms of Kemeny’s median calculation is provided by Litvak in [21].

7 Generalization of suggested approaches to the case of different weights of information sources

Let us now assume that IS weights {w1..wn} (reflecting their credibility) have been provided by experts, calculated based on previous estimation experience (as described in [ 15, 16 ]), or obtained in some other way. In this case, respective formulas for ranking aggregation will change, as all rankings, provided by individual IS, will be taken into consideration together with their respective weights. Formula (1) (corresponding to Borda aggregation method) will look as follows.

n n  w r j lj  rk  rl ,k ,l  1..M (7)

j kj   w r j 1 j 1 Formula (3) (corresponding to Condorcet method) will look as follows.

D  { dik ;i,k  1..M } where  n 0, w jdi(kj )  0  j 1  n dik  1, w jdi(kj )  0  j 1  n  1,  w jdi(kj )  0  j1 (8) In formula (6) weight coefficients will become multipliers for respective Kemeny distance values.

n R  arg min  w j K( R, R j )

RT j 1 (9).

Extrapolation of Markov chain based methods, described in [ 11 ], represents a problem that should be addressed in a separate research.

Experience and numerous publications indicate that when a group of experts participates in decisionmaking process, individual expert competence should be taken into consideration (providing, the expert group is relatively small) (for details – see [ 24-26 ]). Similarly, if several IS are used to build a ranking of alternatives according to their relevance in terms of a particular information query or in the process of some analytical research, we should take their weights into account as well.

Now let us address the issue of calculation of IS weights in greater detail. So far we have come up with several conceptual approaches to IS weight calculation. These approaches are outlined in the next section.

8 Approaches to calculation of the relative weights of information sources 1) Experience-based approach was set forth in [ 15, 16 ]. In essence, experts or evaluators provide their ranking of alternatives and then, based on their ranking and rankings, provided by individual IS, the weights of IS are calculated (under assumption that rankings are aggregated using Borda or Condorcet rule).

2) Definition of relative IS weights based on quantity and quality of provided information. It is reasonable to assume, that the relative weight of an IS should depend on quantity and quality of information, provided by the source in terms of every information query (in the given subject domain). Criteria, representing quality and quantity of information search, following a given query can be formulated as follows.  Number of alternatives (references or links in case of online information search), provided by the IS in response to a particular query;  Relevance of these alternatives (references, links), i.e. their correspondence to the specific query.

Relevance of the results of functioning of an IS can be defined based on the estimates of users (evaluators, experts) from the given subject domain. These estimates of IS can be based on previous experience of information search queries.

Relevance of results of IS work can be considered in terms of search queries of some particular type (search for information in some given language, formulas, images etc.). If it is possible to outline some query type, expert estimate will depend upon IS capability of processing this particular type of queries. Otherwise (when it is problematic to outline specific query types) it makes sense to use an average value of expert estimate (across different query types).

In the context of this subsection we suggest calculating IS weight using the approach, similar to methods, used by Totsenko for calculation of expert competence in [ 27, 28 ]. In accordance to this approach, expert competence should depend on self-estimate, objective estimate, and mutual estimate.

k  s(x1b  x2v) ,

(10) Vi  nmi ,  mk k1 Oi  m

i ,

P wi  nwwi*i* , i1 where wi* is the non-normalized IS weight, , x1 , x2 are the respective coefficients of components’ relative importance, and Ei is the non-normalized expert estimate value. Normalized values are calculated as follows. where k is the relative competence of the expert, s is self-estimate, b is an objective component, v is mutual estimate of expert group members, while x1 , x2 represent the coefficients of relative importance of objective and mutual estimates’ respectively.

When the weights of “generalized” IS (i.e. experts, search engines, documents, etc.) are calculated, mutual estimate can be replaced by the ration between the number of alternatives, provided by the i-th IS in relation to the total number of alternatives, provided by all IS Vi : where mi is the number of alternatives, provided by i-th IS, n is the total number of information sources.

Self-estimate in formula (10) can be replaced by the value of expert estimate Ei . This estimate depends on previous experience of IS usage. If weights of several IS are estimated by several experts through pair-wise comparisons of the IS, then expert estimates can be aggregated using combinatorial method, described in [ 29 ]. If expert estimates are not consistent enough for aggregation, then consistency should be improved through feedback with experts, based on spectral approach, as described in [ 30 ]. For each particular query type (for instance, Ukrainian, English, image, formula, other) we should apply the respective average (aggregate) value of Ei . Definition of query types is not the subject of this paper and should be addressed separately.

Objective estimate in formula (10) can be replaced by the ratio between the number of alternatives, provided by i-th IS and the total number of unique alternatives, provided by all IS Oi (thus, it will reflect the ability of IS to provide unique information). (11) (12) (14) where wi is the normalized relative IS weight.

We propose to follow the assumption that “objective” and “mutual” IS estimates should form a convex combination ( x1  x2  1), so x1 and x2 should lie within the [0;1] range and vary depending on particular information query.

As a result of each information search, every IS provides several alternatives (in the general case they are different, as mentioned above). Figure 1 represents an example, where 5 IS provide 9 unique alternatives. 1st and 6th alternatives were provided by 2 IS, 2nd and 5th – by 3 IS, 3rd alternative – by 5 IS, while other alternatives were provided by 1 IS each.

In order to illustrate the behavior of dependence of x1 and x2 on information search results, let us consider several extreme cases.

Extreme case 1

Let us assume that all IS provided completely different alternative sets. Generalized information search results are displayed on Fig. 2: 5 IS provide 9 different alternatives and each of the alternatives is provided by a single IS (for instance, IS1: a1, a2; IS2: a3, a4; IS3: a5; IS4: a6, a7, a8; IS5: a9). If all IS provide different alternatives in response to some query, the relevance of these alternatives seems questionable. In such cases we suggest using expert estimate (based on previous experience of IS usage) as the key component of IS weight, as it reflects the experience of previous information search sessions and the ability of IS to find new information.

Extreme case 2

Let us assume, that all IS provided the same alternatives. An example of a hypothetical information search results are presented on fig. 3: 5 9 alternatives are provided by all 5 IS (although the order, in which alternatives are ranked by different sources, may be different).

In such a case, IS weights should be equal to Ei , because all IS are equally efficient in providing alternatives in response to information query. Their weights can be defined, again, only based on the previous history of information search sessions that is reflected by the expert estimate.

In order to devise a formal expression for x1 and x2 , based on these intuitive considerations, let us introduce the indicator  , characterizing the alternative frequency function or density of representation of alternatives across all information sources.

P  hj   j1 nP , (15) where hj is the quantity of IS, that provided j-th alternative, while n is the total quantity of IS. In “extreme case 1”  assumes the minimum value   1 .

n In “extreme case 2”  assumes maximum values   1.

In view of above-mentioned convexity requirements, we suggest calculating x2 as x2   ; x1  1  x2 .

(16) In “extreme case 1” the normalized relative IS weight assumes the value of w  i “extreme case 2” it assumes the value of wi  Ei .

3) Statistical approach represents a modification of the previous approach. In order to take the quantity of information, provided by the IS (“objective” weight component in formula (13)) into account, we can normalize the numbers of alternatives across all IS (see formula (11)) and then calculate the importance of the respective component (i.e. value of x1) as dispersion if this indicator (normalized number of alternatives). 1 1 Ei ((1  )Oi 

n n 1  E j ((1  )O j  j1 n

V ) n i 1 n V j ) , and in x1  D(V ); x2  1  x1 (17) In this case, if all IS provide the same number of alternatives, the respective component of their relative weights can be neglected as it does not vary across different IS. The respective component will only come into play when the numbers of alternatives across IS are different.

9 Conclusions

Several approaches to aggregation of alternative rankings provided by multiple individual IS have been described. Applicability of this or that approach depends on the specificity of IS (which can be represented by experts, analysts, search engines, online or paper documents etc.) and on the number of alternatives in the rankings. Several ways of IS weight definition have been suggested, based on heuristic expert decision support and statistical methods. IS weights calculation methods can be applied to aggregation of data, coming from IS of different nature.

Although the approaches, suggested in the paper are, to a large extent, heuristic, they are based on the fundamental assumption, that during aggregation of information, coming form different sources, relative weights of IS should depend on quality and quantity of information, provided by these sources and, beside that, reflect expert estimate of their credibility, based on previous experience of their usage.

Further research envisions experimental study of the suggested methods.

This paper is prepared as part of project #F73/23558 “Development of Decision-making Support Methods and Means for Detection of Information Operations”. The project won the contest #F73 for grant support of scientific research projects held by The State Fund for Fundamental Research of Ukraine and Belarusian Republican Foundation for Fundamental Research.

References

1. Saaty , T.L. , 1994 . Fundamentals of Decision Making and Priority Theory with The Analytic Hierarchy Process . / T.L.Saaty; RWS Publications, Pittsburgh PA , 204220 .

3. Kadenko S.V. Prospects and Potential of Expert Decision-making Support Techniques Implementation in Information Security Area / S .V.Kadenko // CEUR Workshop Proceedings - 2016 - pp. 8 - 14 .

4. Rzążewski , K. Każdy głos się liczy! Wędrówka przez krainę wyborów / K.Rzążewski, W. Słomczyński , K. Życzkowski - Wydawnictwo Sejmowe Warszawa 2014 - 420 p.

5. J. C. Borda . Memoire sur les elections au scrutin . Histoire de l' Academie Royale des Sciences , 1781 .

6. M.-J. Condorcet . Essai sur l'application de l'analyse a la probabilite des decisions rendues a la pluralite des voix , 1785 .

7. Arrow , K. J. Social Choice and Individual Values / K.J. Arrow - 2nd edition . Yale University Press - 1970 - 144 p.

8. Kemeny , J. G. Mathematics without Numbers / John G. Kemeny // Daedalus Vol. 88 , No. 4, Quantity and Quality (Fall, 1959 ), pp. 577 - 591 .

9. Copeland A. H. A reasonable social welfare function. / A. H. Copeland . Mimeo, University of Michigan, 1951 .

10. Aleskerov , Fuad T. Arrovian Aggregation Models / Fuad T. Aleskerov. Springer, 1999 - 226 p.

11. Dwork C . Rank aggregation methods for the Web / C. Dwork , R. Kumar , M. Naor , D. Sivakumar // WWW '01 Proceedings of the 10th international conference on World Wide Web - 2001 - Pages 613-622.

12. Renda M.E. Web metasearch: rank vs. score based rank aggregation methods / M.E . Renda, U. Straccia // Proceedings of the 2003 ACM symposium on Applied computing - 2003 - Pages 841-846.

14. Totsenko V.G. Method of Determination of Group Multicriteria Ordinal Estimates with Account of Expert Competence / V.G. Totsenko // Journal of Automation and Information Sciences . - 2005 . - vol. 37 . - N10 . - P. 19 - 23 .

15. Kadenko S. V Defining the Relative Weights of the Alternatives Estimation Criteria Based on Clear and Fuzzy Rankings / S.V. Kadenko // Journal of Information and Automation Sciences - 2013 - vol. 45 , i . 2 - pp. 41 - 49 .

16. Kadenko S .V. Determination of Parameters of Criteria of "Tree" Type Hierarchy on the Basis of Ordinal Estimates / S .V.Kadenko // Journal of Information and Automation Sciences - 2008 - vol. 40 , i .8 - pp. 7 - 15 .

17. Ramanathan , R. Group Preference Aggregation Methods Employed in AHP: An Evaluation and Intrinsic Process for Deriving Members ' Weightages / R.Ramanathan, L.S.Ganesh// European Journal of Operational Research 79 - 1994 , pp. 249 - 265 .

18. Forman , E. Aggregating individual judgments and priorities with the analytic hierarchy process/ Forman , E. & Peniwati , K // European Jouranl of Operational Research, Vol. 108 , 1998 - pp. 131 - 145

19. Yang Q , Du P -a, Wang Y , Liang B . A rough set approach for determining weights of decision makers in group decision making . PLoS ONE 12 ( 2 ) - 2017: e0172679 . doi: 10 . 1371

22. Tsyganok V.V. On Sufficiency of the Consistency Level of Group Ordinal Estimates / V.V. Tsyganok , S.V.Kadenko // Journal of Automation and Information Sciences . - 2010 . - v. 42 , issue 8.- p. 42 - 47 .

23. Kadenko S.V. A Method for Improving the Consistency of Individual Expert Rankings during Their Aggregation / S .V. Kadenko , V.V.Tsyganok // Journal of Automation and Information Sciences . - 2012 . - v. 44 , issue 4. - p. 23 - 31 .

25. Tsyganok V.V. Simulation of Expert Judgements for Testing the Methods of Information Processing in DecisionMaking Support Systems / V.V.Tsyganok., S.V. Kadenko , O.V. Andriichuk // Journal of Automation and Information Sciences . - 2011 . - v. 43 , issue 12. - p. 21 - 32 .

26. Tsyganok , V. Significance of expert competence consideration in group decision making using AHP . / V. Tsyganok, S. Kadenko , O.Andriichuk// International Journal of Production Research . Vol. 50 , Issue 17 , 2012 - pp. 4785 - 4792

28. Totsenko V.G. Matching and Aggregation of Experts Estimates Taking into Account Experts' Competence while Group Estimation of Alternatives for Decision Making Support / V.G. Totsenko // Journal of Automation and Information Sciences . - 2002 .- v. 34 . - No.8.

29. Tsyganok , V. Investigation of the aggregation effectiveness of expert estimates obtained by the pairwise comparison method / V.Tsyganok // Mathematical and Computer Modeling, 52 ( 3-4 ) - 2010 - pp. 538 - 544 .

30. Olenko Andriy & Tsyganok Vitaliy Double Entropy Inter-Rater Agreement Indices. Applied Psychological Measurement . - 2016 . - v. 40 ( 1 ). - P. 37 - 55 .