=Paper=
{{Paper
|id=Vol-2067/paper19
|storemode=property
|title=Considering Importance of Information Sources during Aggregation of Alternative Rankings
|pdfUrl=https://ceur-ws.org/Vol-2067/paper19.pdf
|volume=Vol-2067
|authors=Vitaliy V. Tsyganok,Sergii V. Kadenko,Oleh V. Andriichuk
|dblpUrl=https://dblp.org/rec/conf/its2/TsyganokKA17
}}
==Considering Importance of Information Sources during Aggregation of Alternative Rankings==
<pdf width="1500px">https://ceur-ws.org/Vol-2067/paper19.pdf</pdf>
<pre>
     Considering Importance of Information Sources during
             Aggregation of Alternative Rankings
     © Vitaliy V. Tsyganok               © Sergii V. Kadenko        © Oleh V. Andriichuk
        Institute for Information Recording of National Academy of Sciences of Ukraine,
                                          Kyiv, Ukraine
     tsyganok@ipri.kiev.ua             seriga2009@gmail.com          oleh.andriichuk@i.ua

                                                      Abstract
         The paper outlines several approaches to aggregation of rankings of alternatives, provided by a set of
individual information sources (IS). It is shown, that, depending on dimensionality of the set of alternatives,
different aggregation methods can be applied. Particular attention is given to weighted aggregation of alternative
rankings provided by different IS. Some intuitive assumptions concerning IS weight calculation are set forth. It is
assumed that IS weight should reflect both quantity and quality of information it provides, as well as expert estimate
of IS credibility based on previous experience of IS usage. Based on these assumptions, expressions for IS weight
calculation are suggested. The approaches, suggested in the paper can be applied to information, provided by
sources of different nature, i.e., experts, search engines, paper and online documents, etc.
         Keywords: alternative ranking, information source, rank ordering, expert competence, relative weight,
aggregate ranking.

                                                  1 Introduction
           In the process of informational and analytic research a common problem, often faced by analysts is
compiling a ranking of a set of objects or alternatives (products, electoral candidates, political parties etc) according
to some criterion. This criterion may be explicitly formulated by a decision-maker, or presented in the form of a
particular topical query (such as “top comic actors”, “best online footwear stores”, “downtown restaurants”, etc).
Even these randomly selected formulations demonstrate that it is problematic to select these alternatives, satisfying
the respective queries based on solely numeric data, as there is no quantitative criterion, according to which the
alternatives could be measured. A common approach, summarized in the most explicit way, perhaps, by Tom
Saaty ([1]), is as follows: if you cannot measure the alternatives, the best way to numerically describe them, is to
compare them among themselves. Conceptually, comparisons of alternatives according to some pre-defined criterion
can be classified as either cardinal or ordinal estimates. Cardinal pair-wise comparisons of alternatives bear
information about numeric relation between them according to the given criterion, while ordinal comparisons
indicate only the ordering of these alternatives. Cardinal pair-wise comparisons are good for relatively small
numbers of alternatives, lying within the same order of magnitude. Ordinal pair-wise comparisons are easier to
obtain and process and they can be used to compile rankings of large numbers of alternatives.
           Amount of data, used in informational and analytical research is, usually, large, and that is why ordering of
alternatives is more frequently used. For example, this is one of the possible reasons why web search engines
operate mostly with rankings (orderings) of references and not with their numerically expressed ratings of some
kind.
           In order to compile a ranking of alternatives based on all available information, rankings, coming from
several information sources, need to be taken into consideration and aggregated in some way. Authors of [2] show
that information space is influenced by multiple factors of qualitative nature, which are difficult to formalize and
describe in numeric terms. Based on these characteristics, it was shown in [3] that information space was a typical
example of a weakly structured subject domain. In order to provide at least some analytical description of a weakly
structured domain, data, coming from any available sources, needs to be taken into consideration. In this paper we
will try to address the most general case, when information sources can include paper and online documentation,
web search results, and expert judgments.
           The problem with aggregation of data, coming from different information sources is that, in the general
case, these sources have different weights, depending, again, on several factors. For example, it is reasonable to
assume, that more credible information source, providing larger amount of information, should be assigned greater
weight prior to data aggregation. While there is, roughly, a dozen of methods of ranking aggregation (proposed by
Borda, Condorcet, Kemeny, and others), that are commonly known and described in a multitude of publications, the
problem of defining the weights of data sources (voters, judges, experts, search engines etc.) during data aggregation
is still relevant and open to discussion.
           So, in further sections of our paper, we are going to set forth a method, allowing to aggregate data,
presented in the form of alternative rankings, coming from different information sources, particularly focusing on
different aspects of information source weight definition problem.

                                                          132
                                              2 Literature review
          The problem of rank aggregation became relevant back in the ancient times, when democratic societies and
voting systems started to emerge. An extensive analysis of voting rules and their evolution is provided, for example,
in [4].
          The first rank aggregation methods, that emerged in the process of democratic voting evolution, which are
still relevant today, were suggested in the late 18th century by Borda ([5]) and Condorcet ([6]). Since then multiple
approaches to aggregation of individual preferences using different social welfare functions were devised. We find it
most appropriate to mention the period of 1950-s and 1960-s – the time when Arrow’s impossibility theorem was
formulated (see [7]) and attempts were made to bypass its constraints (particularly, by Kemeny [8] and Copeland
[9]). F.Aleskerov in [10] summarizes Arrovian aggregation rules in his book.
          In the 2000s, with growing popularity of online information search engines, the problem of rank
aggregation became even more relevant, as it became evident, that beside data obtained from voters, experts, and
analysts, it was necessary to take online information sources into account (sometimes, with minimum human
involvement). In this context, we should mention several papers from the 2000s, particularly [11-13]. In these
publications the authors suggest and compare several approaches, targeted at aggregation of data from online
information sources.
          Definition of weights of information sources and weighted aggregation of rankings represents a separate
matter. Methods of Borda and Condorcet can be easily extrapolated to the case when information sources have
different weights, as shown by Totsenko in [14]. Kemeny’s median is a more problematic case when information
sources have different weights. Ordinal factorial analysis methods, described in [15, 16] allow analysts to calculate
weights based on available sets of alternative rankings, previously provided by experts. This approach can be
extrapolated to calculation of weights of online information sources as well, if evaluators provide some global
alternative ranking a priori.
          Dwork et al in [11] stress the importance of involvement of human evaluators in the process of preliminary
information source weight definition. However, their paper focuses mostly on comparative analysis of rank
aggregation methods and does not address the problem of IS weight calculation directly.
          As for IS weights, we can, again, mention Tom Saaty ([1]) and his academic school (including
Ramanathan & Ganesh [17], Forman & Peniwati [18], as well as Yang et al [19]). However, their methods are
primarily targeted at aggregation of expert estimates, represented in the form of pair-wise comparison matrices
(PCM), provided in some ratio scales (and not rankings).
          In our current paper we are going to suggest information source weight calculation methods based on both
preliminary human evaluation of information source credibility and amount and quality of information, provided by
a given information source.

                                              3 The problem statement
          What is given: n information sources (IS1 – ISn). Each of IS provides a ranking Ri, that includes mi
alternatives (i=1..n). Information sources can be experts, search engines, analysts who monitor online content, paper
or online documents, etc. We should stress, that the number of alternatives, provided by different information
sources is, in the general case, different ( mi  m j ; i, j  1..n ).
        We should find an aggregated (global) ranking of alternatives.

                    4 Solution ideas for the case of equal weights of information sources

         We suggest building a solution algorithm based on available methods of aggregation of individual rankings
(ordinal estimates). The most common ranking aggregation methods were proposed by Borda, Condorcet and,
perhaps, Kemeny. We should stress that if we are dealing with experts, the number of alternatives an expert can
analyze “in one go” is no more than 7±2. However, search engines can return rankings of hundreds of links. That is
why, usage of domination matrix – based methods is not recommended for such dimensionalities. For example, if
we are dealing with 10 IS, providing rankings of 1000 alternatives each, then we have to analyze and process 10
matrices with 1000×1000 cells. In such cases Borda method seems to be the most appropriate option, as it operates
with ranking vectors and not domination matrices. Besides, it is easier to extrapolate Borda method to the case when
IS have different weights (while, for in stance, extrapolation of Markov chain-based methods looks more
problematic). If all IS have the same weight, we can use the following algorithm.
         1) Find the length of a ranking with all unique alternatives, featured in individual rankings
              n
M  card ({ A(ji ) ; j  1..mi }) .
             i 1




                                                        133
         2) Form a unified set of alternatives. For this purpose we should add to each ranking Ri of mi alternatives
                                    n
all the remaining alternatives     {A ; j  1..m } /{ A ; j  1..m } with rank m +1. During unification of
                                   k 1
                                               (k )
                                               j             k
                                                                   (i )
                                                                   j               i       i


alternatives, provided by different IS, we should check whether alternatives are conceptually the same. If IS are
experts, then for this purpose we can use the methods of semantic similarity determination, suggested in [20].
          3) As a result, we will get M alternatives in each ranking. Each ranking Ri will include mi alternatives with
different ranks and (M – mi ) alternatives with the same rank (mi +1). (A similar approach was mentioned by Dwork
et al in [11]).
          4) Add the ranks of each alternative across all IS.
          5) Sort (order) the alternatives in the order of increment of individual rank sums. This sorting will result in
the rank order that we should find.

                                          n            n
                                         rkj   rlj  rk  rl , k ,l  1..M (1),
                                        j 1          j 1

where rkj и rlj – are the ranks of alternatives Ak and Al in the ranking, provided by IS number j.

        5 Usage of different methods under smaller cardinality of the set of alternatives
          If the number of alternatives amounts to one or two dozens, or if the decision-maker (analyst) is interested
only in the rank order of the first few alternatives, provided by IS, then it makes sense to use other ranking methods
(beside or instead of Borda) in order to aggregate the individual ranking results.
          For example, let us assume, we have 10-15 IS, and each of them provides a ranking of alternatives, from
which we are particularly interested in the top 10-20 items. If all alternatives (items) are different, then we have 300
(1520) different items at most. In fact (especially if we are talking about search engines), many items featured in
individual rankings across different IS are the same. So, if we have 15 IS and each of them provides a ranking of 20
alternatives, then the cardinality of the set of unique items will amount to approximately 100 alternatives.
Consequently, if we use aggregation methods operating with ordinal pair-wise comparison matrices (PCM) (such as
Condorcet rule or Kemeny’s median) and not with ranking vectors (like Borda or Markov Chain based methods), we
will have to analyze 15 square matrices containing 100100 cells. As strict rank order relationship is reciprocally
symmetrical (if alternative А1 ranks higher than А2, then А2 ranks lower than А1), during aggregation of PCM we will
have to analyze just matrix elements above the principal diagonal {dik, i<k}. That is why, even if the aggregate
ranking features 100 alternatives, then we will have to analyze (10099)/2 or 4950 cells in each PCM. These
calculations demonstrate, that under smaller cardinality of alternative set, we can use Condorcet rule and Kemeny’s
median for aggregation of individual ranking results.
          In our view, the disadvantage of Borda method is that during aggregation of individual rankings we are
witnessing an explicit heuristic transition from ordinal preference scale to ratio scale. However, this transition is
inevitable, because representation of alternative estimates in ratio scale is the necessary and sufficient condition of
existence of linear global criterion, allowing us to aggregate these estimates (as proven by Litvak in [21]). For
example, an alternative with rank 2 does not necessarily have to be exactly 2 times better than alternative with rank
4 (we should remind, that dominating alternatives are assigned smaller ranks).
          Condorcet rule, Markov chains, and Kemeny’s method allow us to avoid such an explicit transition from
ranks to ratios.

                                    6 Existing methods: a brief overview
          Condorcet rule. The key idea of Condorcet method is as follows. If alternative Аi dominates over
alternative Ak in the majority of individual rankings, then this preference relationship should be maintained in the
global (aggregate) ranking. In order to get an aggregate ranking we should first build individual ordinal PCMs. If in
some individual ranking Rj alternative Ai dominates over Ak, then the respective element of ordinal PCM equals 1, if
Ai is dominated by Ak – (-1), if the alternatives are equal – 0:

                                                              1, Ai  Ak
                                                        ( j)                (2)
                                                      d ik  0 , Ai  Ak
                                                             1, A  A
                                                              i      k


In order to aggregate rankings, provided by several information sources according to Condorcet’s rule, we should
limit the number of alternatives to be featured in the global ranking by a fixed number M and build ordinal PCM
                                                                 134
based on rankings, provided by all IS. { D ( j )  { d ik( j ) ; i , k  1..M }; j  1..n } . After that we should calculate
the sums of all respective elements of individual PCM and fill ordinal PCM, corresponding to the global rank
ordering relationship (tournament).

                                                                      n ( j)
                                                                     0,  dik  0
                                                                      j 1
                              D  { dik ; i , k  1..M } where        n               (3).
                                                               dik  1,  dik( j )  0
                                                                       j 1
                                                                            n
                                                                                  ( j)
                                                                       1,  d ik  0
                                                                          j 1


After that we should add the elements in each line of the global rank ordering relationship matrix D and rank the
obtained line sums.

                                              M           M
                                              dik   dlk  ri  rl (4),
                                             k 1         l 1


where ri and rl are ranks of alternatives Ai and Al in the global ranking R.
          The disadvantage of Condorcet’s rule (beside dimensionality limitation) is that equal alternative ranks may
emerge and transitivity of the global preference relationship may be violated as a result of the so-called “Condorcet
paradox”. The paradox is one of the specific cases of violation of Arrow’s requirements to social choice functions
[7]. If feedback is acceptable, then transitivity of the global preference relationship can be achieved if we use
algorithms, described in [22, 23].
          In the context of aggregation of results of online data search, provided by several engines, the advantage of
Condorcet’s rule (mentioned by Dwork et al in [11]) is its ability to filter spam content.
          A more “just” aggregation rule in view of Arrow’s impossibility theorem is Kemeny’s rule (sometimes
called Kemeny’s median).
          Kemeny’s median can be considered the analogue of average mean for ordinal (non-cardinal) estimates.
          As ordinal estimates (or ranks) do not bear information on quantitative relation between alternatives, such
concepts as Euclidian distance metric or center of mass (center of gravity) do not apply to rank order vectors.
          Distance between two rankings of a given set of alternatives depends upon the number of elementary
permutations that is needed to obtain one ranking from the other. Kemeny’s distance between two rankings is based
on Haming metric. If we have two rankings R1, R2 of a set of alternatives A={A1..Am}, and respective domination
matrices D1 and D2 are built based on these rankings according to formula (2), then Kemeny’s distance K is
calculated as follows.

                                                      m m
                                     K ( R1,R2 )    dik( 1 )  d ik( 2 )          (5),
                                                      i 1 k 1

        If we have a set of n rankings {R1..Rn} of alternatives A={A1..Am}, then, by definition ([8]), Kemeny’s
median of this set of rankings is a ranking R:

                                                                 n
                                         R  arg min  K ( R , R j )          (6),
                                                    RT       j 1

where T is a set of all possible rankings of alternatives. Kemeny’s median calculation procedure is very labor-
intensive. Some estimates of complexity of this problem are provided in [11]. One of the “simplest” algorithms of
Kemeny’s median calculation is provided by Litvak in [21].

  7 Generalization of suggested approaches to the case of different weights of information
                                         sources
         Let us now assume that IS weights {w1..wn} (reflecting their credibility) have been provided by experts,
calculated based on previous estimation experience (as described in [15, 16]), or obtained in some other way. In this
case, respective formulas for ranking aggregation will change, as all rankings, provided by individual IS, will be

                                                                 135
taken into consideration together with their respective weights. Formula (1) (corresponding to Borda aggregation
method) will look as follows.

                                  n              n
                                  w j rkj   w j rlj  rk  rl , k ,l  1..M (7)
                                  j 1          j 1


Formula (3) (corresponding to Condorcet method) will look as follows.

                                                                   n           ( j)
                                                                  0,  w j dik  0
                                                                   j 1
                           D  { dik ; i , k  1..M } where        n                   (8)
                                                            dik  1,  w j dik( j )  0
                                                                    j 1
                                                                         n
                                                                                   ( j)
                                                                    1,  w j dik  0
                                                                    j 1

        In formula (6) weight coefficients will become multipliers for respective Kemeny distance values.

                                                          n
                                         R  arg min  w j K ( R , R j ) (9).
                                                 RT     j 1


         Extrapolation of Markov chain based methods, described in [11], represents a problem that should be
addressed in a separate research.
         Experience and numerous publications indicate that when a group of experts participates in decision-
making process, individual expert competence should be taken into consideration (providing, the expert group is
relatively small) (for details – see [24-26]). Similarly, if several IS are used to build a ranking of alternatives
according to their relevance in terms of a particular information query or in the process of some analytical research,
we should take their weights into account as well.
         Now let us address the issue of calculation of IS weights in greater detail. So far we have come up with
several conceptual approaches to IS weight calculation. These approaches are outlined in the next section.

           8 Approaches to calculation of the relative weights of information sources
         1) Experience-based approach was set forth in [15, 16]. In essence, experts or evaluators provide their
ranking of alternatives and then, based on their ranking and rankings, provided by individual IS, the weights of IS
are calculated (under assumption that rankings are aggregated using Borda or Condorcet rule).
         2) Definition of relative IS weights based on quantity and quality of provided information. It is
reasonable to assume, that the relative weight of an IS should depend on quantity and quality of information,
provided by the source in terms of every information query (in the given subject domain). Criteria, representing
quality and quantity of information search, following a given query can be formulated as follows.
        Number of alternatives (references or links in case of online information search), provided by the IS in
response to a particular query;
        Relevance of these alternatives (references, links), i.e. their correspondence to the specific query.
         Relevance of the results of functioning of an IS can be defined based on the estimates of users (evaluators,
experts) from the given subject domain. These estimates of IS can be based on previous experience of information
search queries.
         Relevance of results of IS work can be considered in terms of search queries of some particular type (search
for information in some given language, formulas, images etc.). If it is possible to outline some query type, expert
estimate will depend upon IS capability of processing this particular type of queries. Otherwise (when it is
problematic to outline specific query types) it makes sense to use an average value of expert estimate (across
different query types).
         In the context of this subsection we suggest calculating IS weight using the approach, similar to methods,
used by Totsenko for calculation of expert competence in [27, 28]. In accordance to this approach, expert
competence should depend on self-estimate, objective estimate, and mutual estimate.

                                         k  s ( x1b  x2v ) ,             (10)


                                                              136
where k is the relative competence of the expert, s is self-estimate, b is an objective component, v is mutual estimate
of expert group members, while x1 , x2 represent the coefficients of relative importance of objective and mutual
estimates’ respectively.
         When the weights of “generalized” IS (i.e. experts, search engines, documents, etc.) are calculated, mutual
estimate can be replaced by the ration between the number of alternatives, provided by the i-th IS in relation to the
total number of alternatives, provided by all IS Vi :

                                                     mi             ,         (11)
                                            Vi  n
                                                   m
                                                   k 1
                                                            k




where mi is the number of alternatives, provided by i-th IS, n is the total number of information sources.
          Self-estimate in formula (10) can be replaced by the value of expert estimate Ei . This estimate depends on
previous experience of IS usage. If weights of several IS are estimated by several experts through pair-wise
comparisons of the IS, then expert estimates can be aggregated using combinatorial method, described in [29]. If
expert estimates are not consistent enough for aggregation, then consistency should be improved through feedback
with experts, based on spectral approach, as described in [30]. For each particular query type (for instance,
Ukrainian, English, image, formula, other) we should apply the respective average (aggregate) value of Ei .
Definition of query types is not the subject of this paper and should be addressed separately.
          Objective estimate in formula (10) can be replaced by the ratio between the number of alternatives,
provided by i-th IS and the total number of unique alternatives, provided by all IS Oi (thus, it will reflect the ability
of IS to provide unique information).

                                                    mi
                                            Oi        ,                      (12)
                                                    P
                                                                                             n
where P is the number of unique alternatives, provided by all IS, that is P  card (        { A ; j  1..m }) . If IS are
                                                                                            i 1
                                                                                                   (i )
                                                                                                   j         i

experts, semantic similarity methods, suggested in [20] can be used to define whether the alternatives are relevant in
the context of the given query.
         Similarly to expert competence, weight of IS will be calculated as follows.

                                        wi*  Ei ( x1Oi  x2Vi ) ,                   (13)

          *
where wi is the non-normalized IS weight, , x1 , x2 are the respective coefficients of components’ relative
importance, and Ei is the non-normalized expert estimate value. Normalized values are calculated as follows.

                                                          wi* ,               (14)
                                             wi      n
                                                                *
                                                    w
                                                     i 1
                                                                i




where   w is the normalized relative IS weight.
          i
          We propose to follow the assumption that “objective” and “mutual” IS estimates should form a convex
combination ( x1  x2  1 ), so x1 and x2 should lie within the [0;1] range and vary depending on particular
information query.
          As a result of each information search, every IS provides several alternatives (in the general case they are
different, as mentioned above). Figure 1 represents an example, where 5 IS provide 9 unique alternatives. 1st and 6th
alternatives were provided by 2 IS, 2nd and 5th – by 3 IS, 3rd alternative – by 5 IS, while other alternatives were
provided by 1 IS each.




                                                                        137
                        Fig. 1 Example of a unified set of alternatives, provided by several IS

         In order to illustrate the behavior of dependence of x1 and x2 on information search results, let us consider
several extreme cases.
         Extreme case 1
         Let us assume that all IS provided completely different alternative sets. Generalized information search
results are displayed on Fig. 2: 5 IS provide 9 different alternatives and each of the alternatives is provided by a
single IS (for instance, IS1: a1, a2; IS2: a3, a4; IS3: a5; IS4: a6, a7, a8; IS5: a9).




           Fig. 2 Example of a unified set of alternatives, where each alternative is provided by a single IS

If all IS provide different alternatives in response to some query, the relevance of these alternatives seems
questionable. In such cases we suggest using expert estimate (based on previous experience of IS usage) as the key
component of IS weight, as it reflects the experience of previous information search sessions and the ability of IS to
find new information.
          Extreme case 2
          Let us assume, that all IS provided the same alternatives. An example of a hypothetical information search
results are presented on fig. 3: 5 9 alternatives are provided by all 5 IS (although the order, in which alternatives are
ranked by different sources, may be different).


                                                          138
              Fig. 3 Example of a unified set of alternatives, where each alternative is provided by all IS

          In such a case, IS weights should be equal to Ei , because all IS are equally efficient in providing
alternatives in response to information query. Their weights can be defined, again, only based on the previous
history of information search sessions that is reflected by the expert estimate.
          In order to devise a formal expression for x1 and x2 , based on these intuitive considerations, let us
introduce the indicator  , characterizing the alternative frequency function or density of representation of
alternatives across all information sources.

                                                    P

                                                   h
                                                   j 1
                                                            j

                                                              ,         (15)
                                                    nP

where h j is the quantity of IS, that provided j-th alternative, while n is the total quantity of IS.

In “extreme case 1”  assumes the minimum value   1 .
                                                    n
In “extreme case 2”  assumes maximum values   1 .

In view of above-mentioned convexity requirements, we suggest calculating x2 as

                                         x2   ; x1  1  x2 .                  (16)

                                                                                              1     1
                                                                                   Ei ((1  )Oi  Vi )
In “extreme case 1” the normalized relative IS weight assumes the value of                    n     n       , and in
                                                                           wi  n               1     1
                                                                               j 1
                                                                                     E j ((1  )O j  V j )
                                                                                                n     n
“extreme case 2” it assumes the value of    w E.
                                               i        i
        3) Statistical approach represents a modification of the previous approach. In order to take the quantity of
information, provided by the IS (“objective” weight component in formula (13)) into account, we can normalize the
numbers of alternatives across all IS (see formula (11)) and then calculate the importance of the respective
component (i.e. value of x1) as dispersion if this indicator (normalized number of alternatives).

                                              x1  D(V ); x2  1  x1 (17)


                                                                    139
In this case, if all IS provide the same number of alternatives, the respective component of their relative weights can
be neglected as it does not vary across different IS. The respective component will only come into play when the
numbers of alternatives across IS are different.

                                                    9 Conclusions
         Several approaches to aggregation of alternative rankings provided by multiple individual IS have been
described. Applicability of this or that approach depends on the specificity of IS (which can be represented by
experts, analysts, search engines, online or paper documents etc.) and on the number of alternatives in the rankings.
Several ways of IS weight definition have been suggested, based on heuristic expert decision support and statistical
methods. IS weights calculation methods can be applied to aggregation of data, coming from IS of different nature.
         Although the approaches, suggested in the paper are, to a large extent, heuristic, they are based on the
fundamental assumption, that during aggregation of information, coming form different sources, relative weights of
IS should depend on quality and quantity of information, provided by these sources and, beside that, reflect expert
estimate of their credibility, based on previous experience of their usage.
         Further research envisions experimental study of the suggested methods.

          This paper is prepared as part of project #F73/23558 “Development of Decision-making Support Methods
and Means for Detection of Information Operations”. The project won the contest #F73 for grant support of
scientific research projects held by The State Fund for Fundamental Research of Ukraine and Belarusian Republican
Foundation for Fundamental Research.

                                                      References
     1. Saaty, T.L., 1994. Fundamentals of Decision Making and Priority Theory with The Analytic Hierarchy Process. /
         T.L.Saaty; RWS Publications, Pittsburgh PA, 204220.
     2. Горбулін В.П., Додонов О.Г., Ланде Д.В. Інформаційні операції та безпека суспільства: загрози, протидія,
         моделювання: монографія – К., Інтертехнологія, 2009 – 164 с.
     3. Kadenko S.V. Prospects and Potential of Expert Decision-making Support Techniques Implementation in
         Information Security Area / S.V.Kadenko // CEUR Workshop Proceedings – 2016 – pp. 8-14.
     4. Rzążewski, K. Każdy głos się liczy! Wędrówka przez krainę wyborów / K.Rzążewski, W.Słomczyński,
         K.Życzkowski - Wydawnictwo Sejmowe Warszawa 2014 – 420 p.
     5. J. C. Borda. Memoire sur les elections au scrutin. Histoire de l'Academie Royale des Sciences, 1781.
     6. M.-J. Condorcet. Essai sur l'application de l'analyse a la probabilite des decisions rendues a la pluralite des voix,
         1785.
     7. Arrow, K. J. Social Choice and Individual Values / K.J. Arrow – 2nd edition. Yale University Press – 1970 – 144
         p.
     8. Kemeny, J. G. Mathematics without Numbers / John G. Kemeny // Daedalus Vol. 88, No. 4, Quantity and
         Quality (Fall, 1959), pp. 577-591.
     9. Copeland A. H. A reasonable social welfare function. / A. H. Copeland. Mimeo, University of Michigan, 1951.
     10. Aleskerov, Fuad T. Arrovian Aggregation Models / Fuad T. Aleskerov. Springer, 1999 – 226 p.
     11. Dwork C. Rank aggregation methods for the Web / C. Dwork, R. Kumar, M. Naor, D. Sivakumar // WWW '01
         Proceedings of the 10th international conference on World Wide Web – 2001 – Pages 613-622.
     12. Renda M.E. Web metasearch: rank vs. score based rank aggregation methods / M.E. Renda, U. Straccia //
         Proceedings of the 2003 ACM symposium on Applied computing – 2003 – Pages 841-846.
     13. Аборок, Ш.К. Метапоисковая система с использованием показателя обобщенной релевантности
         документа для формирования результатов поиска / Ш.К. Аборок, В.И. Костин // Информатика и
         компьютерные технологии-2005: сб. тр. 1-ой Междунар. студ. науч.-техн. конф., 15 дек. 2005 г. - Донецк,
         2006. - С. 268-269.
     14. Totsenko V.G. Method of Determination of Group Multicriteria Ordinal Estimates with Account of Expert
         Competence / V.G. Totsenko // Journal of Automation and Information Sciences. – 2005. – vol. 37. -N10. - P.19-
         23.
     15. Kadenko S.V Defining the Relative Weights of the Alternatives Estimation Criteria Based on Clear and Fuzzy
         Rankings / S.V. Kadenko // Journal of Information and Automation Sciences – 2013 – vol. 45, i. 2 – pp. 41-49.
     16. Kadenko S.V. Determination of Parameters of Criteria of "Tree" Type Hierarchy on the Basis of Ordinal
         Estimates / S.V.Kadenko // Journal of Information and Automation Sciences – 2008 – vol. 40, i.8 – pp. 7-15.
     17. Ramanathan, R. Group Preference Aggregation Methods Employed in AHP: An Evaluation and Intrinsic Process
         for Deriving Members’ Weightages / R.Ramanathan, L.S.Ganesh// European Journal of Operational Research 79
         – 1994, pp. 249-265.
     18. Forman, E.Aggregating individual judgments and priorities with the analytic hierarchy process/ Forman, E. &
         Peniwati, K // European Jouranl of Operational Research, Vol. 108, 1998 – pp.131-145
     19. Yang Q, Du P-a, Wang Y, Liang B. A rough set approach for determining weights of decision makers in group
         decision making. PLoS ONE 12(2) – 2017: e0172679. doi:10.1371


                                                            140
20. Андрійчук О.В. Метод змістової ідентифікації об’єктів баз знань систем підтримки прийняття рішень /
    О.В. Андрійчук // Реєстрація, зберігання і обробка даних. — 2014. — Т. 16, № 1. — С. 65-78.
21. Литвак Б. Г. Экспертная информация. Методы получения и анализа / Б. Г. Литвак. – М.: Радио и связь,
    1982. – 185 с.
22. Tsyganok V.V. On Sufficiency of the Consistency Level of Group Ordinal Estimates / V.V.Tsyganok,
    S.V.Kadenko // Journal of Automation and Information Sciences. – 2010. – v.42, issue 8.– p.42-47.
23. Kadenko S.V. A Method for Improving the Consistency of Individual Expert Rankings during Their Aggregation
    / S.V.Kadenko, V.V.Tsyganok // Journal of Automation and Information Sciences. – 2012. – v.44, issue 4. –
    p. 23-31.
24. Циганок В.В. Урахування компетентності експертів при визначенні групового ранжирування / В.В.
    Циганок, О.В. Андрійчук // Реєстрація, зберігання і обробка даних. – 2011. – т.13. – №1. С. 94-105.
25. Tsyganok V.V. Simulation of Expert Judgements for Testing the Methods of Information Processing in Decision-
    Making Support Systems / V.V.Tsyganok., S.V.Kadenko, O.V.Andriichuk // Journal of Automation and
    Information Sciences. – 2011. – v.43, issue 12. – p.21-32.
26. Tsyganok, V. Significance of expert competence consideration in group decision making using AHP. / V.
    Tsyganok, S. Kadenko, O.Andriichuk//International Journal of Production Research. Vol. 50, Issue 17, 2012 –
    pp. 4785-4792
27. Тоценко В.Г. Методы и системы поддержки принятия решений. Алгоритмический аспект / В.Г. Тоценко.
    – К.: Наук. думка, 2002. – 382 с.
28. Totsenko V.G. Matching and Aggregation of Experts Estimates Taking into Account Experts' Competence while
    Group Estimation of Alternatives for Decision Making Support / V.G. Totsenko // Journal of Automation and
    Information Sciences. – 2002.– v.34. – No.8.
29. Tsyganok, V. Investigation of the aggregation effectiveness of expert estimates obtained by the pairwise
    comparison method / V.Tsyganok // Mathematical and Computer Modeling, 52(3-4) – 2010 – pp. 538–544.
30. Olenko Andriy & Tsyganok Vitaliy Double Entropy Inter-Rater Agreement Indices. Applied Psychological
    Measurement. – 2016. – v.40(1). – P.37-55.




                                                   141

</pre>