Abstract

The paper is dedicated to experimental comparison of several incomplete ranking-dependent expert estimation methods (best-worst, best-second best, and an original method of maximum difference). Such methods allow decision-makers to mitigate the impact of various cognitive biases upon expert estimation process and reduce the number of times the experts need to be addressed in order to provide a complete preference structure on the set of estimated objects. Results of such comparison will allow us to define, which ranking-dependent is more stable, i.e. resilient against expert errors.

Keywords

• eol>ranking
• expert estimate
• pair-wise comparison matrix
• priority vector
• genetic algorithm

1. Introduction

PC comparison methods represent a separate group [ 9 ]. These methods have also been the subject of in-depth research during recent years [ 9-11 ].

2. Research Objectives

While incomplete PC methods have been compared according to their stability to expert errors (for example in [ 12 ]), there has been no separate comparative studies dedicated solely to PC methods, taking rankings into consideration.

The concept of experiments allowing us to compare PC methods in terms of resilience against expert errors, has been available for quite a while now [ 12 ]. This concept is defined by the specificity of weakly formalizable subject domains. As we have mentioned, criteria, describing these domains, are mostly intangible ones, they often do not have benchmark values or measurement units. So, expert estimates often become the most reliable source of information in such domains. At the same time, involving sufficient numbers of actual experts in the experiment would require considerable resources. Therefore, experimental studies in which different expert estimation methods are compared, are usually based on simulation of the whole expert estimation process.

We should note that in [ 8 ] PC methods, using different amounts of ordinal information were compared (including best-worst, best-random, 3(-quasi)-regular-graphs) according to both cardinal and ordinal stability indicators. The issue of legitimacy of such “mixed” comparative studies is still

The main task of our present study is to try to compare three ranking-dependent PC methods (best-worst [ 13, 14 ], TOP 2 (best-second best) [ 8 ], and max difference [ 11, 15 ]) according to stability 6

3 1 1 4

7 5 3. Solution Idea compared.

6 3 4

2 5 In [ 16 ] we have defined the conditions under which the three listed PC methods could be We should remind, that if the ranking of compared objects is available, then the best-worst method, obviously, suggests comparing all the objects only to the best and the worst one. Respectively, the best-second best or TOP 2, method suggests comparing all the objects to the best and the second-best ones.

Graphs representing preference structures in best-worst and TOP 2 methods are shown on Fig. 1. Object sizes on Fig. 1 indicate their positions in the ranking.

7 1 2 1 ⎠ ⎝

As we’ve shown in [ 16 ], both methods are incomplete and the number of PC in both methods is 2 − 3. In [ 16 ] we refer to max difference method as “queues” method, but now we feel that “max difference” is a more adequate name (as the most distant object pairs are compared in the first place). Max difference method implies a certain sequence of PC, which is defined by the initial ranking of objects. Let the ranking of objects look as follows: > > ⋯ > , where is the object with rank , = 1, , and is the total number of objects. In this case, the sequence of PCs, which, according to [ 11 ], ensures the most adequate and consistent estimation results, is as follows: batch 1: ( , ) (ranks differ by ( − 1)); batch 2: ( , ) or ( , ) (ranks differ by ( − 2)); batch 3: ( , ) or ( , ) or ( , ) (ranks differ by ( − 3));… ; batch ( − 1): ( , ) or ( , ) or … or ( , ) (ranks differ by 1). The PCM is filled starting from the top right corner. PCM elements lying above the principal diagonal are filled the last. An example of PCM filling for = 5 is shown below: ⎛ ⎜ ⎝ 1

In [ 11, 16 ] we have shown that it is not necessary to fill all the PCM in max difference method. The main task is to ensure the connectivity of the preference structure. PCs in max difference method are performed in batches. Within a batch all PCs are equally important and informative. However, first we should look at PCs, which are required to ensure the connectivity. For instance, if is odd, then in order to get a connected preference graph, we need to perform or ( − 1 )⁄2 of PC batches + 1 PC from the batch number (( − 1 )⁄2) + 1. This is a PC ( , ), i. e. comparison of the 1st element and the “central” one in the ranking. Alternatively, this can be a PC of the last and the central elements ( , ). Once the “central” element is included in the preference structure, the structure becomes connected. If is even, then there are 2 central elements in the ranking, so, to achieve connectivity, we need to perform

− 1 comparison batches + 2 PCs from batch number ( ⁄2): ( , ), ( , ).

6 3 4

2 5 7 1

An example of a connected PC structure on the set of 7 objects is shown on Fig. 2. Batch 1 includes the only PC ( , ), batch 2 – PCs {( , ), ( , )}, batch 3 – PCs {( , ), ( , ), ( , )}, batch 4 – PCs {( , ), ( , ), ( , ), ( , )}, while connectivity is achieved after PC ( , ) or ( , ).

By definition, the number of a batch equals the number of PCs in it. So, the general number of PCs in batches 1 to is a sum of an arithmetic progression: = ( + 1 )⁄2. Therefore, to ensure connectivity of a set of PCs of compared objects, we need to perform the following number of PCs: = need to perform (2 − 3) PCs. So, to place all three listed methods in equal conditions for comparison, we need to ensure that the number of PCs performed in each method is the same. Therefore, condition

≤ (2 − 3) should be fulfilled. − 1 − 1 4 methods the dependence is a linear function ( = ( )), while for max difference it is a square function ( зв = ( )). Its graphs for odd and even are parabolic.

Required number of comparisons 50 45 40 35 30 25 20 15 10 5 0 2 3 4 5 6 7 8

PCs, we need to keep generating new PCs until their number reaches (2 − 3).

If > 14 , then we suggest starting PC generation process with a basic set, described in [ 11, 15 ], which is a bi-partite spanning tree graph, in which the best object is compared with objects from the 2nd half of the ranking , … ,

, while the worst object is compared with objects from the 1st half of the ranking ( , … ,

). Diameter of such a graph (a union of two stars [ 8, 11 ]) always equals 3, irrespectively of , while its edges represent PCs from the first batches. These properties ensure stability of the respective preference structure to expert errors.

Number of objects connectivity number ( );

2 − 3

Batch number number reaches (2 − 3). A simplified flow chart of PC generation algorithm is shown on Fig. 5.

Presently, it is problematic to find a universal algorithm for generating PCs in max difference method for every dimensionality . Table 1 illustrates the sequence of PCs in max difference method for ∈ [3. .16].

Comparisons (i.e. compared object numbers and ranks) 3; 4; 5; 6; 7; 8; 9; 10; 12 11; 16 = 2 = 3 = 4 = 5 = 7 = 8 = 11 = = 3 5 7 9 11 13 15 17 19 1 2 1 2 3 1 2 3 4 1 2 3 4 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 6 1 2 3 4 5 (1;2); (2;3); (3;4) *random 2 of these 3 (1;2); (2;3); (3;4); (4;5) *random 1 of these 4 (1;3); (2;4); (3;5); (4;6) *random 3 of these 4 (1;3) (1;4) 21 23 25 27

(1;6); (2;7); (3;8); (4;9); (5;10); (6;11) *at least one of the red ones (critical for connectivity) and random 3 (or 2) of the remaining 4 PCs from this batch (1;12) compared to best-worst and TOP 2 methods using an experiment, in which the whole expert estimation process is simulated, as shown in [ 8, 12, 17 ].

We should note that the psycho-physiological constraints of human mind allow an average expert to compare only op to = (7 ± 2) at a time [ 18, 19 ]. In real problems the number of objects or criteria presented to experts, usually, does not exceed this number. So, in our experiment we are going to focus on this range of dimensionalities.

4. Experiment outline

In order to experimentally compare the three listed methods, we suggest simulating the whole expert estimation process for a given number of objects without involving actual experts. Our experiment will include the following steps: 1. Set initial object weights (priorities) ( , … , ) 2. Build an ideally consistent PCM on their basis { = ; , = 1. . } 3. Build an incomplete PCM according to one of the three listed methods [ 6, 7 ]. 4. Fluctuate the incomplete PCM: { = (1 + )± ; , = 1. . }) 5. Calculate priorities ( , … , ) based in the fluctuated incomplete PCM using the combinatorial spanning tree enumeration method [ 13 ]. 6. Calculate the difference between initial and calculated priorities (object weights) ∆. Smaller difference ∆ indicates better resilience of the respective method to expert errors. In the present research we suggest using the following formula for ∆: = ; ; − 100% (2)

Mathematically, the best way to find maximum ∆ is genetic algorithm (GA) [ 20-22 ]. In terms of the GA, ∆ is the fitness function, while “individuals” are the fluctuated PCMs. These individuals are subjected to cross-breeding and mutations until ∆ stops increasing. A similar approach has been demonstrated, for instance, in [ 12, 17 ].

5. Results and interpretation

On Fig. 6 we can see typical results of comparison of the 3 methods for the case when = 5 objects. Horizontal axis shows PCM fluctuation level, while vertical axis shows maximum deviation of weights, obtained based on fluctuated incomplete PCM, from initial priority values (∆, %). Fig. 7 illustrates a similar experiment for the case when = 7.

As we can see, in these experiments, best-second best (TOP 2) method has a certain advantage over the other two methods in terms of stability. Besides that, larger dimensionality of the experiment leads to larger deviations of the resulting weight vectors from initial values. Additionally, we should note that the larger the “expert error” (PCM fluctuation) grows, the steeper the resulting curve becomes (i.e., as expert error approaches 50%, ∆ increases much faster than under smaller PCM fluctuations). best-worst

Figure 7 An example of dependence of deviation of object weights on the level of PCM fluctuation (%) for best-worst, best-second best, and max difference methods when = 7

Experiments, displayed on Fig. 6 and 7, do not simulate and/or take into account the sequence in which object pairs are presented to experts for comparisons. At the same time, this sequence is critical, especially, for max difference method. Moreover, comparisons of more ordinally distant objects tend to be more informative and accurate [ 11 ]. This property of PCs also resonates with the outcomes of experiments in cognitive psychology performed by Stevens & Galanter [ 23 ] and Tversky & Kahneman [ 24 ]. According to Stevens and Galanter’s study, respondents tend to make greater mistakes while evaluating objects closer to the middle of the ranking. Tversky and Kahneman are famous for outlining the so-called “anchoring bias” [25], which makes evaluators shift their preferences towards either the smallest (lowest-ranking) or the largest (highest-ranking) object in 51 ,001 ,006 ,011 ,016 ,012 ,003 ,040 ,005 ,075 ,100 ,152 ,170 ,202 ,302 ,450 ,700 ,0010 ,1500 ,2000 ,3000 the set. In fact, according to Jafar Rezaei, invention of the best-worst method [ 13, 14 ] was an attempt to overcome this specific bias by helping experts properly calibrate their preference and balance them. Anchoring bias was also addressed in the multi-criteria decision domain by developers of the TOPSIS method [26-28]. At the same time, experiment results illustrated by Fig 6 and 7 above do not reflect any of these cognitive issues.

,001 ,006 ,011 ,016 ,021 ,003 ,040 ,005 ,075 ,100 ,152 ,107 ,220 ,302 ,450 ,700 ,0010 ,1500 ,2000 ,3000

Figure 9 An example of dependence of deviation of object weights on the level of PCM fluctuation (%) for best-worst, best-second best, and max difference methods, when = 7 (modified fluctuation formula)

Since our experiment does not involve real experts, in order to enhance the significance of PC belonging to batches with smaller numbers, we propose to modify the formula for fluctuation of 52 PCM elements (step 4 of the algorithm). At this point of our research, we suggest using the following modified formula for fluctuating the PCM elements:

| | = (1 +

)± ; , = 1. . ; 0 < < 1 (3)

If PCM elements are fluctuated according to this formula (3), results of comparative experiments look as shown on Fig. 8, Fig. 9.

6. Conclusions

We have shown that usage of both complete and incomplete ranking-dependent PC methods can allow decision-makers to reduce the labor-intensity of expert sessions, while increasing the credibility of obtained results.

We have introduced an algorithm for a comparative study of the three different rankingdependent PC methods (best-worst, TOP 2, and max difference) and formulated conditions under which the results of such a study can be deemed credible.

We have conducted the respective simulation-type experiment to compare the three listed ranking-dependent PC methods according to their stability to expert errors. Previous studies indicate that the order (sequence) in which PCs of objects are performed by experts significantly influences the accuracy of estimation results. Particularly, under assumption that comparisons of more ordinally distant objects are more accurate:  

Priority calculation error becomes several times smaller The results of max difference method tend to be slightly more accurate than the results of best-worst and best-second best (TOP 2) methods.

Therefore, ranking-dependent PC methods, indeed, allow us to reduce the number of comparisons the expert has to perform, while improving the accuracy of estimation. Based on obtained experimental results, max difference method is no less credible than best-worst and TOP 2 methods.

During further studies we will search for alternative ways of modeling (simulating) the expert estimation process, allowing us to take both object ranking and PC sequence into consideration.

Declaration on Generative AI

The author(s) have not employed any Generative AI tools. [2] Saaty, T.L. (1980) The Analytic Hierarchy Process. McGraw-Hill, New York. [3] Saaty, T., Vargas, L., & Cahyono, St. (2022). The Analytic Hierarchy Process. Springer. [4] Wedley, W. C. (2009) Fewer Comparisons – Efficiency via Sufficient Redundancy. Proceedings of the 10th International Symposium on the Analytic Hierarchy/Network Process. Multi-criteria

Decision Making, Pitsburg, Pensilvania, USA, July 30 – August 2, 2009. Available at: http://www.isahp.org/2009Proceedings/Final_Papers/94_Wedley_Fewer_Comparisons_REV_F

IN.pdf. Accessed on May 30, 2025.

[5] Bozóki, S., Fülöp, J., & Rónyai, L. On optimal completion of incomplete pairwise comparison matrices, Mathematical and Computer Modelling, Volume 52, Issues 1–2, 2010, Pages 318-333,

References

[1] Kesavan , E. ( 2024 ). A Review on Ethical and Legal Issues in Artificial Intelligence and Data Privacy . International Journal of Innovations in Science, Engineering and Management . 3 , 4

[6] Bozóki , S. , Csató , L. , & Temesi , J. ( 2016 ). An application of incomplete pairwise comparison matrices for ranking top tennis players . European Journal of Operational Research , 248 ( 1 ), 211 - 218 . https://doi.org/10.1016/j.ejor. 2015 . 06 .069.

[7] Szádoczki , Z. , Bozóki , S. , & Tekile , H. ( 2022 ). Filling in pattern designs for incomplete pairwise comparison matrices: (quasi-)regular graphs with minimal diameter . Omega , 107 . doi: 10 .1016/j.omega. 2021 . 102557 .

[8] Szádoczki , Z. , Bozóki , S. , Juhász , P. et al. ( 2023 ) Incomplete pairwise comparison matrices based on graphs with average degree approximately 3 . Ann Oper Res 326 , 783 - 807 . https://doi.org/10.1007/s10479-022-04819-9.

[9] Kadenko , S. , Tsyganok , V. , Szádoczki , Z. et al. ( 2021 ) Improvement of Pair‐wise Comparison Methods Based on Graph Theory Concepts . CEUR Workshop Proceedings . Vol. 3241 , 46 - 55 .

[10] Andriichuk , O. & Kadenko , S. ( 2022 ). Ranking‐dependent Decision Support Methods for Weakly‐structured Domains . CEUR. Workshop Proceedings , Vol. 3503 , 32 - 41 .

[11] Andriichuk , O. , Kadenko , S. , & Tsyganok , V. ( 2024 ). Significance of the order of pair-wise comparisons in Analytic Hierarchy Process: an experimental study . Journal of Multi-Criteria Decision Analysis , 31 ( 3-4 ), e1830 . https://doi.org/10.1002/mcda.183.

[12] Tsyganok , V. ( 2010 ) Investigation of the aggregation effectiveness of expert estimates obtained by the pairwise comparison method . Mathematical and Computer Modelling , 52 ( 3-4 ), 538 - 544 . https://doi.org/10.1016/j.mcm. 2010 . 03 .052

[13] Rezaei , J. ( 2015 ). Best-worst multi-criteria decision-making method . Omega , 53 , 49 - 57 . https://doi.org/10.1016/j.omega. 2014 . 11 .009.

[14] Rezaei , J. ( 2016 ) Best-worst multi-criteria decision-making method: Some properties and a linear model . Omega , 64 , 126 - 130 . https://doi.org/10.1016/j.omega. 2015 . 12 .001

[15] Andriichuk , O. , Tsyganok , V. , Kadenko , S. , & Porplenko , Y. ( 2020 ) Experimental Research of Impact of Order of Pairwise Alternative Comparisons upon Credibility of Expert Session Results Proceedings of the 2nd IEEE International Conference on System Analysis & Intelligent Computing, (SAIC) , 05 - 09 October, 2020 , Kyiv, Ukraine, pp. 1 - 5 . https://doi.org/10.1109/SAIC51296. 2020 .9239126

[16] Kadenko , S. , Andriichuk , O. , & Tsyganok , V. ( 2024 ) Stability of Ranking-dependent Pair-wise Comparison Patterns in the Analytic Hierarchy Process . In Proceedings of the 18th International Symposium on the Analytic Hierarchy/Network Process (ISAHP 2024 ). Web Conference. Dec 13-15 , 2024 . Available at: https://www.isahp.org/uploads/26_001. pdf ( accessed May 28 , 2025 ); https://doi.org/10.13033/isahp.y2024. 009

[17] Kadenko , S. , Tsyganok , V. , Szádoczki , Z. , & Bozóki , S. ( 2021 ). An update on combinatorial method for aggregation of expert judgments in AHP . Production, 31 : 1 - 17 . doi: 10 .1590/ 0103 - 6513 . 20210045

[18] Miller , G. A.. The magical number seven, plus or minus two: Some limits on our capacity for processing information ( 1956 ). Psychological Review , 63 ( 2 ), 81 - 97 . https://doi.org/10.1037/h0043158

[19] Saaty T. L. ; Ozdemir , M. S. ( 2003 ) Why the magic number sever plus or minus two , Mathematical and Computer Modelling , 38 ( 3-4 ), 233 - 244 . https://doi.org/10.1016/S0895- 7177 ( 03 ) 90083 - 5

[20] Holland , J. ( 1975 ) Adaptation in Natural and Artificial Systems . University of Michigan Press, Ann Arbor, USA.

[21] Mitchell, M. ( 1996 ). An Introduction to Genetic Algorithms . Cambridge, MA: MIT Press.

[22] Pétrowski , A. , & Ben-Hamida , S. ( 2017 ). Evolutionary algorithms . John Wiley & Sons. ISBN 978- 1-119-13638-5.

[23] Stevens , S.S. & Galanter , E.H. ( 1957 ) Ratio Scales and Category Scales for a Dozen Perceptual Continua . Journal of Experimental Psychology , Vol. 54 , No 6, 377 - 411 . https://doi.org/10.1037/h0043680

[24] Tversky , A. , & Kahneman , D. ( 1974 ) Judgment under Uncertainty: Heuristics and Biases . Daniel Science, New Series, Vol. 185 , No. 4157 , pp. 1124 - 1131 . DOI: 10 .1126/science.185.4157.1124