This paper focuses on the need to evaluate the sustainability of Analytic hierarchy process at the Ranking of more than 10 alternatives. The proposed method is based on simulation modeling of the process of improving expert pair-wise comparison judgments. The represented method provides a stepwise improvement of the pair-wise comparison matrix transitivity. The average discrepancy and coincidence of ranks in multiple modeling are proposed as estimates of the rating stability. The application of the developed method was studied on a statistical sample formed according to the final tables of the England, Germany and Spain football championships. The method for determining probability of some alternatives ranks is developed. It is possible to modify the method for predicting the results of sports competitions and for the case of ranking with partially missing expert ratings.
Analytic hierarchy process (AHP) [
AHP solves the ranking alternatives problem on the basis of expert filled pair-wise comparison matrix (PCM).The result of the ranking is subjective.
The property of the ranking stability lies in the different susceptibility to changes in the results with insignificant variations in the assessments of the expert.
The question is how the results of ranking the same alternatives may differ from another expert with the same knowledge and experience remains open. It is difficult to assess how much the order of alternatives can change with minor or small changes in the expert’s opinion when comparing several pairs of alternatives. Another problem with AHP is that removing one or more alternatives from the list of ranked ones can change the remaining alternative ranks relative to each other.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The stability problem is considered when solving two problems [
Let PCM A = [aij ]
with i, j = 1,…, n be a reciprocal positive matrix corresponding to
the paired comparisons on a Saaty scale [
Objectives of the study (in relation to PCM of significant dimension): • offer indicators of ranking stability for AHP; • establish patterns of change in these indicators with a slight variability in expert assessments; • propose a method for assessing the stability of ranking in solving specific problems; • assess the stability of the ranking with some expert incompetence, with the possibility of evaluations such as “hard to say” or “I don't know”. 3
This paper investigates the ranking stability of a single-level AHP to change part of the expert’s ratings with an improvement in the degree of the PCM transitivity and consistency. The results of the research are also applicable to AHP modifications with ranking alternatives for incomplete data.
The ranking for incomplete data [
Harker [11] on PCM with the number of alternatives 6–9 filled randomly investigated the possibilities of 5% underfilling.
In most cases, the stability of the ranking is determined by the “what happens if” principle, interactively changing grades and tracking the ranking results. The authors of [12] integrate the AHP with the stochastic multicriteria acceptability analysis, an inverse-preference method, to make pair-wise comparisons uncertain. A simulation experiment is used to determine how the accuracy of the decisions and the ability of the model to find the best option deteriorates as the uncertainty increases.
The next area of research [
Aguarón and Moreno-Jiménez [
The method was further developed in [
To calculate these intervals, the row geometric mean method as the prioritization procedure, the geometric consistency index as the consistency measure and a local situation with one criterion were considered.
There are several other interesting studies to improve the consistency of PCMs. An optimization algorithm for minimizing the consistency ratio (CR) has been proposed, assuming that the expert statements have an accuracy of 10% [17]. The places of greatest inconsistency in [18-21] are calculated by the matrix model of the induced bias of the Hadamard product [19], by the M Outflow method [21]. The expert is encouraged to change his opinion on his own or on the basis of a determined recommendation. In [22], the number of paired comparisons is reduced to ensure that the ranking is not precise but approximate. A variety of other options were considered in [23] to solve the problems of inconsistency of expert judgments and incompleteness of PCM.
The research [
The stability of AHP ranking at significant PCM dimensions was not researched in the reviewed papers. 4
The method of improvement consistency expert comparison judgments (ICECJ) is based on simulation modeling of the expert’s work to improve the consistency of his assessments.
Consider transformations of PCM A. Part of the expert’s ratings, randomly selected, is reset to zero. Then the zero values are replaced by averaged estimates according to the transitivity property of the remaining estimates, according to the sequence of calculations: then and then aij = 1 ∑n aik ⋅ akj , n1 k =1 aji = 1 ∑n ajk ⋅ aki , ,.
n1 k =1 aij = (aij + 1 ) / 2,
aji [aij ], if aij ≥ 1 aij = [1/ aij ] otherwise
9, if aij > 9 , aij = aij otherwise , aji = 1/ aij (1) n 1, if x = y where n1 = ∑δ (aik ⋅ akj ,0) , δ (x, y ) = , [x] –the integer part of number x .
i =1 0, otherwise
Thus, PCM is modified, AHP ranking is performed. Repeatedly executing this procedure allows you to get a set of solutions for ranking alternatives. The analysis of many solutions allows us to determine some estimates of the solutions accuracy: • the probability of an alternative appearing at each ranking level; • the average divergence of ranking in the solutions set; • the average coincidence of ranking in the solutions set; • the average divergence of ranking place in the initial solution and on the set of modified PCMs for all alternatives. 5 5.1
Due to the subjectivity of the AHP process, the ranking results have some error (instability). To estimate this error, methods of probability theory and mathematical statistics should be applied.
To do this, you must have a sufficient amount of statistical information. Due to the fact that in most cases AHP is used in strategic planning, there are few numbers of its constant and repeated use, especially with a large number of alternatives (more than 10), and the authors are not aware of such information.
To assess the sustainability of the ranking results, this paper uses ranking data without an expert. The role of the expert is played by nature. For research, we will use the information about the ranking of teams as a result of the football championships: Germany Bundesliga 53 seasons from 1963/1964 to 2015/2016, Spain La Liga 29 seasons from 1987/1988 to 2015/2016 and England Premier League 24 seasons from 1992/1993 to 2015/2016. A total of 105 final tables of football championships were processed. They were attended by 16 (2 times), 18 (49 times), 20 (49 times) or 22 (5 times) teams.
PCM is also filled here and rankings are performed, but on a different scale than proposed by T. Saaty. Moreover, paired comparisons are performed by nature and depend on many random factors (player disqualifications, injuries, physical and psychological state of the players, etc.). PCM has some natural inconsistency, but the ranking results are fairly objective. 5.2
To reseach AHP, there was made a transition from the traditional football championships of Spain, England and Germany to the Saaty scale, in which the superiority of one team over another is estimated by integers from 1 to 9 and, accordingly, concession by the inverse number from 1 to 1 / 9.
The converting was performed as follows: 1, if mij = 0 [bmij + c], if [bmij + c] ≤ 9 & mij > 0 aij = 9, if [bmij + c] > 9 & mij > 0 where are aij – the PCM elements, mij – is the difference between goals scored and conceded in two championship matches between the i-th and j-th teams, b and c are some (desired) coefficients, [x] – is rounding to the nearest integer.
To assess the quality of the transition from a traditional football table to PCM on the Saaty scale, the following indicators are proposed: – the average discrepancy of places in the traditional table and ranking by AHP SRj = 1 ∑n (OiTj - OiSj ) , where nj – the number of teams that participated in the j -th nj i =1 championship, OiTj – the rank of the i -th team in the traditional table of the j -th championship, OiSj – the team rank after ranking by AHP using (2); – the average proportion of team that retained their ranks after the conversion SEj = 1 ∑nδ (OiTj , OiSj ) .
nj i =1
The optimal values of b and c are determined for each of the three championships separately. They were determined as a result of two-criteria optimization: minimizing the discrepancy of ranks and maximizing the number of teams that retained their rank after converting the tables of football championships to PCM on the Saaty scale: (b,c ) = arg(min 1 ∑N SRj (b,c) - mjin SRj (b,c ) b,c N j =1 m ajx SRj (b,c) - mj in SRj (b,c)
SEj (b,c-) min SEj (b,c)
j max SEj (b,c- ) min SEj (b,c) j j ) , (3) where N – is the number of championship tables, arg – defines functions arguments b,c at which a minimum is reached. The corresponding quality indicators and Consistency ratio (CR) are given in table 1.
An example of converting the traditional German championship table of the 1963/1964 season to PCM is given in table 2 and 3.
In table football teams marked as C1 – Braunschweiger TSV Eintracht 1895, C2 – Eintracht Frankfurt, C3 – BV Borussia 09 Dortmund, C4 – SV Werder Bremen, C5 – Hamburger SV, C6 – Hertha BSC, C7 – MSV Duisburg, C8 – FC Köln, C9 – FC Kaiserslautern, C10 – Karlsruher SC, C11 – TSV 1860 München, C12 – FC Nürnberg, C13 – SC Preußen 06 Münster, C14– FC Saarbrücken, C15 – FC Schalke 04, C16– VfB Stuttgart 1893.
To unify the processing, points were awarded according to modern rules: for a victory – 3 points, a draw – 1 and a loss – 0. Therefore, some of the ranking results in the final tables of the championships (including the ones given in table 2) differ from the official ones.
In the above example, the following indicators were obtained: CR = 0.15, SR = 1,75 and SE = 0,375 . The results of the conversion to PCM (average discrepancy of about two ranks and one third of the teams that retained their rank) are close to the average for the Germany Bundesliga. You should not expect a better matching because of a significant, non-linear change in the measurement scale
As a result of the transformations, the statistical material PCM (105 matrices) was obtained, which is close to that was formed by nature and sufficient to study the stability of PCM. Experiment 1. The goal is to assess the probability of an alternative being placed in appropriate ranks.
The ICECJ method of increasing PCM transitivity with a small percentage of redefined expert ratings (5% here) and a sufficiently large number of parallel experiments occupy their ranks. You can count on the objectivity of this situation. Alternative C1 is almost the same for the 14th and 15th positions. For this alternative you need to provide additional criteria.
Experiment 2. The goal is to identify patterns of change in the sustainability indicators of ranking results and PCM consistency.
In the experiment, the number of modified estimates varied from 5 to 30%. Each experiment was repeated 50 times.
As a result of the experiment, it was possible to quantitatively establish the degree of improvement in PCM consistency when changing expert estimates (and increasing the degree of transitivity) and the corresponding losses estimated by averaged: discrepancies before and after changes in the ranking of alternatives, fractions of alternatives that have retained their place and the difference in weight of alternatives (Fig. 1 and 2).
The studies were conducted separately on PCM, transformed according to the football championships results of the three countries.
The reliability of the results obtained is confirmed by its proximity and the small confidence intervals obtained for both 90 and 95% level of consistency. An example of the dependence SRj for PCM formed on the results of the Germany Bundesliga is shown in Fig. 3.
From 2% (with 5% of the changed estimates) to 20% (at 30%), the сonsistency of MPS improves, from CR= 0.113 to CR= 0.09. In this case, the average discrepancy in the weights of the alternatives varies by 10-25%, respectively.
It was found that even insignificant (in five percent) changes in the estimates of alternatives (even in the direction of improving consistency) lead to 45% of the divergence of places in the ranking. With a 5% change in expert ratings, about 50% of teams retain their places, and with 30% only one fifth.
We take as a basis the accuracy of the transition from traditional methods of pairwise comparisons in football championships to the assessment according to AHP, as natural. It can be noted that with 20-25% of the expert’s assessments being changed, the percentage of teams that have retained their place and the average divergence of places are approaching natural accuracy.
In other words, when additionally determining 20-25% of the missing expert ratings in AHP, the accuracy of the ranking corresponds to the natural accuracy of reproducibility. The teams ranking by generally accepted rules of football championships is also in some measure unstable. The place of the team may depend on one or two points (goals) and even the difference between goals scored and goals that were conceded. If at least productive judicial errors are corrected, the ranking results may differ by 1-3, and sometimes even more, positions.
The AHP-based ranking is also in some measure unstable. The expert makes a decision on the basis of information available to him, which is fundamentally incomplete, changes over time, and is also partially fuzzy, contradictory and erroneous.
Since the proposed ICECJ method involves improving the transitivity in expert assessments, the PCM consistency assessment is improved. Consequently, the stability estimates obtained are “optimistic”.
Many users of information systems do not adequately interpret the results of tasks. Sometimes the results are perceived as the ultimate truth, the characteristic statement is “that was counted by computer”. Moreover, this applies to integer results, in our case these are ranks, places of alternatives. The concepts of error, stability among users are often absent.
Therefore, information systems, solving such problems, should inform the user about the errors of decisions and their stability. The results of the study show that they are quite significant. Changing several estimates of pairs of alternatives can often lead to a shift of the alternative in the resulting ranking by 1-5 positions. This also applies to the leading alternative.
In this paper, we studied the stability of ranking alternatives by AHP with one criterion. Further development of the work is connected with multicriteria tasks. At the same time, tasks with a significant number of criteria and alternatives (over 10), a small number of criteria and alternatives, a significant number of criteria, and a small number of alternatives should be considered separately.
It can be assumed that with an increase in the total number of pair-wise comparisons, in view of their certain (albeit small and constant) inconsistencies, ranking errors increase statistically.
In the future, the ICECJ method to predict the results of sports competitions (at the final stage) can be modified. The solution of similar problems is a separate area of research (for example, [14, 15]).
It is also assumed to be promising for methods application in the case of AHP ranking with part of missing expert estimates and a predicted error estimate (solution stability). 7. Takeda, E., Yu, P. L.: Eliciting the relation weights from incomplete reciprocal matrices.
Proceedings of International Symposium on the Analytic Hierarchy Process. Tianjin university, Tianjin, China, 192-200 (1988). 8. Fortes, I., Mora-L’opez, L., Morales, R., Triguero, F.: Inductive learning models with missing values. Mathematical and Computer Modelling, 44, 790–806 (2006). 9. Hong, T. P., Tseng, L. H., Wang S. L.: Learning rules from incomplete training examples by rough sets. Expert Systems with Applications, 22, 285–293 (2002). 10. Lakićević, M. D., Reynolds, K. M., Srđević, B. M.: Assessing landscape plans with abbreviated pair-wise comparisons in the AHP (Analitic Hierarchy Process). Zbornik Matice srpske za prirodne nauke, 136, 183-194 (2019). doi:10.2298/ZMSPN1936183L 11. Harker, Р. Т.: Incomplete pair-wise comparisons in the Analytic hierarchy process.
Mathematical Modelling, 9(11), 837–848 (1987). 12. Durbach, I., Lahdelma, R., Salminen, P.:. The analytic hierarchy process with stochastic judgements. European Journal of Operational Research, 238(2), 552-559 (2014). 13. Altuzarra, A., Gargallo, P., Moreno-Jiménez, J.M., Salvador, M.: Influence, relevance and discordance of criteria in AHP-Global Bayesian prioritization. International Journal of Information Technology & Decision Making, 12(04), 837-861 (2013). doi:10.1142/S0219622013500314 14. Stübinger, J., Mangold, B. and Knoll, J.: Machine Learning in Football Betting: Prediction of Match Results Based on Player Characteristics, Applied Sciences, 10(1), 46 (2020). doi: 0.3390/app10010046 15. Aguaron, J., Escobar, M.T., Moreno-Jiménez, J.M.: Consistency stability intervals for a judgement in AHP decision support systems. European Journal of Operational Research, 145(2), 382-393 (2003). doi:10.1016/S0377-2217(02)00544-1 16. Araujo Fernandes, M.: Using Soft Computing Techniques for Prediction of Winners in Tennis Matches. Machine Learning Research, 2(3), 86-98 (2017). doi: 10.11648/j.mlr.20170203.12 17. Borkar, P., Sarode, M. V.: Modality of teaching learning based optimization algorithm to reduce the consistency ratio of the pair-wise comparison matrix in analytical hierarchy processing. Evolving Systems, 9(2), 169-180 (2018). doi 10.1007/s12530-017-9185-9 18. Jarek, S.: Removing Inconsistency in Pairwise Comparisons Matrix in the AHP. Multiple
Criteria Decision Making, 11, 63-76 (2016). doi: 10.22367/mcdm.2016.11.05 19. Kou, G., Ergu, D., Lin, C., Shang, J.: Enhancing data consistency in decision matrix: Adapting Hadamard model to mitigate judgment contradiction. European Journal of Operational Research, 236, 261–271 (2014). doi:10.1016/j.ejor.2013.11.035 20. Prasad, V. S., Kousalya, P.: Role of consistency in analytic hierarchy process–consistency improvement methods. Indian Journal of Science and Technology, 10(29), 1-5 (2017). doi: 10.17485/ijst/2017/v10i29/100784 21. Nedashkovskaya, N. I.: Investigation of methods for improving consistency of a pairwise comparison matrix. Journal of the Operational Research Society, 69(12), 1947-1956 (2018). doi:10.1080/01605682.2017.1415640 22. Heckel, R., Simchowitz, M., Ramchandran, K., Wainwright, M. J.: Approximate ranking from pairwise comparisons. arXiv preprint arXiv:1801.01253 (2018). 23. Kou, G., Ergu, D., Lin, C., Chen, Y.: Pairwise comparison matrix in multiple criteria decision making. Technological and Economic Development of Economy, 22(5), 738-765 (2016). doi:10.3846/20294913.2016.1210694