An Overview of Decision Support Software: Strategic Planning Perspective

Abstract

The paper features a review of available software tools and systems that utilize expert data for decision support in weakly-structured subject domains. Existing tools are analyzed and compared among themselves from the standpoint of implemented mathematical approaches and functions, particularly, those related to strategic planning. We outline the key trends of decision support tools development, witnessed during the last decades. We show that existing automated decision support tools have a bunch of limitations and drawbacks. Based on conducted analysis, we formulate relevant requirements to modern decision support software and specific recommendations as to selection of decision support tools for strategic planning and further improvement of existing decision support products.

Keywords

• Decision-making Support
• Expert Estimation Scale
• Target-oriented Hierarchic Decomposition
• Strategic Planning
• Resource Allocation
• Scenario Analysis

-

knowledge. By a strategy we propose to denominate a long-term, step-by-step, constructive, rational, and uncertainty-proof plan. Its implementation is accompanied by constant analysis and monitoring, and is targeted at achievement of a certain main goal. An essential component of strategic planning in a given domain should be an adequate model of this domain (built, for example, in the form of a hierarchy of impact factors [ 2,3 ]). Building of such a model allows the DM to solve a set of problems, such as: - Strategic planning itself; - Ranking and rating of goals, projects, factors, criteria, and alternatives; - Evaluation of relative efficiency of projects; - Priority-setting in the DM’s activity; - Distribution of limited resources among projects; - Defining the efficiency of potential decision variants; - Generating and analysis of situation development scenarios, etc.

In order to obtain maximum information, required for decision-making, it makes sense for the DM to involve not one expert, but a group of experts in the process. Recommendations, formulated as a result of consensus between expert group members, will be better substantiated, than advice of a single specialist. Under modern realities, a desirable requirement to decision support (DS) tools is an opportunity to organize expert work in remote mode (online).

High level of complexity and uncertainty of weakly structured subject domains, the need for interdisciplinary approaches, consideration of large numbers of heterogenous impact factors and opinions of multiple experts from different spheres (who work online, in the general case) call for automation of DS process, particularly, during strategic planning. Modern decision support tools and systems (DSS) are intended to simplify, and, at the same time, improve the process of collection and processing of expert information, in order to facilitate its most thorough usage. If information, obtained from experts, is complete, consistent, and detailed; if it is not distorted during collection and processing, then recommendations for the DM, formulated on its basis, become more credible, and respective decisions become better substantiated.

In this paper, in order to analyze the capacities of modern DSS tools, we suggest to use a widely-approbated original technology of strategic planning [ 4 ] as basis. The technology involves the following conceptual phases: 1) The DM formulates the main goal of the strategic plan in the given subject domain and recruits the expert group (for example – using co-authorship networks [ 5 ]); 2) The experts hierarchically decompose the main goal into factors, influencing its achievement. Decomposition is performed until the level of specific projects is reached. Projects are “atomic” factors, within the competency of the DM. 3) The experts estimate relative impact of each factor from the hierarchy. The outcome of this phase is a DSS knowledge base (KB), describing the subject domain in the context of the specific main goal of the strategic plan, represented by a weighted hierarchy graph. 4) Finding the optimal (rational) distribution of limited available resources among projects, which maximizes the strategic goal achievement degree. Calculation of this degree is performed for a given moment of time and takes into consideration the current impact of all factors and threshold values of project funding.

The technology demonstrated its high efficiency in such applications as: - Planning of defense sphere development; - building of development strategy and evaluation of efficiency of measures o space activity and production of space equipment in Ukraine [ 4 ]; - environmental protection sphere [ 6 ]; - information operation research and recognition [ 7 ]; - building and analysis of situation development scenarios [ 8, 9 ].

Manufacturers of DS tools worldwide also use similar approaches, and this fact additionally testifies in favor of the technology [ 10 ].

We should note that any automated DS tools have their own original intended purposes and pre-conditions of development. That is why it is problematic to single out a unified common criterion, based on which all DS software products could be compared with each other. In this paper we try to review the features of the most well-known tools in the context of their ability to solve the problems, that emerge during strategic planning in weakly structured subject domains (based on expert and other information). 2

An Analytic Overview of the Most Widely-used DS Tools

Due to recent expansion of DS tools usage, we are witnessing an increase in demand for intelligent systems intended to solve DS problems. World-known DSS, developed in recent years, include the following ones: ExpertChoice, SuperDecisions, DecisionLens, D-Sight, Promethee, ОЦЕНКА И ВЫБОР (Estimate and choice), Solon, Consensus, and others.

Common characteristic features of most DSS, irrespectively of their intended purpose, are as follows: a KB, a subject domain model (decision context and estimation criteria), and a user interface [ 11 ]. We propose to focus on universal-purpose DSS, whose mathematical ware incorporates the most popular contemporary decision support methods. 2.1

SuperDecisions DSS and Similar Products

One of the most widely used decision support methodologies today is the analytic hierarchy/network process (AHP/ANP), developed by Tom Saaty [ 2 ]. It is implemented in SuperDecisions DSS [ 12 ]. This system largely replaced another popular product, ExpertChoice DSS [ 13 ]. The system is designed to calculate the relative efficiency (weight) of alternatives based on multi-criteria estimation. In the general case, the hierarchy of criteria includes four sub-graphs: B – benefits, O – opportunities, C – costs, and R – risks.

Weights of each of alternatives and importance of criteria are defined through expert pair-wise comparisons performed in fundamental scale, or through direct estimation. Obtained estimates are aggregated through weighted summation. Estimates according to criteria, representing benefits and opportunities are considered with “plus” sign, while estimates according to criteria, representing costs and risks – with “minus” sign.

The range of applications of the system is extremely wide. Proceedings of the International Symposiums for the Analytic Hierarchy Process for the past 32 years (19882020) prove it [ 14 ].

SuperDecisions is just one of the many DSS, based on AHP/ANP methods. Other DSS, where these methods are used, include DecisionLens, ExpertChoice, MakeItRational, MindDecider, RationalFocalPoint (RFP), SmarterGovernment. At the same time, complete strategic planning cycle, outlined in the introduction, is implemented only in DecisionLens system (which has a license cost of 25,000-30,000 euro, and no trial version). Other systems are focused on ranking and rating of decision variants according to some aggregate criterion.

So, SuperDecisions and most of the similar systems feature only separate steps of the strategic planning technology. DecisionLens, the only system, in which resource allocation phase is actually implemented, is much less affordable, than other DSS, due to high license cost. Besides that, most systems have limitations, concerning remote expert session organization. 2.2

PROMETHEE Visual DSS

Just like SuperDecisions, this DSS is intended for ranking of several decision variants (alternatives) based on their estimates according to several criteria. PROMETHEE DSS and DS method of the same name (the abbreviation, translated from French, stands for “the method for organization of preferences for enrichment of estimates”), embedded in it, were developed and described by Belgian researchers [ 15 ]. Promethee Visual is a “successor” of previous products, in which the method was implemented, such as PromCalc and DecisionLab.

The estimates are aggregated through weighted summation. Object estimation criteria can be both quantitative and qualitative. Estimates can be absolute or relative. The peculiar feature of the method is the so-called “preference function” (U-shaped, Vshaped, Gaussian, stepwise), which indicates the specificity of preference relationship, depending on one or several determinant parameters.

The system allows users to solve only separate sub-tasks of the aforementioned strategic planning procedure. Its core functions do not include resource distribution or remote expert sessions. 2.3

1000minds DSS 1000minds [ 16 ] is a tool for group decision-making in remote mode. Alternatives are ranked based on their estimates according to several criteria. The system does not include any desktop versions or applications – all the work on decision-making support is performed online.

The mathematics behind the system is based on PAPRIKA method (Potentially All Pairwise RanKings of all possible Alternatives) [ 17 ]. The experts are asked to provide ordinal pair-wise comparisons of alternatives from a given set (answer the question: “which of the two alternatives is better?”). Often the task is to reach a compromise between several criteria. For example, the expert is offered to choose a costly, but promising project, or a less promising project, that costs less. The number of criteria can be more than two. Particular order of pair-wise comparisons is intended to minimize the number of times the expert is addressed; many ordinal pair-wise comparisons are “restored” according to transitivity rules, based on already available expert judgments. The final result of the expert session is the ranking and the rating of alternatives.

As we can see, the system is mainly focused on the first phases of the strategic planning procedure. Resource allocation function is unavailable. 2.4

AIRM online DSS “AIRM online” DSS implements the aggregated indices randomization method (AIRM) [ 18 ]. The method is intended to reduce the uncertainty, emerging during estimation of weight coefficients: the information on the weights of criteria and alternatives is often provided in the form of ordinal instead of cardinal estimates, or numeric intervals. This information is often insufficient to calculate weight coefficients. Weight vector is chosen based on Bayesian randomization, from among all possible values, which can be assumed by its coordinates. It is presumed that each coordinate is evenly distributed on a certain numeric interval, and, consequently, if several objects are compared according to several criteria, then the aggregate quality indicator Q, is also a random variable. Its value is defined as mathematical expectation of this variable, and precision is characterized by standard deviation of the indicator.

The system is targeted at a certain type of estimates, and allows users to solve only a certain type of problems, i.e. alternative weight calculation (this is just one of the phases of the strategic planning technology). 2.5

Analytica Package

The visual software package “Analytica” developed by Lumina Decision Systems [ 19 ], uses the so-called relationship or impact charts for illustrative representation of a situation, calling for decision-making.

Mathematical background of the system is, mainly, based on statistical tools, rather than on some specific multicriteria decision support methods. Considerable attention is dedicated to building of probability distribution functions, that characterize alternative decision options.

The system does not support estimate input in the form of pair-wise comparisons – the expert is required to input measurement units, ranges, extreme values of indicators, that influence the achievement of a certain goal.

So, a considerable advantage of Analytica package is that an expert has an opportunity to set the estimation scale in the process of the expert session. However, again, the package implements only certain phases of strategic planning cycle.

D-Sight DSS

D-Sight software [ 20 ] is intended for multicriteria evaluation (ranking) of alternatives in different qualitative and quantitative scales. An expert can design his(her) own scale, such as “a scale of professional competence”. Just like many other DSS, the system allows us to input estimates of alternatives and weights of criteria. The software has many tools for visualization of expert session results, particularly, using PROMETHEE/GAIA method and multi-attribute utility theory (MAUT).

So, just like “Analytica” D-Sight has some advantages, related to selection of estimation scales, however, it does not incorporate the whole strategic planning cycle, and does not support remote work of a group of experts. 2.7

MakeItRational DSS

MakeItRational is another AHP-based DSS [ 21 ]. The system is intended for multi-criteria evaluation and selection of alternatives from a given set. Initially, a set of alternatives is input. Then a global estimation criterion is formulated. It is decomposed into a hierarchy of sub-criteria, representing both negative and positive influences (costs and benefits). Next, alternatives are estimated according to criteria. Their estimates can be input in the form of both numerical values and pair-wise comparisons in Saaty’s fundamental scale. The final result of the expert session is the rating of alternatives according to the global criterion and its sub-criteria. The system supports exporting of expert session results into MS Excel and automatic compiling of a document with the report on the expert session outcomes.

Thus, just like SuperDecisions, MakeItRational, mainly, focuses on problems, related to rating of alternatives. Obtained ratings can provide the basis for further strategic planning and priority-setting, however resource allocation functions are not implemented in the software. 2.8

MindDecider DSS

Just like other systems, already listed in the paper, MindDecider DSS [ 22 ] is intended for multicriteria estimation and rating of alternatives. The expert session is divided into several stages, each represented by a respective operation mode: planning and design, estimation, terminal analysis, report/log compiling. First, the main goal is formulated, then – a set of estimation criteria and ranges of estimate values. After that, alternatives are estimated according to criteria. After aggregation of alternative estimates according to specific criteria their global rating (in per cents) and global ranking are compiled.

MindDecider does not incorporate the whole strategic planning cycle. 2.9

LogicalDecisions DSS

Logical Decisions DSS [ 23 ] utilizes the tools of AHP and MAUT. The system is intended for multicriteria estimation of alternatives from a given set. Criterion weights («measures») are defined through direct estimation, or pair-wise comparison of alternatives in an arbitrarily chosen scale. Alternatives are estimated through pair-wise comparisons, in the arbitrary scale as well. So, Logical Decisions also incorporates just certain phases of the strategic planning cycle. 2.10

TreeAgePro DSS

Tree Age Pro [ 24 ] is intended for evaluation of different decision variants. Possible options are presented in the form of decision tree branches. Decision tree graph (similar to impact diagrams from Analytica system) can include nodes of different types: “chance” (probability node), “Markovian node”, “decision node”, “Boolean node” etc. Again, similarly to Analytica, TreeAgePro provides a whole environment for automation and structuring of expert estimation and choice process while solving various kinds of problems. However, the mathematical tools, providing the backbone of the system, are mostly based on probability theory. We should stress, that TreeAgePro and Analytica packages have broad opportunities for adaptation to specific problems, but respective functions have to be implemented separately. 2.11

RFP (RationalFocalPoint) DSS

RFP (Rational Focal Point) [ 25 ] is a complex of means for automation of decisionmaking support process at municipal level. Particularly, these decisions concern funding of various projects, portfolio investment, and others. Decision variants are evaluated according to quantitative and qualitative criteria. The system supports both pairwise comparisons and direct expert estimation. Since the software is intended for municipal-level decision support, a large number of factors, whose impacts change with time, and opinions of multiple experts, who work in remote mode, need to be taken into consideration. So, developers of the system (first – IBM, then – Unicom) focused on online operation mode and on the features related to modeling of different decision options (scenarios) in time. Thus, the opportunity of working in remote mode is a strong advantage of the system. However, narrow intended purpose of the system limits its functional capacity a bit. 2.12

VeryGoodChoice DSS

Very Good Choice [ 26 ] system (somewhat outdated from the standpoint of implemented software solutions and no longer supported by the developers), built by MVLsoft company, is based on a famous decision support method – ELECTRE (Elimination Et Choix Traduisant la Realite – elimination and choice, reflecting the reality) [ 27 ]. The software is an add-in for Microsoft Excel package. Its capabilities in the context of strategic planning are limited. 2.13

“ESTIMATION AND CHOICE” («ОЦЕНКА И ВЫБОР») DSS «ОЦЕНКА И ВЫБОР» is an internet-based software system, intended for solving multicriteria decision-making problems. These problems might be hierarchically structured (from upper-level to lower-level criteria: Purpose of analysis – Generalized indicators – Indicators). «ОЦЕНКА И ВЫБОР» uses different decision support methods: AHP, value functions, ordinary weighting method, Pareto dominant analysis, BENEFIT/COST analysis.

The final result of an expert session is the rating of alternatives, calculated as weighted sum of the respective indicators, characterizing the alternatives. The indicators may be qualitative, quantitative, or Boolean ones.

We should stress that the system is initially oriented towards remote decision support problem solution.

Remote operation and support for different estimate types are the key advantages of the system, while definition of an optimal strategy remains beyond the scope of its capacities. Detailed description of the system’s features and functioning is provided in [ 28, 29 ]. 2.14

“SVIR’” («СВИРЬ») Instrumental System According to the system’s developers, “SVIR’” is an instrumental system for multicriteria decision-making problem solution. It satisfies such requirements as: universality, simultaneous usage of objective and subjective estimates, high-dimensionality problem solution, autonomous problem solution, opportunity for integration with other systems for data exchange and processing, ergonomic design, ability to evolve.

Multi-criterial choice problems are solved by the system using multi-criteria optimization and classification methods, as well as AHP. Object estimation criterion weights are set through direct estimation, or calculated based on pair-wise comparison matrices based on expert preferences, or equivalence of features in tables or primary features.

The results are presented in a unified colored quality scale, which can be configured and saved. On the output, the system also allows users to obtain charts of object distribution according to criteria and to aggregate estimate, as well as, analyze the contributions of criteria into the aggregate estimate (perform sensitivity analysis).

So, the advantages of the system are its support for different estimation scales and graphic representation of expert session results. Certain phases of the strategic planning technology (such as resource allocation) need to be implemented separately, or performed by additional external modules. The system and its modifications are described in numerous publications, such as [ 30, 31 ]. 2.15

“Solon-3” and “Consensus-2” DSS “Solon-3” DSS is a tool for decision-making support based on complex target-oriented program (CTP) building. A CTP is a set of activities, united by a common goal and shared resources. The key tasks performed during CTP formation are 1) main goal formulation, 2) defining prospective directions of its fulfillment (sub-goals), 3) selection of the most efficient means (projects) and 4) resource distribution among selected projects.

In order to complete tasks 2) and 3), one needs to rank the objects (projects, goals).

Technological decision-support process using “Solon-3” DSS [ 32 ] includes the following phases: decomposition of the main goal and building of a hierarchy of goals, expert estimation of partial impact coefficients of sub-goals, calculation of relative efficiency of different ways of program implementation and formulation of alternative projects, calculation of coefficients of projects’ impact upon the main goal, which are used as indicators of relative efficiency of the projects. To calculate the relative efficiency of the projects, the given DSS uses the method of dynamic target-oriented evaluation of alternatives (MDTEA). Detailed description of “Solon-3” DSS can be found in [ 3,32 ].

The system of distributed collection of expert information (SDCEI) “Consensus-2” [ 33, 6 ] was developed and implemented to simplify the process of group expert session organization in remote mode. The system is a cloud solution; it allows the expert group members to join an online session, organized to solve a specific problem, and to perform group decomposition of this problem. The system’s interface significantly simplifies task 2) from the above-mentioned list. For resource allocation to projects additional special software modules need to be connected to the system. 2.16

Other Decision-making Support Software Products and Applications

As existing DS software is highly demanded by representatives of different application fields, it is constantly improved and upgraded. At the same time, new products are being developed, in the form of both standalone solutions and add-ons or apps. For instance, Medical Sapiens web-platform for medical decision-making [ 34 ], Total Decision software [ 35 ], and Decision Mentor app [ 36 ] were developed just a few months ago (as of the moment of this paper preparation). Besides, Decision Mentor is an example of one of the latest trends – DS solutions going mobile.

Extensive assortment of DS solutions, offered by online software marketplaces, also signifies the evolution process. For instance, Capterra platform [ 37 ] (in addition to already mentioned DSS 1000minds, DecisionLens, Analytica, ExpertChoice) features dozens of software products and whole packages. Each of them is implemented in its own particular way and has its own intended purpose in a specific subject domain (customer support (Zingtree), visualization of complex group decision-making situation in large organizations (EIDOS), and others).

We should also note that “decision support software” concept assumed wider meaning in the eyes of online marketplace customers. For example, Wolfram Mathematica software (also available through Capterra platform) is a cloud solution (in fact, a whole environment), intended for image recognition, data visualization, and other functions, using machine learning, neural network algorithms, data mining, and other top-notch approaches. Yet, it is still attributed to decision support software.

Moreover, criteria, according to which the software tools are compared, are shifted towards their commercial attractiveness. Capterra platform offers the following list of product comparison characteristics: initial cost, availability of a free trial version, way of implementation (cloud solution, web application, iOS/Windows/Android, mobile app etc), necessity for coaching sessions for users (online tutorials, documentation, webinars etc), and availability of tech support (weekdays only, 24/7 online support, etc). Almost no attention is dedicated to consideration (or analysis) of specific mathematical and technological solutions, providing the basis of this or that DS tool. So, it is very problematic to consider the respective tools in comparative context, especially if you are an academic researcher rather than a beginner-level user with a narrow-profile specific demand.

We should stress that attribution of DS software tools and applications to specific narrow problems and subject domains (most probably resulting from business interests) significantly limits the opportunities for their usage. From the standpoint of users, a DSS should be universal, and, at the same time, easily adaptable to specific applied problems in each domain (for example, through hierarchical decomposition these problems and swift adjustment of subject domain models). Otherwise, a new separate DSS would have to be applied for each specific subject domain or problem.

Authors of [ 38, 39 ] provide a comparative analysis of some DSS, including those, not listed in our review. Having appended the review with information on some other DSS [ 38, 39 ], we are able to obtain aggregate comparison data on DS software, particularly, from the standpoint of mathematical and technological solutions implemented in them (see Table 1).

If we take just the 4 conceptual phases of the above-mentioned strategic planning cycle as DSS comparison criteria, the comparison makes no sense: all the systems we consider do allow users to rate alternatives, factors, and projects (incorporate 3 phases out of 4), while only one system supports resource allocation functionality. That is why, in order to compile an illustrative comparison table, we have selected more constructive (specific) criteria.

In addition to opportunities for alternative estimation scale selection (pair-wise comparison scale or others), group estimation, and online expert session organization (through a cloud software version), several other DSS functions also significantly influence the credibility of decisions (including strategic ones). These functions include consideration of time lags in implementation of certain components of a strategic goal, as well as automated analysis of sensitivity of the final decision variants’ rating/ranking to perturbations of input data.

Our review is not an exhaustive one, however it illustrates the general trends in the development of modern DSS, using expert data (including pair-wise comparisons and other types of estimates in various scales).

We should note, that none of the listed systems allows its users to organize the whole strategic planning cycle (outlined in the introduction) using expert information in remote mode. For example, DecisionLens (one of the most expensive DS software systems in the world) combines the maximum number of respective functions, required for strategic planning (including resource allocation).

DSS name 1000Minds AIRM Online Analytica DecisionLab DecisionLens D-Sight Logical Decisions Make It Rational MindDecider PROMETHEE Visual RFP TreeAgePro Very Choice ОЦЕНКА И ВЫБОР (Estimation and choice) SVIR’ SOLON ExpertChoice

AHP SuperDecisions

AHP, ANP Good

ELECTRE Implemented methods PAPRIKA AIRM AHP, ANP MAUT, PROMETHEE AHP, MAUT AHP AHP PROMETHEE AHP, ANP AHP, utility Y function, Pareto Analysis Multi-criteria optimization, Classification, AHP MTDEA

Y N N Y Y Y Y Y Y N Y Y Y N Y Y

Y

However, it has its own limitations concerning online expert sessions and expert estimation scale selection. 1000minds, RFP, “ОЦЕНКА И ВЫБОР” (“Estimation and choice”) systems are intended for remote work of experts, but they focus on ranking and rating of alternatives. D-Sight and PROMETHEE Visual offer wide opportunities for estimation scale selection, but have limitations in terms of remote expert sessions. “Solon” and “Consensus-2” DSS, appended with special add-ons for aggregation of estimates, provided in different scales, and for defining of an optimal resource distribution (i.e., the one that maximizes the degree of strategic goal achievement), allow us to cover the whole spectrum of problems from strategic planning cycle only when used in combination. And still, certain aspects require improvement.

The review allows a DM, an expert session organizer, or a knowledge engineer, to select DS tools, which are most suitable for specific problems the expert session is intended to solve. 3

Some other aspects and directions of DS tools improvement

We feel, that there are some other aspects of DS process, which remain beyond the scope of our overview, but still need to be taken into consideration during improvement of existing and development of new DS tools. Let us briefly list these aspects. 1) Defining the relative competence of experts during group expert sessions. In order to define the relative competence of experts in each of the issues under consideration, we should be able to take several components into account: objective component, mutual estimate, and self-estimate of expert group members, as well as the quality of information, provided by the expert during a specific session [ 40 ]. 2) Ordinal-data-based decision-making techniques – for cases when experts can provide only ordinal comparisons of alternatives [ 41,42 ]. 3) Consideration of peculiarities of expert data collection [ 7 ], such as human psychophysiological constraints [ 43 ], requirement of keeping the estimates within one order of magnitude, defining of the necessary number of pair-wise comparisons to be performed (if estimates are provided in the form of pair-wise comparisons) [ 44 ], opportunity to change the sequence of alternative presentation to the expert for comparison [ 45 ]. 4) Opportunity to organize feedback with experts in order to improve the quality (consistency, compatibility, completeness, and detail) of expert information [ 46 ]. 5) Opportunity to consider not only expert but also open-data-based information in the DSS [ 47 ]. This information is directly related to the level of development of natural language processing (NLP) tools, used for DS. 6) Opportunity to analyze situation development scenarios under changing impact of this or that factor. Scenario analysis can be based on sensitivity analysis principles. 4

Conclusions

The paper presents a brief overview and comparative analysis of available DS tools from the standpoint of their applicability to strategic planning problem-solving in weakly structured subject domains.

It has been shown, that not a single automated DSS, featured in the review, can solve all the range of problems, emerging in the process of strategic planning based on expert and other information. We have suggested a set of constructive requirements to modern DS tools. We have obtained a set of recommendations on selection of software tools for solving DS problems in the process of strategic planning, as well as on improvement of existing (and development of new) DSS.

In spite of increasing volumes of available open data, that can be used for DS, the relevance of tasks, associated with obtaining and processing of expert information, and, especially, expert knowledge, is growing. Consequently, the problem of development and improvement of DS tools, allowing to obtain and process both expert and other information most thoroughly and without distortions, also remains relevant.

The most promising directions of further research on the paper’s topic include improvement of DSS tools’ features, related to organization of remote expert sessions and incorporation of data on subject domain from all available sources.

References

1. Kadenko , S. : Prospects and Potential of Expert Decision-making Support Techniques Implementation in Information Security Area . In CEUR Workshop Proceedings . Vol- 1813 , pp. 8 - 14 ( 2016 ).

2. Saaty T. : Decision Making with Dependence and Feedback: The analytic Network Process . RWS Publicaitons , Pittsburgh ( 1996 ).

3. Totsenko , V. Methods and Systems of Decision-making Support. Algorithmic Aspect (in Russian) . Kyiv, Naukova Dumka ( 2002 ).

4. Tsyganok V.V. , Kadenko S.V. , Andriichuk O.V. Using Different Pair-wise Comparison Scales for Developing Industrial Strategies . Int. J. Management and Decision Making . Vol. 14 , No 3. - pp. 224 - 250 ( 2015 ).

5. Balagura I. , Kadenko S. , Andriichuk O. , Gorbov I . Defining Potential Academic Expert Groups based on Joint Authorship Networks Using Decision Support Tools. Selected Papers of the XIX International Scientific and Practical Conference on Information Technologies and Security (ITS 2019) Kyiv 2019 - pp. 222 - 233 ( 2019 ).

6. Tsyganok , V. , Kadenko , S. , Andriichuk , O. , Roik , P. : Usage of multicriteria decision-making support arsenal for strategic planning in environmental protection sphere . Journal of Multi-criteria Decision Analysis , 24 ( 5-6 ), 227 - 238 ( 2017 ).

7. Dodonov , A. , Lande , D. , Tsyganok , V. , Andriichuk , O. , Kadenko , S. , Graivoronskaya , A. : Information Operations Recognition. From Nonlinear Analysis to Decision-Making . Lambert Academic Publishing. ( 2019 ).

8. Zgurovsky , M. Scenario Analysis as a System Foresight Methodology (in Ukrainian) . System Research and Information Tecnologies, i.1 , pp. 7 - 38 ( 2002 ).

9. Podberyozkin , A. Probable scenario of international situation development after 2021 (in Russian) . M., MGIMO-UNiversitet ( 2015 ).

10. Applications and Theory of Analytic Hierarchy Process. Decision Making for Strategic Decisions . De Felice, F . (Ed.). IntechOpen, ( 2016 ).

11. Power , D. J. Decision support systems: concepts and resources for managers . Westport, Conn., Quorum Books ( 2002 ).

12. Super Decisions . Available online at: http://www.superdecisions.com/. Accessed 30.11 . 2020 .

13. Expert Choice Desktop: Powerful Performance for Organizational Decision-Making . Available online at: http://expertchoice.com/products-services/ expert-choice-desktop . Accessed 30 . 11 . 2020 .

14. ISAHP proceedings: Available online at: http://www.isahp.org/proceedings/. Accessed 30.11 . 2020 .

15. Mareschal B. , Brans J.-P. "PROMETHEE Methods " , Ch 5 in: Figueira, J. , Greco , S. , and Ehrgott , M., eds., Multiple Criteria Decision Analysis: State of the Art Surveys Series / New York: Springer, ( 2005 ).

16. Decision-making Support System “1000minds” Available online at : http://www.1000minds.com/. Accessed 30.11 . 2020 .

17. Hansen P. , Ombler F. A new method for scoring multi-attribute value models using pairwise rankings of alternatives . Journal of Multi-Criteria Decision Analysis. 15 , pp. 87 - 107 ( 2008 ).

18. Hovanov N. , Yudaeva M. , Hovanov K. Multicriteria estimation of probabilities on basis of expert non-numeric, non-exact and non-complete knowledge / European Journal of Operational Research . Vol. 195 , Issue 3. - P. 857 - 863 ( 2009 ).

19. Decision-making Support System “Analytica” Available online at: https://lumina.com/products/free101/. Accessed 30.11 . 2020 .

20. Decision-making Support System “D-sight ” Available online at: http://www.d-sight. com/. Accessed 30.11 . 2020 .

21. Decision-making Support System “MakeItRational” Available online at: http://makeitrational.com/. Accessed 30.11 . 2020 .

22. Decision-making Support System “MindDecider” Available online at: http://www.minddecider.com/. Accessed 30.11 . 2020 .

23. Decision-making Support System “Logical Decisions ” Available online at: http://www.logicaldecisions.com/. Accessed 30.11 . 2020 .

24. Decision-making Support System “Tree Age Pro ” Available online at: http://www.treeage. com. Accessed 30 . 11 . 2020 .

25. Decision-making Support System “RFP (Rational Focal Point )” Available online at: https://www.teamblue.unicomsi.com/products/focal-point/. Accessed 30.11 . 2020 .

26. Decision-making Support System “Very Good Choice” Available online at: https://mvlsoftvery-good-choice-add-in-for-exce . software.informer.com/download/. Accessed 30.11 . 2020 .

27. Roy B . Classement et choix en présence de points de vue multiples (la méthode ELECTRE) . La Revue d'Informatique et de Recherche Opérationnelle (RIRO) . N8 . - P. 57 - 75 . ( 1968 ).

28. Информационно-аналитическая система “ОЦЕНКА И ВЫБОР” (Information and analytical system “Estimation and Choice”) Available online at: http://www.deol.ru/users/DecisionSupporter/projects/iasctc.html. Accessed 30 . 11 . 2020 .

29. Demo Project - Compare car models 2001 -2012 / ESTIMATION & CHOICE Available online at: http://decisionsupporter.com/Projects.asp. Accessed 30 . 11 . 2020 .

30. Mikony , S. V. “SVIR'” - a system for selection and ranking. (in Russian) Proceedings of an international congress “Artificial intelligence in the XXI century” . Divnomorskoye 3- 8 . 09 . 2001 . Vol. 1 , pp. 500 - 507 ( 2001 ).

31. Mikony , S. V. , Burakov , D. P. , Sorokina , M. I. Implementation of ergonomics and intelligence principles in SVIR' system . Software products and systems, #3 , 28 - 32 ( 2002 ).

32. Solon- 3 DSS. Available online at: http://www.dss-lab. org.ua/Solon-3.htm. Accessed 30 . 11 . 2020 .

33. System "Consesnsus-2" for distributed collecting of expert information for DSS . Available online at: http://www.dss-lab. org.ua/Consensus-2.htm. Accessed 30 . 11 . 2020 .

34. Medical Sapiens platform . Available online at: https://www.medicalsapiens.cl/. Accessed 30.11 . 2020 .

35. Total Decision DSS . Available online at: https://vilenio.com/td_download. html. Accessed 30 . 11 . 2020 .

36. Decision Mentor app . Available online at: https://www.decisionmentor.app/. Accessed 30.11 . 2020 .

37. Best Decision Support Software | 2020 Reviews of the Most Popular Tools & Systems, Available online at: https://www.capterra.com.au/directory/30544/decision-support /software. Accessed 30 . 11 . 2020 .

38. Weistroffer , H.R. , Smith C.H. , and Narula , S.C. Multiple criteria decision support software . Ch 24 in: Figueira J., Greco S. and Ehrgott M. eds, Multiple Criteria Decision Analysis: State of the Art Surveys Series / N.Y.: Springer, ( 2005 ).

39. Buckshaw D . Decision analysis software survey / OR/MS Today . Volume 37 , # 5 . - P. 44 - 53 , ( 2010 ).

40. Kadenko , S. : Defining Relative Weights of Data Sources during Aggregation of Pair-wise Comparisons . In CEUR Workshop Proceedings . Vol- 2067 , pp. 47 - 55 ( 2017 ).

41. Tsyganok , V. , Kadenko , S. On sufficiency of the consistency level of group ordinal estimates . Journal of Automation and Information Sciences . 42 ( 8 ). P. 42 - 47 ( 2010 ).

42. Kadenko , S. Determination of Parameters of Criteria of" Tree" Type Hierarchy on the Basis of Ordinal Estimates . Journal of Automation and Information Sciences . 40 ( 8 ). P. 7- 15 ( 2008 ).

43. Miller G. A. The Magical Number Seven, Plus or Minus Two . The Psychological Review . 63 , 81 - 97 , ( 1956 ).

44. Wedley , William C. Fewer Comparisons - Efficiency via Sufficient Redundancy . Proceedings of the 10th International Symposium for the Analytic Hierarchy Process - Pittsburgh , PA, July 30 - August 3 , 2009 . Available online at: http://www.isahp.org/2009Proceedings/Final_Papers/94_Wedley_Fewer_ Comparisons_REV_FIN.pdf . Accessed 30 . 11 . 2020 .

45. Andriichuk , O. , Tsyganok , V. , Kadenko , S. , and Porplenko , Y. Experimental Research of Impact of Order of Pairwise Alternative Comparisons upon Credibility of Expert Session Results . Proceedings of 2020 IEEE 2nd International Conference on System Analysis & Intelligent Computing (SAIC) . Kyiv , 5 - 9 Oct., 2020 . DOI: 10 .1109/SAIC51296. 2020 . 9239126 ( 2020 ).

46. Tsyganok V. , Kadenko S. , Andriichuk O. , and Roik P. Combinatorial Method for Aggregation of Incomplete Group Judgments . Proceedings of 2018 IEEE 1st International Conference on System Analysis & Intelligent Computing (SAIC) . Kyiv , 08 - 12 Oct., 2018 . P. 25 - 30 ( 2018 ).

47. Tsyganok , V. , Kadenko , S. , and Andriichuk , O. Hybrid Decision Support Methodology Based on Objective and Expert Data . 2020 IEEE 11th International Conference on Dependable Systems, Services and Technologies (DESSERT) , Kyiv, Ukraine, 2020 , pp. 265 - 271 ( 2020 ).