=Paper= {{Paper |id=Vol-2763/CPT2020_paper_s4-3 |storemode=property |title=About methods of support for management decision-making under conditions of significant uncertainty |pdfUrl=https://ceur-ws.org/Vol-2763/CPT2020_paper_s4-3.pdf |volume=Vol-2763 |authors=V.I.Baluta,V.P.Osipov,O.Y.Yakovenko }} ==About methods of support for management decision-making under conditions of significant uncertainty== https://ceur-ws.org/Vol-2763/CPT2020_paper_s4-3.pdf
      About methods of support for management decision-making under
                   conditions of significant uncertainty
                                             V.I.Baluta1,2, V.P.Osipov1, O.Y.Yakovenko3
                                  Vbaluta@yandex.ru | Osipov@keldysh.ru | Javdet@gmail.com
                                   1
                                    Keldysh Institute of Applied Mathematics, Moscow, Russia;
                                 2
                                   Plekhanov Russian University of Economics, Moscow, Russia;
                                 3
                                   Institute of applied modeling and forecasting, Moscow, Russia

    Attempts to improve the quality of managerial decisions by introducing modern advances in information technology in various areas
of public administration (socio-humanitarian, strategic, foreign policy, etc.) do not give the desired effect, comparable to the effect of
their implementation in the manufacturing sector. The solution to this problem requires qualitatively new approaches to the issues of
information and analytical support for decision-making in conditions of significant uncertainty. This article highlights the difficulties of
predictive management of social processes based on direct computer modeling of social systems, considers the disadvantages of expert
decision support methods used in practice, and proposes new technologies for applying expert knowledge and competencies based on
the use of computer modeling and research. Current approaches to working with experts can be described as methods of coordinating
the opinions of a group of experts based on their personal views (models) on the issue under discussion. Our experience has shown that
creating a common integrated model by a group of experts gives a much better result. While the traditional approach can be called
"group" intelligence, the new approach is called "collective" intelligence. In addition, methods of decision support using artificial
intelligence systems are currently being intensively developed. We propose to begin work on the creation of "hybrid" intelligence with
the integration of these approaches to obtain a synergistic effect.
    Keywords: public administration, decision support, computer modeling, expert judgement, artificial intelligence, group intelligence,
collective intelligence, hybrid intelligence

                                                                         development of general society and its institutions, to
1. Introduction                                                          predict the events and reactions of people, and therefore
    Many researchers note the fundamental problem of                     cannot serve as a support in the development of managerial
insufficient compliance of traditionally slow-evolving                   decisions at the state level. Modern systems of government
public administration systems with modern challenges of                  are forced to function in conditions of a significant level
world development.                                                       of uncertainty, which is due to the high dynamics and poor
    Large-scale institutional transformations of the world               predictability of possible trajectories of changes in the
community in the course of scientific and technological                  current situation, affecting the dynamics of current
progress have formed and continue to modify the new                      priorities in the military-political, economic, social and
reality of human and society existence. An increase in the               humanitarian spheres.
number of connections and relations into which each                          Attempts to compensate for an insufficient
element of the social world order is somehow interwoven                  understanding of the phenomena occurring in these areas
due to the processes of globalization and information and                with data volumes, as a rule, lead more to overloading of
technological development increases the complexity of                    management systems than to improving the quality of
developing and making managerial decisions by orders of                  decision-making.
magnitude. The empirical knowledge previously                                In fact, modern civilization challenges can be
accumulated in the field of social sciences and humanities               interpreted as a consequence of the growing gap, on the
in the new conditions is of little use, since it does not allow          one hand, between the demands of social development
us to explain with sufficient degree of certainty the nature             and, on the other hand, the capabilities of management
and tendencies of the changes that are taking place, derive              systems to develop and implement adequate solutions (fig.
their regularities and predict the further course of                     1).




                                       Fig. 1. Generalized view of public process management problems




Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY
4.0)
                                                                the field of applied social and humanitarian research in
2. The State of Research                                        domestic science, several points of concentration of
     As a subject of scientific research, the field of public   advanced scientific thought can be noted. So, at the
administration appears as a system of tasks of developing       Institute of Management Problems of the Russian
managerial decisions and tasks of reproduction and              Academy of Sciences, quite a lot of attention is paid to the
development of management mechanisms in relation to             development of general theoretical approaches to
certain areas and directions, for example, such as politics,    modeling various aspects of managing complex systems
economics, defense, production, international relations,        under the leadership of D. Novikov, methods for solving
society etc. Obviously, each of these areas has its own         practical problems in the face of uncertainty - by V.
specificity, its own range of problematic issues and            Kulba's group, substantiation of the theoretical
research methods, its own forms and methods of analyzing        foundations of decision-making - by the group of
the situation, developing and implementing solutions. The       Aleskerov F. et al. The approaches to a comprehensive
key tools for analyzing, developing and supporting the          review of socio-political, economic, demographic
implementation of solutions to date are various means of        processes in their mutual influence are being developed by
monitoring and visualizing data about the situation,            a team of researchers at the Plekhanov Russian University
including partially processed by relatively simple              of Economics led by A. Kugaenko.
analytical methods.
                                                                3. Basic approaches
     In addition to fairly simple analytical processing tools
for monitoring data and their visual presentation forms,            According to management theory, there are three basic
recently in concepts and development plans for decision         methodologies for improving the efficiency of developing
support systems, much attention has been paid to the            managerial decisions in relation to social and economic
development of mathematical modeling methods and                processes. These include full-scale experiments, expert
various approaches to creating simulation models of             assessments and mathematical modeling [26].
complex controlled systems. Based on such models, it is             In the practice of public administration, targeted field
supposed to conduct research and forecasting the behavior       experiments are usually used to test the effectiveness in
of the corresponding prototypes in various situations. As       local pilot areas of any regulatory innovations or proposed
the analysis of literary sources shows, the level of            solutions in order to study the likely consequences of their
development of modeling forms in various areas of public        implementation. Although it is believed that field
administration differs in their specificity and prevalence:     experiments are most reliable, the cases of their
if the economy has been using predominantly econometric         application (unless, of course, the traditional “trial and
and statistical models for a long time, and the military is     error method” is referred to them) are extremely limited,
persistently making attempts to introduce simulation, then      in particular, they are unacceptable in crisis or urgent
in social the humanitarian sphere still focuses on scenario     situations, and most cases and simply impossible.
modeling, and, as a rule, at the verbal level.                      An expert study of a problem situation is characterized
     Based on a relatively large number of publications of      by the fact that general information about the situation is
the corresponding orientation in foreign countries, we can      limited to the expert’s personal knowledge. Although
conclude that at present, there are multiple attempts to        expert knowledge has the important property of focusing
comprehend the possibilities and limits of the application      on individual groups of alternatives, it is characterized by
of traditional and new technologies of model research in        an extremely high level of subjectivity and limitedness
order to increase management efficiency in conditions of        simply because of the characteristics of human thinking.
significant uncertainty.                                        To increase confidence in expert conclusions, groups of
     We can distinguish several among the most intensively      experts are usually involved, but the factor of subjectivity
developing areas modeling in the world. One of the main         is not eliminated in that case.
methods for analyzing the political situation remains               Model studies, as a rule, are associated with the
models based on the application of the principles of            formalization of the description of the control object and
equilibrium [1-3]. Methods based on the agent approach          the current situation, the selection of criteria for the
are getting a great deal of development, including their        adequacy of modeling. A direct study of the situation on
variety - multi-agent modeling (as a rule, requiring            the model with an assessment of the possible
significant computational resources, as well as significant     consequences ends with an interpretation of the simulation
costs for collecting and preparing data for modeling for        results to redistribute the preference of alternatives.
practical purposes) [4-6]. The method of system dynamics            As the capabilities of computer technology and the
remains quite popular, which over the past decades has          development of model research methods have grown, the
acquired many varieties, such as the use of technology of       field of applicability of the results of model studies of
cognitive maps, fuzzy modeling, etc. [7-19].                    control objects and the environment has significantly
Unfortunately, we have not come across work with the            expanded to solve the problems of managing complex
idea of simultaneous research of one object using a             systems such as the state, economy, armed forces, etc.
scientifically based of different models, which does not            Thus, in practice, there is a competition between two
allow us to evaluate their comparative effectiveness.           fundamental approaches - an orientation toward the use of
Although some generalizing works can be mentioned that          information systems and ever-increasing complexity of
could form the basis of such an approach [20-25].               models or at attracting the experience and knowledge of
     Regarding the development of modeling methods in           experts. It is clear that this separation is somewhat
contingent, since domain experts also participate in the             experts uses their own ideas about the situation, on the
creation of models, and any expert in solving the problem            basis of which they form their own conclusions about its
relies, inter alia, on the results of a model representation of      development. To reduce the importance of the factor of
the subject of study.                                                specialization and subjectivity of experts, to date, many
                                                                     standard methods for rational construction of the processes
4. Problems of expert approaches                                     of formation of expert conclusions and conclusions by
    One of the main difficulties in applying expert                  expert groups have been proposed. All of them are, to one
approaches is the impossibility of a full-scale vision of the        degree or another, based on methods of reconciling the
analyzed picture in a holistic form by any of the experts            resulting opinions by selecting the most preferred of them
with the degree of detail that is inherent in each individual        for some reason (acceptability, validity, authority, etc.).
specialist in the field of his competencies. This does not           This approach allows us to reach consensus and increase
allow, within the framework of traditional approaches, a             the level of confidence in the results, however, despite the
qualitative analysis of multi-parameter problems with the            group nature of the discussion, in fact even the adjusted
complex nature of the interconnections within the systems            final conclusion is based on the limited vision of one
under consideration, which include most of the tasks of              expert with whom the others agree, which virtually
public administration. Moreover, it is obvious that the              eliminates the possibility of original solutions appearing.
impossibility of a holistic vision of the problem, situation,        In other words, the technology used in practice (fig. 2),
goals and objectives inevitably leads to a decrease in the           which we will arbitrarily call the technology of “group”
quality of management decisions.                                     intelligence, involves procedures for agreeing on
    Even when solving problems in a group, each of the               particular conclusions that do not allow going beyond the
                                                                     scope of voiced proposals and gaining new knowledge.




                              Fig. 2. Conditional presentation of an “agreed” group decision of experts

                                                                     many of the approaches currently being developed are in
5. Problems of modeling social systems                               line with such a mechanistic approach - in the direction of
    From a research point of view, society, society is a             the greatest possible detail in the model reproduction of
variety of large complex systems with many elements that             the system under consideration. As you know, the more
are in a certain relationship and connections but with a             complex the model, the more resources are needed for its
large number of degrees of freedom. For this reason,                 creation and validation. Moreover, the more complex and
various kinds of political, public, social, societal and             accurate the model, the more difficult and costly it is to
another processes are characterized by a significant degree          provide it with satisfactory input data, that is, in addition
of subjective uncertainty due to the high complexity of the          to the development itself, additional resources are also
behavior of individual elements of the system, the                   needed for collecting, validating, formalizing and entering
complexity of the mechanisms for linking the behavior of             large volumes of heterogeneous input data.
the system and its elements and the objective limitation of              When solving engineering, technical, and physical
the subject in perception and understanding of reality.              problems, where the behavior of modeling objects is
    To build a detailed model of such a system, it is                objectively limited by the scope of well-studied physical
necessary to describe the behavior of almost each of the             laws, or in the production sphere, which is regulated by
elements in all its relationships with others. Moreover,             strict restrictions on resource balances and the rules of
according to modern ideas about rational approaches to               interaction of elements that reduce the degree of
modeling, to obtain a qualitative model of the object of             arbitrariness, such approaches are justified and give
study, it is necessary to build a model of interaction of its        satisfactory results. But in humanitarian spheres, objective
components to the third structural level of the hierarchy of         laws and restrictions that reduce the degrees of freedom of
elements, which significantly increases the scale and                elements have not yet been established (if they exist for
complexity of the model.                                             such systems at all with respect to the time and place scales
    Due to the increasing capabilities of computing tools,           in question), so the creation of complex models for
describing social processes encounters insurmountable           summation or averaging of the finished conclusions of
difficulties.                                                   individual experts, but a synergistic summation of the
                                                                competencies of experts conducting a joint search
6. The lack alternate to expert methods                         solutions.
    If we go to the limit views, we can conclude that in the
                                                                7. The technology of "collective" intelligence
field of public administration for solving practical
problems it is impossible to build large-scale detailed             When performing several search and research works
models or provide them with initial data with details           aimed at creating tools for predictive management of
satisfying the necessary and sufficient requirements to         complex systems under significant uncertainty, a new
reduce the level of uncertainty in order to develop effective   technology was tested - the technology of “collective”
managerial of decisions.                                        intelligence, combining expert analysis and modeling of
    However, one cannot deny the importance of modern           complex systems (fig. 3).
methods of computer modeling for studying complex
systems, testing hypotheses, identifying obscure
relationships and dependencies, etc. In other words, if
direct modeling of specific processes in society is
objectively impossible, then the use of various forms of
modeling the systems under study in this area can be
effective, first of all, to build up expert experience and
knowledge.
    By the way, human evolution itself can serve as a
confirmation of the higher efficiency of expert approaches
in solving managerial problems in comparison with
numerical modeling. There are facts about the unique
abilities of individuals, for example, with absolute
memory, able to perform ultrafast and voluminous
calculations, etc. That is, the human brain is initially able
                                                                     Fig. 3. Conditional representation of the “collective”
to do what computers are doing more and more well. And
                                                                                    intelligence of experts
physiologists believe that the potential of the human brain
for certain types of information processing is almost
                                                                    The main contours of this technology were outlined in
unlimited. Nevertheless, evolution did not follow the path
                                                                the process of considering problematic issues of modeling
of improving the mechanisms of deep analysis and
                                                                and studying complex systems, searching for rational
miscalculation of details, but the path of developing
                                                                (primarily in terms of mathematical optimality) solutions
mechanisms for a “broad” assessment of situations for an
                                                                and predictive evaluation of their performance. The
operational response in difficult conditions with high
                                                                accompanying result of these works was a complex of
uncertainty.
                                                                various research and service software tools, some of which
    Strange as it may seem, despite the intensive
                                                                have found application in various areas of solving
development of multi-agent technologies, the prospects
                                                                managerial problems, allowing to ensure their
for the development of decision-making support systems
                                                                implementation at a high scientific and technical level.
based on computer technologies are also seen in the
                                                                During the development, the foundations and the
creation of “artificial intelligence”, that is, imitating the
                                                                procedure for applying the technology were laid, its
intellectual functions of a person in terms of “wide” and
                                                                operability was verified, and its higher efficiency was
operational analysis of reality and the development of
                                                                confirmed in comparison with other expert methods for
rational management decisions in conditions of high
                                                                solving a certain range of tasks.
uncertainty. From the above it follows that at present, the
                                                                    The general algorithm of work using the technology of
use of expert approaches in management systems is seen
                                                                "collective" intelligence involves the following set of
as the most rational direction for the development of
                                                                interdependent procedures: analysis and classification of a
decision support systems. The only question is how to
                                                                problematic issue; determination of the required
most effectively use the progress in the development of
                                                                composition of professional and target competencies of
modern computer technologies to increase the
                                                                the expert group; formation of a team of experts; teaching
effectiveness of the application of these approaches and
                                                                them the rules of technology; organization of coordinated
their further improvement.
                                                                interaction, including verification of psychological
    It seems that now the society is in a situation where a
                                                                compatibility, distribution of roles, coordination of
transition to new, more effective forms of information
                                                                understanding of goals and perception of the subject of
support for management is needed, based on a
                                                                research; assignment of indicators and criteria of decision
combination of the use of expert knowledge with the
                                                                quality; the construction of a generalized cognitive model,
possibilities of modeling situations, as well as on ways to
                                                                as well as the selection of ready-made or the creation of a
“objectify” the subjective knowledge of experts . In our
                                                                set of express models of various subsystems and
opinion, a transition from “group” intelligence to
                                                                subprocesses; study of the model for consistency and
“collective” is needed, when in the process of analyzing
                                                                consistency; conducting iterative studies on model
the situation and making decisions, there is not a simple
                                                                examples; adjustment and refinement of the model;
conducting experiments; comparison, analysis and                    “artificial” intelligence technologies.
interpretation of results; preparation of an opinion. It is             As noted above, it is advisable to use the modern
clear that, depending on the situation, some deviations             capabilities of modeling complex systems to build up
from the regulations described here are possible in the             expert knowledge [27, 28]. One of the forms of using
implementation process.                                             “artificial” intelligence can be the use of an electronic
    As a result, a look at the constituent elements of the          training ground with a model of the complex system under
model and the forms of their relationship with respect to           study deployed on it and constantly being improved. As an
the competencies of each expert remains subjective,                 example, we can cite the results presented in [28, 29],
however, the complex model obtained as a result of                  which relate to the methodology of training an intelligent
collective work has the properties of an independent object         agent at an “electronic training ground” with a layout of
alienated from personalities, on the basis of which it is           the environment that allows, during many experiments, to
possible to study the behavior of a controlled system for           determine the mechanisms of formation of rational
subsequent collective analysis of the resulting results.            trajectories of movement to a target in the phase space of
    To some extent, here we can draw an analogy with the            an agent’s state from an arbitrary provisions.
method of “abstracting from the problem” within the                     After training at the training ground and introducing
framework of the so-called “Harvard School principal                the function of assessing one’s current position by an
negotiations”, the substantive essence of which is not a            arbitrary number of features, one can use the rules
direct search for a compromise, but preliminary agreement           developed at the “training ground” to make decisions in
on the principles of its search.                                    the current situation, including the situation understood
                                                                    with a certain degree of uncertainty. Considering that an
8. Technology of "hybrid" intelligence                              “electronic polygon” is still some abstract model
    The technology of the "collective" intellect is                 representation of the environment, it is advisable to
considered as an intermediate stage in the development of           combine both the construction of the model on polygon
decision support systems. It is expected that as the                and the recommendations resulting from its application
technology of “artificial” intelligence develops, the               with expert assessment by specialists. Thus, with the help
“collective” intelligence described here can evolve into            of artificial intelligence technologies, it is possible to
some form of “hybrid” intelligence, forming a symbiosis             expand the knowledge base on the basis of which
with such properties that will ensure that the control              alternatives are evaluated and recommendations are made
system meets the new challenges of civilizational                   on the most rational actions in the face of uncertainty. One
development. The possibility and pace of evolutionary               of the possible versions of the model of the “hybrid”
development of management information support methods               intellect based on the “collective” and “artificial” is
in the direction of “hybrid” forms will be substantially            presented in fig. 4.
determined by the progress in the development of




                      Fig 4. A possible scheme of the symbiosis of technologies based on hybrid intelligence

                                                                    to the search for compromises that are oriented towards
9. Conclusion                                                       traditional approaches for coordinating expert opinions.
    It can be summarized that in order to increase the                  The originality of the proposed approach lies in the
effectiveness of information and analytical support for             appropriate combination of social engineering methods
government in the face of significant uncertainty, it is            with modern advances in the field of information
necessary to develop methodological foundations,                    technology. Implementation of the technology of “hybrid”
principles of construction and architecture of a hybrid             intelligence will allow to obtain the results of an expert
automated decision support system by synthesizing the               assessment of the situation and alternatives by
collective intelligence of experts with artificial                  synergistically integrating expert knowledge with the
intelligence based on methods computer modeling of                  capabilities of “artificial” intelligence.
complex systems and solving optimization problems. The
                                                                    References
development of forms of expert management support
should be focused on obtaining qualitatively new results            [1] Bischoff, I., and L.-H. Siemers. (2011). Biased Beliefs
through the integration of expert competencies, as opposed              and Retrospective Voting: Why Democracies Choose
     Mediocre Policies. Public Choice 156: 163-180.                Dynamic Model of the Rivalry Process.” American
[2] Young, J. 2005. Research, Policy and Practice: Why             Journal of Political Science 55, no. 2 (2011): 263-75.
     Developing Countries Are Different. Journal of           [20] Burton, R. M., & Obel, B. (2011). Computational
     International Development 17: 727-734.                        modeling for what-is, what-might-be, and what-
[3] Rausser, G.C., J. Swinnen, and P. Zusman. (2011).              should-be studies-and triangulation. Organization
     Political Power and Economic Policy: Theory,                  Science, 22(5), 1195-1202.
     Analysis, and Empirical Applications. Cambridge:         [21] Chiu, C.-Y., & Qiu, L. (2014). Communication and
     Cambridge University Press.                                   culture: A complexity theory approach. Asian Journal
[4] Allen, T. T. (2011). Introduction to discrete event            of Social Psychology, 17, 108-111
     simulation and agent-based modeling: voting              [22] Cioffi-Revilla,     C.    (2017).     Introduction   to
     systems, health care, military, and manufacturing.            computational social science: Principles and
     London, UK: Springer                                          applications (2nd ed.). New York, NY: Springer.
[5] Blais, A., Erisen, C., & Rheault, L. (2014). Strategic    [23] Clark, W. R., & Golder, M. (2015). Big data, causal
     voting and coordination problems in proportional              inference, and formal theory: Contradictory trends
     systems: An experimental study. Political Research            in political science? PS: Political Science & Politics,
     Quarterly, 67(2), 386-397                                     48(1), 65-70
[6] Fieldhouse, E., Lessard-Phillips, L., & Edmonds, B.       [24] Saaty T. Theory and Aplications of the Analytic
     (2016). Cascade or echo chamber? A complex agent-             Network Process: Decision Making with Benefits,
     based simulation of voter turnout. Party Politics,            Opportunities, Cost and Risk. - Pittsburgh, RWS
     22(2), 241-256                                                Publications, 2009.
[7] Fisunoğlu, A. (2019). System Dynamics Modeling in         [25] Casey K.L. Defining Political Capital: A
     International Relations. All Azimuth: A Journal of            Reconsideration of Bourdieu’s Interconvertibility
     Foreign Policy and Peace, 8 (2), 231-253. DOI:                Theory // Paper University of Missouri. - St. Louis,
     10.20991/allazimuth.476884                                    2007.
[8] Abdollahian, Mark A., et al. “Dynamics of Cultural        [26] Petrovsky, A.B.: Decision Making Theory.
     Change: The Human Development Perspective.”                   Publishing Center Academiya, Moscow (2009). in
     International Studies Quarterly 56, no. 4 (2012): 827-        Russian
     42.                                                      [27] Baluta V.I., Osipov V.P., Yakovenko O.Yu. The
[9] Akgul, Oner. “A Bibliographical Study on the                   environment of modeling, forecasting and expertise as
     Academic Research of Peace and Conflict in Turkey.”           the intellectual core of support management complex
     Paper presented at the 7th Eurasian Peace Science             systems. KIAM Preprint № 82, Moscow, 2015
     Meeting, İstanbul, 2018.                                 [28] M.A.Berberova, K.I.Chernyavskii, «Comparative
[10] Chamberlain, Todd. “Systems Dynamics Model of                 assessment of the NPP risk (on the example of Rostov
     Al-Qa’ida and United States ‘Competition’.” Journal           and Kalinin NPP). Development of risk indicators
     of Homeland Security and Emergency Management                 atlas for Russian NPPs», GraphiCon 2019 Computer
     4, no. 3 (2007): 1-23.                                        Graphics and Vision. The 29th International
[11] Chen, Yu-Che, and Michael J. Ahn. Routledge                   Conference on Computer Graphics and Vision.
     Handbook on Information Technology in                         Conference Proceedings (2019), Bryansk, Russia,
     Government. New York: Routledge, 2017.                        September 23-26, 2019, Vol-2485, urn:nbn:de:0074-
[12] Demir, Cenker Korhan, and Engin Avci. “Turkish                2485-1, ISSN 1613-0073, DOI: 10.30987/graphicon-
     Terrorism Studies: A Preliminary Assessment.” All             2019-2-290-294,                 http://ceur-ws.org/Vol-
     Azimuth 7, no.1 (2018): 21-44.                                2485/paper67.pdf, p. 290-294.
[13] Forrester, Jay W. Industrial Dynamics. Cambridge:        [29] Baluta V.I., Osipov V.P., Chetverushkin B.N.,
     The M.I.T. Press, 1961.                                       Yakovenko O.Y. Simulation of dynamics and
[14] Ghaffarzadegan, Navid, John Lyneis, and George P.             adaptation in the space of conflict interaction of the
     Richardson. “How Small System Dynamics Models                 intelligent agents. KIAM Preprint № 111, Moscow,
     Can Help the Public Policy Process.” System                   2019.
     Dynamics Review 27, no. 1 (2011): 22-44.
[15] Goertz, Gary, and James Mahoney. A Tale of Two           About the autors
     Cultures: Qualitative and Quantitative Research in the       Baluta Viktor I. - Candidate of Technical Sciences, senior
     Social Sciences. Princeton: Princeton University         researcher, Keldysh Institute of applied mathematics of the
     Press, 2012. 253 System Dynamics.                        Russian Academy of Sciences, Plekhanov Russian University of
[16] Hjorth, Peder, and Ali Bagheri. “Navigating Towards      Economics. E-mail: Vbaluta@yandex.ru
     Sustainable Development: A System Dynamics                   Osipov Vladimir P. - Candidate of Technical Sciences,
     Approach.” Futures 38, no. 1 (2006): 74-92.              leading researcher Institute of applied mathematics. M. V.
                                                              Keldysh, RAS, E-mail: Osipov@keldysh.ru
[17] Jervis, Robert. System Effects: Complexity in
                                                                  Yakovenko Oleg Yu., Leading Researcher. Institute of
     Political and Social Life. Princeton: Princeton
                                                              applied modeling and forecasting. E-mail: Javdet@gmail.com
     University Press, 1997.
[18] Kirkwood, Craig W. System Dynamics Methods: A
     Quick Introduction. College of Business, Arizona
     State University, 1998.
[19] Morey, David S. “When War Brings Peace: A