<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>About methods of support for management decision-making under conditions of significant uncertainty</article-title>
      </title-group>
      <contrib-group>
        <aff id="aff0">
          <label>0</label>
          <institution>Baluta Viktor I. - Candidate of Technical Sciences, senior researcher, Keldysh Institute of applied mathematics of the Russian Academy of Sciences, Plekhanov Russian University of Economics</institution>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Institute of applied modeling and forecasting</institution>
          ,
          <addr-line>Moscow</addr-line>
          ,
          <country country="RU">Russia</country>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>Keldysh Institute of Applied Mathematics</institution>
          ,
          <addr-line>Moscow</addr-line>
          ,
          <country country="RU">Russia</country>
        </aff>
        <aff id="aff3">
          <label>3</label>
          <institution>Plekhanov Russian University of Economics</institution>
          ,
          <addr-line>Moscow</addr-line>
          ,
          <country country="RU">Russia</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Attempts to improve the quality of managerial decisions by introducing modern advances in information technology in various areas of public administration (socio-humanitarian, strategic, foreign policy, etc.) do not give the desired effect, comparable to the effect of their implementation in the manufacturing sector. The solution to this problem requires qualitatively new approaches to the issues of information and analytical support for decision-making in conditions of significant uncertainty. This article highlights the difficulties of predictive management of social processes based on direct computer modeling of social systems, considers the disadvantages of expert decision support methods used in practice, and proposes new technologies for applying expert knowledge and competencies based on the use of computer modeling and research. Current approaches to working with experts can be described as methods of coordinating the opinions of a group of experts based on their personal views (models) on the issue under discussion. Our experience has shown that creating a common integrated model by a group of experts gives a much better result. While the traditional approach can be called "group" intelligence, the new approach is called "collective" intelligence. In addition, methods of decision support using artificial intelligence systems are currently being intensively developed. We propose to begin work on the creation of "hybrid" intelligence with the integration of these approaches to obtain a synergistic effect.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>Many researchers note the fundamental problem of
insufficient compliance of traditionally slow-evolving
public administration systems with modern challenges of
world development.</p>
      <p>Large-scale institutional transformations of the world
community in the course of scientific and technological
progress have formed and continue to modify the new
reality of human and society existence. An increase in the
number of connections and relations into which each
element of the social world order is somehow interwoven
due to the processes of globalization and information and
technological development increases the complexity of
developing and making managerial decisions by orders of
magnitude. The empirical knowledge previously
accumulated in the field of social sciences and humanities
in the new conditions is of little use, since it does not allow
us to explain with sufficient degree of certainty the nature
and tendencies of the changes that are taking place, derive
their regularities and predict the further course of
development of general society and its institutions, to
predict the events and reactions of people, and therefore
cannot serve as a support in the development of managerial
decisions at the state level. Modern systems of government
are forced to function in conditions of a significant level
of uncertainty, which is due to the high dynamics and poor
predictability of possible trajectories of changes in the
current situation, affecting the dynamics of current
priorities in the military-political, economic, social and
humanitarian spheres.</p>
      <p>Attempts to compensate for an insufficient
understanding of the phenomena occurring in these areas
with data volumes, as a rule, lead more to overloading of
management systems than to improving the quality of
decision-making.</p>
      <p>In fact, modern civilization challenges can be
interpreted as a consequence of the growing gap, on the
one hand, between the demands of social development
and, on the other hand, the capabilities of management
systems to develop and implement adequate solutions (fig.
1).</p>
      <p>Fig. 1. Generalized view of public process management problems</p>
    </sec>
    <sec id="sec-2">
      <title>2. The State of Research</title>
      <p>As a subject of scientific research, the field of public
administration appears as a system of tasks of developing
managerial decisions and tasks of reproduction and
development of management mechanisms in relation to
certain areas and directions, for example, such as politics,
economics, defense, production, international relations,
society etc. Obviously, each of these areas has its own
specificity, its own range of problematic issues and
research methods, its own forms and methods of analyzing
the situation, developing and implementing solutions. The
key tools for analyzing, developing and supporting the
implementation of solutions to date are various means of
monitoring and visualizing data about the situation,
including partially processed by relatively simple
analytical methods.</p>
      <p>In addition to fairly simple analytical processing tools
for monitoring data and their visual presentation forms,
recently in concepts and development plans for decision
support systems, much attention has been paid to the
development of mathematical modeling methods and
various approaches to creating simulation models of
complex controlled systems. Based on such models, it is
supposed to conduct research and forecasting the behavior
of the corresponding prototypes in various situations. As
the analysis of literary sources shows, the level of
development of modeling forms in various areas of public
administration differs in their specificity and prevalence:
if the economy has been using predominantly econometric
and statistical models for a long time, and the military is
persistently making attempts to introduce simulation, then
in social the humanitarian sphere still focuses on scenario
modeling, and, as a rule, at the verbal level.</p>
      <p>Based on a relatively large number of publications of
the corresponding orientation in foreign countries, we can
conclude that at present, there are multiple attempts to
comprehend the possibilities and limits of the application
of traditional and new technologies of model research in
order to increase management efficiency in conditions of
significant uncertainty.</p>
      <p>
        We can distinguish several among the most intensively
developing areas modeling in the world. One of the main
methods for analyzing the political situation remains
models based on the application of the principles of
equilibrium [
        <xref ref-type="bibr" rid="ref1 ref2 ref3">1-3</xref>
        ]. Methods based on the agent approach
are getting a great deal of development, including their
variety - multi-agent modeling (as a rule, requiring
significant computational resources, as well as significant
costs for collecting and preparing data for modeling for
practical purposes) [
        <xref ref-type="bibr" rid="ref4 ref5 ref6">4-6</xref>
        ]. The method of system dynamics
remains quite popular, which over the past decades has
acquired many varieties, such as the use of technology of
cognitive maps, fuzzy modeling, etc. [
        <xref ref-type="bibr" rid="ref10 ref11 ref12 ref13 ref14 ref15 ref16 ref17 ref18 ref19 ref7 ref8 ref9">7-19</xref>
        ].
Unfortunately, we have not come across work with the
idea of simultaneous research of one object using a
scientifically based of different models, which does not
allow us to evaluate their comparative effectiveness.
Although some generalizing works can be mentioned that
could form the basis of such an approach [
        <xref ref-type="bibr" rid="ref20 ref21 ref22 ref23 ref24 ref25">20-25</xref>
        ].
      </p>
      <p>Regarding the development of modeling methods in
the field of applied social and humanitarian research in
domestic science, several points of concentration of
advanced scientific thought can be noted. So, at the
Institute of Management Problems of the Russian
Academy of Sciences, quite a lot of attention is paid to the
development of general theoretical approaches to
modeling various aspects of managing complex systems
under the leadership of D. Novikov, methods for solving
practical problems in the face of uncertainty - by V.
Kulba's group, substantiation of the theoretical
foundations of decision-making - by the group of
Aleskerov F. et al. The approaches to a comprehensive
review of socio-political, economic, demographic
processes in their mutual influence are being developed by
a team of researchers at the Plekhanov Russian University
of Economics led by A. Kugaenko.</p>
    </sec>
    <sec id="sec-3">
      <title>3. Basic approaches</title>
      <p>
        According to management theory, there are three basic
methodologies for improving the efficiency of developing
managerial decisions in relation to social and economic
processes. These include full-scale experiments, expert
assessments and mathematical modeling [
        <xref ref-type="bibr" rid="ref26">26</xref>
        ].
      </p>
      <p>In the practice of public administration, targeted field
experiments are usually used to test the effectiveness in
local pilot areas of any regulatory innovations or proposed
solutions in order to study the likely consequences of their
implementation. Although it is believed that field
experiments are most reliable, the cases of their
application (unless, of course, the traditional “trial and
error method” is referred to them) are extremely limited,
in particular, they are unacceptable in crisis or urgent
situations, and most cases and simply impossible.</p>
      <p>An expert study of a problem situation is characterized
by the fact that general information about the situation is
limited to the expert’s personal knowledge. Although
expert knowledge has the important property of focusing
on individual groups of alternatives, it is characterized by
an extremely high level of subjectivity and limitedness
simply because of the characteristics of human thinking.
To increase confidence in expert conclusions, groups of
experts are usually involved, but the factor of subjectivity
is not eliminated in that case.</p>
      <p>Model studies, as a rule, are associated with the
formalization of the description of the control object and
the current situation, the selection of criteria for the
adequacy of modeling. A direct study of the situation on
the model with an assessment of the possible
consequences ends with an interpretation of the simulation
results to redistribute the preference of alternatives.</p>
      <p>As the capabilities of computer technology and the
development of model research methods have grown, the
field of applicability of the results of model studies of
control objects and the environment has significantly
expanded to solve the problems of managing complex
systems such as the state, economy, armed forces, etc.</p>
      <p>Thus, in practice, there is a competition between two
fundamental approaches - an orientation toward the use of
information systems and ever-increasing complexity of
models or at attracting the experience and knowledge of
experts. It is clear that this separation is somewhat
contingent, since domain experts also participate in the
creation of models, and any expert in solving the problem
relies, inter alia, on the results of a model representation of
the subject of study.</p>
    </sec>
    <sec id="sec-4">
      <title>4. Problems of expert approaches</title>
      <p>One of the main difficulties in applying expert
approaches is the impossibility of a full-scale vision of the
analyzed picture in a holistic form by any of the experts
with the degree of detail that is inherent in each individual
specialist in the field of his competencies. This does not
allow, within the framework of traditional approaches, a
qualitative analysis of multi-parameter problems with the
complex nature of the interconnections within the systems
under consideration, which include most of the tasks of
public administration. Moreover, it is obvious that the
impossibility of a holistic vision of the problem, situation,
goals and objectives inevitably leads to a decrease in the
quality of management decisions.</p>
      <p>Even when solving problems in a group, each of the
experts uses their own ideas about the situation, on the
basis of which they form their own conclusions about its
development. To reduce the importance of the factor of
specialization and subjectivity of experts, to date, many
standard methods for rational construction of the processes
of formation of expert conclusions and conclusions by
expert groups have been proposed. All of them are, to one
degree or another, based on methods of reconciling the
resulting opinions by selecting the most preferred of them
for some reason (acceptability, validity, authority, etc.).
This approach allows us to reach consensus and increase
the level of confidence in the results, however, despite the
group nature of the discussion, in fact even the adjusted
final conclusion is based on the limited vision of one
expert with whom the others agree, which virtually
eliminates the possibility of original solutions appearing.
In other words, the technology used in practice (fig. 2),
which we will arbitrarily call the technology of “group”
intelligence, involves procedures for agreeing on
particular conclusions that do not allow going beyond the
scope of voiced proposals and gaining new knowledge.</p>
    </sec>
    <sec id="sec-5">
      <title>5. Problems of modeling social systems</title>
      <p>From a research point of view, society, society is a
variety of large complex systems with many elements that
are in a certain relationship and connections but with a
large number of degrees of freedom. For this reason,
various kinds of political, public, social, societal and
another processes are characterized by a significant degree
of subjective uncertainty due to the high complexity of the
behavior of individual elements of the system, the
complexity of the mechanisms for linking the behavior of
the system and its elements and the objective limitation of
the subject in perception and understanding of reality.</p>
      <p>To build a detailed model of such a system, it is
necessary to describe the behavior of almost each of the
elements in all its relationships with others. Moreover,
according to modern ideas about rational approaches to
modeling, to obtain a qualitative model of the object of
study, it is necessary to build a model of interaction of its
components to the third structural level of the hierarchy of
elements, which significantly increases the scale and
complexity of the model.</p>
      <p>Due to the increasing capabilities of computing tools,
many of the approaches currently being developed are in
line with such a mechanistic approach - in the direction of
the greatest possible detail in the model reproduction of
the system under consideration. As you know, the more
complex the model, the more resources are needed for its
creation and validation. Moreover, the more complex and
accurate the model, the more difficult and costly it is to
provide it with satisfactory input data, that is, in addition
to the development itself, additional resources are also
needed for collecting, validating, formalizing and entering
large volumes of heterogeneous input data.</p>
      <p>When solving engineering, technical, and physical
problems, where the behavior of modeling objects is
objectively limited by the scope of well-studied physical
laws, or in the production sphere, which is regulated by
strict restrictions on resource balances and the rules of
interaction of elements that reduce the degree of
arbitrariness, such approaches are justified and give
satisfactory results. But in humanitarian spheres, objective
laws and restrictions that reduce the degrees of freedom of
elements have not yet been established (if they exist for
such systems at all with respect to the time and place scales
in question), so the creation of complex models for
describing social processes encounters insurmountable
difficulties.</p>
    </sec>
    <sec id="sec-6">
      <title>6. The lack alternate to expert methods</title>
      <p>If we go to the limit views, we can conclude that in the
field of public administration for solving practical
problems it is impossible to build large-scale detailed
models or provide them with initial data with details
satisfying the necessary and sufficient requirements to
reduce the level of uncertainty in order to develop effective
managerial of decisions.</p>
      <p>However, one cannot deny the importance of modern
methods of computer modeling for studying complex
systems, testing hypotheses, identifying obscure
relationships and dependencies, etc. In other words, if
direct modeling of specific processes in society is
objectively impossible, then the use of various forms of
modeling the systems under study in this area can be
effective, first of all, to build up expert experience and
knowledge.</p>
      <p>By the way, human evolution itself can serve as a
confirmation of the higher efficiency of expert approaches
in solving managerial problems in comparison with
numerical modeling. There are facts about the unique
abilities of individuals, for example, with absolute
memory, able to perform ultrafast and voluminous
calculations, etc. That is, the human brain is initially able
to do what computers are doing more and more well. And
physiologists believe that the potential of the human brain
for certain types of information processing is almost
unlimited. Nevertheless, evolution did not follow the path
of improving the mechanisms of deep analysis and
miscalculation of details, but the path of developing
mechanisms for a “broad” assessment of situations for an
operational response in difficult conditions with high
uncertainty.</p>
      <p>Strange as it may seem, despite the intensive
development of multi-agent technologies, the prospects
for the development of decision-making support systems
based on computer technologies are also seen in the
creation of “artificial intelligence”, that is, imitating the
intellectual functions of a person in terms of “wide” and
operational analysis of reality and the development of
rational management decisions in conditions of high
uncertainty. From the above it follows that at present, the
use of expert approaches in management systems is seen
as the most rational direction for the development of
decision support systems. The only question is how to
most effectively use the progress in the development of
modern computer technologies to increase the
effectiveness of the application of these approaches and
their further improvement.</p>
      <p>It seems that now the society is in a situation where a
transition to new, more effective forms of information
support for management is needed, based on a
combination of the use of expert knowledge with the
possibilities of modeling situations, as well as on ways to
“objectify” the subjective knowledge of experts . In our
opinion, a transition from “group” intelligence to
“collective” is needed, when in the process of analyzing
the situation and making decisions, there is not a simple
summation or averaging of the finished conclusions of
individual experts, but a synergistic summation of the
competencies of experts conducting a joint search
solutions.</p>
    </sec>
    <sec id="sec-7">
      <title>7. The technology of "collective" intelligence</title>
      <p>When performing several search and research works
aimed at creating tools for predictive management of
complex systems under significant uncertainty, a new
technology was tested - the technology of “collective”
intelligence, combining expert analysis and modeling of
complex systems (fig. 3).</p>
      <p>Fig. 3. Conditional representation of the “collective”
intelligence of experts</p>
      <p>The main contours of this technology were outlined in
the process of considering problematic issues of modeling
and studying complex systems, searching for rational
(primarily in terms of mathematical optimality) solutions
and predictive evaluation of their performance. The
accompanying result of these works was a complex of
various research and service software tools, some of which
have found application in various areas of solving
managerial problems, allowing to ensure their
implementation at a high scientific and technical level.
During the development, the foundations and the
procedure for applying the technology were laid, its
operability was verified, and its higher efficiency was
confirmed in comparison with other expert methods for
solving a certain range of tasks.</p>
      <p>The general algorithm of work using the technology of
"collective" intelligence involves the following set of
interdependent procedures: analysis and classification of a
problematic issue; determination of the required
composition of professional and target competencies of
the expert group; formation of a team of experts; teaching
them the rules of technology; organization of coordinated
interaction, including verification of psychological
compatibility, distribution of roles, coordination of
understanding of goals and perception of the subject of
research; assignment of indicators and criteria of decision
quality; the construction of a generalized cognitive model,
as well as the selection of ready-made or the creation of a
set of express models of various subsystems and
subprocesses; study of the model for consistency and
consistency; conducting iterative studies on model
examples; adjustment and refinement of the model;
conducting experiments; comparison, analysis and
interpretation of results; preparation of an opinion. It is
clear that, depending on the situation, some deviations
from the regulations described here are possible in the
implementation process.</p>
      <p>As a result, a look at the constituent elements of the
model and the forms of their relationship with respect to
the competencies of each expert remains subjective,
however, the complex model obtained as a result of
collective work has the properties of an independent object
alienated from personalities, on the basis of which it is
possible to study the behavior of a controlled system for
subsequent collective analysis of the resulting results.</p>
      <p>To some extent, here we can draw an analogy with the
method of “abstracting from the problem” within the
framework of the so-called “Harvard School principal
negotiations”, the substantive essence of which is not a
direct search for a compromise, but preliminary agreement
on the principles of its search.</p>
    </sec>
    <sec id="sec-8">
      <title>8. Technology of "hybrid" intelligence</title>
      <p>The technology of the "collective" intellect is
considered as an intermediate stage in the development of
decision support systems. It is expected that as the
technology of “artificial” intelligence develops, the
“collective” intelligence described here can evolve into
some form of “hybrid” intelligence, forming a symbiosis
with such properties that will ensure that the control
system meets the new challenges of civilizational
development. The possibility and pace of evolutionary
development of management information support methods
in the direction of “hybrid” forms will be substantially
determined by the progress in the development of
“artificial” intelligence technologies.</p>
      <p>
        As noted above, it is advisable to use the modern
capabilities of modeling complex systems to build up
expert knowledge [
        <xref ref-type="bibr" rid="ref27 ref28">27, 28</xref>
        ]. One of the forms of using
“artificial” intelligence can be the use of an electronic
training ground with a model of the complex system under
study deployed on it and constantly being improved. As an
example, we can cite the results presented in [
        <xref ref-type="bibr" rid="ref28 ref29">28, 29</xref>
        ],
which relate to the methodology of training an intelligent
agent at an “electronic training ground” with a layout of
the environment that allows, during many experiments, to
determine the mechanisms of formation of rational
trajectories of movement to a target in the phase space of
an agent’s state from an arbitrary provisions.
      </p>
      <p>After training at the training ground and introducing
the function of assessing one’s current position by an
arbitrary number of features, one can use the rules
developed at the “training ground” to make decisions in
the current situation, including the situation understood
with a certain degree of uncertainty. Considering that an
“electronic polygon” is still some abstract model
representation of the environment, it is advisable to
combine both the construction of the model on polygon
and the recommendations resulting from its application
with expert assessment by specialists. Thus, with the help
of artificial intelligence technologies, it is possible to
expand the knowledge base on the basis of which
alternatives are evaluated and recommendations are made
on the most rational actions in the face of uncertainty. One
of the possible versions of the model of the “hybrid”
intellect based on the “collective” and “artificial” is
presented in fig. 4.</p>
      <p>Fig 4. A possible scheme of the symbiosis of technologies based on hybrid intelligence</p>
    </sec>
    <sec id="sec-9">
      <title>9. Conclusion</title>
      <p>It can be summarized that in order to increase the
effectiveness of information and analytical support for
government in the face of significant uncertainty, it is
necessary to develop methodological foundations,
principles of construction and architecture of a hybrid
automated decision support system by synthesizing the
collective intelligence of experts with artificial
intelligence based on methods computer modeling of
complex systems and solving optimization problems. The
development of forms of expert management support
should be focused on obtaining qualitatively new results
through the integration of expert competencies, as opposed
to the search for compromises that are oriented towards
traditional approaches for coordinating expert opinions.</p>
      <p>The originality of the proposed approach lies in the
appropriate combination of social engineering methods
with modern advances in the field of information
technology. Implementation of the technology of “hybrid”
intelligence will allow to obtain the results of an expert
assessment of the situation and alternatives by
synergistically integrating expert knowledge with the
capabilities of “artificial” intelligence.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <surname>Bischoff</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>L.-H. Siemers.</surname>
          </string-name>
          (
          <year>2011</year>
          ).
          <article-title>Biased Beliefs and Retrospective Voting: Why Democracies Choose Mediocre Policies</article-title>
          .
          <source>Public Choice</source>
          <volume>156</volume>
          :
          <fpage>163</fpage>
          -
          <lpage>180</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <surname>Young</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <year>2005</year>
          .
          <article-title>Research, Policy and Practice: Why Developing Countries Are Different</article-title>
          .
          <source>Journal of International Development</source>
          <volume>17</volume>
          :
          <fpage>727</fpage>
          -
          <lpage>734</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <surname>Rausser</surname>
            ,
            <given-names>G.C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>J. Swinnen</surname>
            , and
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Zusman.</surname>
          </string-name>
          (
          <year>2011</year>
          ).
          <article-title>Political Power and Economic Policy: Theory, Analysis, and Empirical Applications</article-title>
          . Cambridge: Cambridge University Press.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <surname>Allen</surname>
            ,
            <given-names>T. T.</given-names>
          </string-name>
          (
          <year>2011</year>
          ).
          <article-title>Introduction to discrete event simulation and agent-based modeling: voting systems, health care, military, and manufacturing</article-title>
          . London, UK: Springer
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <surname>Blais</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Erisen</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          , &amp;
          <string-name>
            <surname>Rheault</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          (
          <year>2014</year>
          ).
          <article-title>Strategic voting and coordination problems in proportional systems: An experimental study</article-title>
          .
          <source>Political Research Quarterly</source>
          ,
          <volume>67</volume>
          (
          <issue>2</issue>
          ),
          <fpage>386</fpage>
          -
          <lpage>397</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <surname>Fieldhouse</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lessard-Phillips</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          , &amp;
          <string-name>
            <surname>Edmonds</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          (
          <year>2016</year>
          ).
          <article-title>Cascade or echo chamber? A complex agentbased simulation of voter turnout</article-title>
          .
          <source>Party Politics</source>
          ,
          <volume>22</volume>
          (
          <issue>2</issue>
          ),
          <fpage>241</fpage>
          -
          <lpage>256</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <surname>Fisunoğlu</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          (
          <year>2019</year>
          ).
          <article-title>System Dynamics Modeling in International Relations</article-title>
          .
          <source>All Azimuth: A Journal of Foreign Policy and Peace</source>
          ,
          <volume>8</volume>
          (
          <issue>2</issue>
          ),
          <fpage>231</fpage>
          -
          <lpage>253</lpage>
          . DOI:
          <volume>10</volume>
          .20991/allazimuth.476884
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <surname>Abdollahian</surname>
            ,
            <given-names>Mark A.</given-names>
          </string-name>
          , et al. “
          <article-title>Dynamics of Cultural Change: The Human Development Perspective</article-title>
          .”
          <source>International Studies Quarterly</source>
          <volume>56</volume>
          , no.
          <issue>4</issue>
          (
          <year>2012</year>
          ):
          <fpage>827</fpage>
          -
          <lpage>42</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <surname>Akgul</surname>
          </string-name>
          , Oner. “
          <source>A Bibliographical Study on the Academic Research of Peace and Conflict in Turkey.” Paper presented at the 7th Eurasian Peace Science Meeting, İstanbul</source>
          ,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <surname>Chamberlain</surname>
          </string-name>
          , Todd. “
          <article-title>Systems Dynamics Model of Al-Qa'ida and United States 'Competition'</article-title>
          .
          <source>” Journal of Homeland Security and Emergency Management</source>
          <volume>4</volume>
          , no.
          <issue>3</issue>
          (
          <year>2007</year>
          ):
          <fpage>1</fpage>
          -
          <lpage>23</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <surname>Yu-Che</surname>
            , and
            <given-names>Michael J.</given-names>
          </string-name>
          <string-name>
            <surname>Ahn</surname>
          </string-name>
          .
          <source>Routledge Handbook on Information Technology in Government. New York: Routledge</source>
          ,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <surname>Demir</surname>
            ,
            <given-names>Cenker</given-names>
          </string-name>
          <string-name>
            <surname>Korhan</surname>
          </string-name>
          , and Engin Avci. “
          <article-title>Turkish Terrorism Studies: A Preliminary Assessment</article-title>
          .”
          <source>All Azimuth</source>
          <volume>7</volume>
          , no.
          <issue>1</issue>
          (
          <year>2018</year>
          ):
          <fpage>21</fpage>
          -
          <lpage>44</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <surname>Forrester</surname>
            ,
            <given-names>Jay W. Industrial</given-names>
          </string-name>
          <string-name>
            <surname>Dynamics</surname>
          </string-name>
          . Cambridge: The M.I.T. Press,
          <year>1961</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <surname>Ghaffarzadegan</surname>
            , Navid, John Lyneis, and
            <given-names>George P.</given-names>
          </string-name>
          <string-name>
            <surname>Richardson</surname>
          </string-name>
          . “
          <article-title>How Small System Dynamics Models Can Help the Public Policy Process</article-title>
          .”
          <source>System Dynamics Review</source>
          <volume>27</volume>
          , no.
          <issue>1</issue>
          (
          <year>2011</year>
          ):
          <fpage>22</fpage>
          -
          <lpage>44</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <surname>Goertz</surname>
            , Gary, and
            <given-names>James</given-names>
          </string-name>
          <string-name>
            <surname>Mahoney</surname>
          </string-name>
          .
          <article-title>A Tale of Two Cultures: Qualitative and Quantitative Research in the Social Sciences</article-title>
          . Princeton: Princeton University Press,
          <year>2012</year>
          . 253
          <string-name>
            <given-names>System</given-names>
            <surname>Dynamics</surname>
          </string-name>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <surname>Hjorth</surname>
          </string-name>
          , Peder, and Ali Bagheri. “
          <article-title>Navigating Towards Sustainable Development: A System Dynamics Approach</article-title>
          .” Futures 38, no.
          <issue>1</issue>
          (
          <year>2006</year>
          ):
          <fpage>74</fpage>
          -
          <lpage>92</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <surname>Jervis</surname>
          </string-name>
          , Robert.
          <source>System Effects: Complexity in Political and Social Life</source>
          . Princeton: Princeton University Press,
          <year>1997</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <surname>Kirkwood</surname>
            ,
            <given-names>Craig W.</given-names>
          </string-name>
          <string-name>
            <surname>System Dynamics</surname>
          </string-name>
          <article-title>Methods: A Quick Introduction</article-title>
          . College of Business, Arizona State University,
          <year>1998</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <surname>Morey</surname>
          </string-name>
          , David S. “
          <article-title>When War Brings Peace: A Dynamic Model of the Rivalry Process</article-title>
          .”
          <source>American Journal of Political Science</source>
          <volume>55</volume>
          , no.
          <issue>2</issue>
          (
          <year>2011</year>
          ):
          <fpage>263</fpage>
          -
          <lpage>75</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <surname>Burton</surname>
            ,
            <given-names>R. M.</given-names>
          </string-name>
          , &amp;
          <string-name>
            <surname>Obel</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          (
          <year>2011</year>
          ).
          <article-title>Computational modeling for what-is, what-might-be, and whatshould-be studies-and triangulation</article-title>
          .
          <source>Organization Science</source>
          ,
          <volume>22</volume>
          (
          <issue>5</issue>
          ),
          <fpage>1195</fpage>
          -
          <lpage>1202</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <surname>Chiu</surname>
          </string-name>
          , C.-Y., &amp;
          <string-name>
            <surname>Qiu</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          (
          <year>2014</year>
          ).
          <article-title>Communication and culture: A complexity theory approach</article-title>
          .
          <source>Asian Journal of Social Psychology</source>
          ,
          <volume>17</volume>
          ,
          <fpage>108</fpage>
          -
          <lpage>111</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <surname>Cioffi-Revilla</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          (
          <year>2017</year>
          ).
          <article-title>Introduction to computational social science: Principles and applications</article-title>
          (2nd ed.). New York, NY: Springer.
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <surname>Clark</surname>
            ,
            <given-names>W. R.</given-names>
          </string-name>
          , &amp;
          <string-name>
            <surname>Golder</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>2015</year>
          ).
          <article-title>Big data, causal inference, and formal theory: Contradictory trends in political science</article-title>
          ? PS: Political Science &amp; Politics,
          <volume>48</volume>
          (
          <issue>1</issue>
          ),
          <fpage>65</fpage>
          -
          <lpage>70</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24]
          <string-name>
            <surname>Saaty</surname>
            <given-names>T.</given-names>
          </string-name>
          <string-name>
            <surname>Theory</surname>
          </string-name>
          and
          <article-title>Aplications of the Analytic Network Process: Decision Making with Benefits, Opportunities, Cost and Risk</article-title>
          . - Pittsburgh, RWS Publications,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [25]
          <string-name>
            <surname>Casey K.L. Defining Political</surname>
          </string-name>
          <article-title>Capital: A Reconsideration of Bourdieu's Interconvertibility Theory /</article-title>
          / Paper University of Missouri. - St. Louis,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          [26]
          <string-name>
            <surname>Petrovsky</surname>
            ,
            <given-names>A.B.</given-names>
          </string-name>
          :
          <article-title>Decision Making Theory</article-title>
          . Publishing Center Academiya, Moscow (
          <year>2009</year>
          ). in Russian
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [27]
          <string-name>
            <surname>Baluta</surname>
            <given-names>V.I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Osipov</surname>
            <given-names>V.P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Yakovenko</surname>
            <given-names>O.Yu.</given-names>
          </string-name>
          <article-title>The environment of modeling, forecasting and expertise as the intellectual core of support management complex systems</article-title>
          .
          <source>KIAM Preprint № 82</source>
          , Moscow,
          <year>2015</year>
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          [28]
          <string-name>
            <given-names>M.A.</given-names>
            <surname>Berberova</surname>
          </string-name>
          ,
          <string-name>
            <surname>K.I.Chernyavskii</surname>
          </string-name>
          , «
          <article-title>Comparative assessment of the NPP risk (on the example of Rostov and Kalinin NPP)</article-title>
          .
          <article-title>Development of risk indicators atlas for Russian NPPs»</article-title>
          ,
          <source>GraphiCon 2019 Computer Graphics and Vision</source>
          .
          <source>The 29th International Conference on Computer Graphics and Vision</source>
          . Conference Proceedings (
          <year>2019</year>
          ), Bryansk, Russia,
          <source>September 23-26</source>
          ,
          <year>2019</year>
          , Vol-
          <volume>2485</volume>
          , urn:nbn:de:
          <fpage>0074</fpage>
          -
          <lpage>2485</lpage>
          -1, ISSN 1613-0073, DOI: 10.30987/graphicon2019-2-
          <fpage>290</fpage>
          -294, http://ceur-ws.org/Vol2485/paper67.pdf, p.
          <fpage>290</fpage>
          -
          <lpage>294</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          [29]
          <string-name>
            <surname>Baluta</surname>
            <given-names>V.I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Osipov</surname>
            <given-names>V.P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chetverushkin</surname>
            <given-names>B.N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Yakovenko</surname>
            <given-names>O.Y.</given-names>
          </string-name>
          <article-title>Simulation of dynamics and adaptation in the space of conflict interaction of the intelligent agents</article-title>
          .
          <source>KIAM Preprint № 111</source>
          , Moscow,
          <year>2019</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>